Web2py, multiple Ajax calls and session variables stored in MySQL

Question

In order for a progress bar using cache.ram, and an ajax call to work properly, there is a need to clean (or just unlock?) sessions. If the app doesn't use session variables to pass to other controllers then session.forget(response) is sufficient.

If on the other hand, the app needs session variables (as my app does indeed) - then session.forget doesn't do much good since it brakes the session variables chain of transmission to other controllers.

As per Anthony suggestion in another post (Progress bar and sessions) on Web2Py google group:

I tried first using the cookies (with maximum zlib compression - level 9) . Still it's not enough for my needs ... I need more than 10k per session to be passed to other controllers, and as mentioned in the book - keeping sessions in cookies is not a good solution if the session are expected to be large (more than 4k/cookie I assume).

So I am trying session.connect with a MySQL db. I can see the table web2py_session_AppName being populated and I understand the session_data column is pickled with cPickle. However...It looks the data in this column, session_data is encrypted, not just pickled.

How do I unpickle the information in session_data column from a MySQL table ? I tried using cPickle.loads...with the data as is, with the data as a string...but it still throws a "bad pickle error"

Here is the controller:

def index():
    session.Yawza = 'Yawza from index'
    cache.ram(CacheKeyUID, lambda: 0, 0)
    return dict()

def cache_in_ram():
    for i in range(int(1e5)):
        avoda = int((i/(1e5))*100)
        cache.ram(CacheKeyUID, lambda:avoda, 0)
    session.Yawza = "Yawza from the cache_in_ram"
    redirect(URL('Test'))

def cache_reader():
    return cache.ram(CacheKeyUID, lambda: None, None)

def test():
    return dict(message = 'Long process ended')

The above will have the progress bar work ONLY the first time. After that progress bar will not work (due to locking by the multiple ajax calls) - unless I manually clear sessions or programmatically do a session.forget(response) which will unfortunately eliminate the session.Yawza. I need this session.Yawza for OTHER purposes than the progress bar, in other controllers down stream.

Adding session.connect(request,response,cookie_key='yoursecret',compression_level=9) to my db.py helps, but is limited to 4k size cookies. I need bigger cookies than that. Thus, I moved towards storing sessions in the db, which in my case is a MySQL db.

Defined session.connect(request, response, db) in db.py. I can see in MySQL the table web2py_session_AppName being populated, but I do NOT have access to session.Yawza at all. Not from the index function and not from the cache_in_ram function.

That's the reason I thought I should MANUALLY unpickle the session_data column...to get access to session.Yawza which must be hiding there.

Here is a small sample of what can be found in the session_data column in the MySQL table for web2py_sessions_MyAppName : gAJjZ2x1b24uZ2xvYmFscwpTZXNzaW9uCnEBfXECVQVZYXd6YXEDVQ1mcm9tIHRleHQgYm9...

It doesn't look like a kosher pickle to me ... ;-))

What am I doing or not doing correctly in order to benefit from the multiple ajax calls, no sessions locking and still use session variables to transfer info to other controllers ?

Answer 1

You might consider whether you can alter the long running cache_in_ram function without having to write to the session. In that case, you can call session.forget(response) at the beginning of that function, and everything will work fine, even with the default file-based sessions (i.e., no need to move sessions to cookies or the database).

If you must write to the session from within the cache_in_ram request, you can first unlock the session, then run the long loop, and then re-open the session and update it as needed:

def cache_in_ram():
    session.forget(response) # Unlock the session file.
    for i in range(int(1e5)):
        avoda = int((i/(1e5))*100)
        cache.ram(CacheKeyUID, lambda:avoda, 0)
    session.connect() # Re-open/lock the session file.
    session._forget = False # Reverse the "forget" so we can write to the session.
    session.Yawza = "Yawza from the cache_in_ram"
    redirect(URL('Test'))

Alternatively, you can switch to storing sessions in the database, which does not do any locking and therefore will not cause the Ajax progress checks to be blocked by the long-running cache_in_ram request. In that case, there is no need to manually read and unpickle the session data from the database -- just use the session object as usual. If you find you cannot access the session data as expected in this case, then you have done something wrong either in setting up the connection or in how you are writing to/reading from the session. You should post a separate question about that, showing all of your code.