GAE SDK 1.6.4 dev_appserver datastore flush

Question

Hoping to get a comment from the GAE python team on this one.

Has something changed between 1.6.3, 1.6.4 with regards to the way the dev_appserver datastore is flushed to disk on app exit?

I'm using django-nonrel, and on 1.6.3, and before, I used to be able to load up a python shell: python manage.py shell (manage.py is provided by django-nonrel)

I could then import my models and acceess the datastore, modify entities, and quit.

On 1.6.4, I'm able to do that, but when I quit, changes are not saved to the datastore. When I run django-nonrel as a WSGI app, it saves properly, and I see a message on exit ("Applying all pending transactions and saving the datastore").

Thanks Guido, I hacked in a fix for django-nonrel. I suspect any other python users who want to get at the datastore via the python shell will have a similar issue. — dragonx, Apr 08 '12 at 04:20
See Alfred Fuller's response to http://stackoverflow.com/questions/10047315/app-engine-local-datastore-content-does-not-persist for an explanation and source links. — Guido van Rossum, Apr 08 '12 at 14:40
@Guido What about handling exit() from the shell? That doesn't send a TERM or KeyInterrupt signal. Is using atexit, like in my answer below correct? — dragonx, Apr 08 '12 at 23:24

user1325188 · Accepted Answer · 2012-04-22T06:12:33.717

3

Thanks to dragonx for his solution and info. I run my devserver from eclipse, and I was amazed to see my data not beeing saved after upgrading to 1.6.4 I added a flush to the database after every web request, to do that I implemented a base class for all requests and override dispatch:

developmentServer = False

if os.environ.get('SERVER_SOFTWARE','').startswith('Development'):
    developmentServer = True

class BaseRequestHandler(webapp2.RequestHandler):
    def dispatch(self):
        retValue = super(BaseRequestHandler, self).dispatch()
        if developmentServer:
                from google.appengine.tools import dev_appserver 
                dev_appserver.TearDownStubs()

        return retValue

informing about a change in behavior like that in the release notes, would have saved me two days of searching what went wrong in my upgrade.

edited Apr 22 '12 at 06:12

answered Apr 10 '12 at 22:18

user1325188

61
4

This code above doesn't work for me on Windows. I've tried launching the application on the development server (1) using Eclipse and (2) using the Google App Engine Launcher. Either way, i.e. (1) terminating the console in Eclipse and (2) stopping the application from the launcher, transactions are not saved to the development datastore. Anyone having similar issues on Windows? Anyone who could offer some thoughts on this? – cv12 Apr 18 '12 at 05:08
I changed the solution to work in eclipse, save the database every time like before google update – user1325188 Apr 22 '12 at 06:14
What I did finally was to create a button on my app that sends the flush when I press it – user1325188 Apr 22 '12 at 09:17

dragonx · Answer 2 · 2012-04-11T05:20:19.760

2

It looks like there have been some changes. I've been able to hack around the problem with the following:

from google.appengine.tools import dev_appserver
import atexit
atexit.register(dev_appserver.TearDownStubs)

This ensures the datastore is flushed on exit.

edited Apr 11 '12 at 05:20

answered Apr 08 '12 at 04:11

dragonx

14,963
27
44

If you're running "python manage.py shell" or "python manage.py createsuperuser". These cases weren't working before. If you run "python manage.py runserver", you'll get the flush twice, once from the SDK handler, and one from the extra handler I added. I'm not really sure how to distinguish the python shell exit() (missed by SDK) from SIGTERM/KeyInterrupt (caught by SDK) – dragonx Apr 09 '12 at 00:45
thanks, I don't know what I did wrong in my post there, I think I just copy and pasted wrong, but the code in my source tree is what you had. I edited it to fix it. – dragonx Apr 11 '12 at 05:21

Alfred Fuller · Answer 3 · 2012-04-08T05:58:03.140

0

Before 1.6.4, we saved the datastore after every write. This method does not work when simulating the transactional model found in the High Replication Datastore (you would lose the last couple writes). It is also horribly inefficient. We changed it so the datastore dev stub flushs all writes and saves it's state on shut down.

Following the code:

It looks like manage.py should work if the server is shut down cleanly (with a TERM signal or KeyInterrupt).

edited Apr 08 '12 at 05:58

answered Apr 08 '12 at 05:30

Alfred Fuller

773
3
5

That's exactly what I'm seeing. Shutdown with a TERM signal or KeyInterrupt flushes properly. Exiting using exit() in a python shell does not. See my above post. The shell catches KeyInterrupt, so you can't exit that way when using the python shell. – dragonx Apr 08 '12 at 14:12
1

Alfred, can you elaborate on what should happen on Windows? 1.6.4 does not flush neither on py_dev stopping debugging, nor on stopping the Application in the launcher – Michael Kariv Apr 18 '12 at 10:30

GAE SDK 1.6.4 dev_appserver datastore flush

3 Answers3

Linked