GAE-ready asynchronous operations in Python?

Question

I've got a Python app making 3 different api calls one after another in the same block of code. I'd like to execute these calls asynchronously, and then perform an action when they're all complete.

A couple notes:

Other answers regarding async actions point to frameworks like Twisted and Celery, but I'm building a Web2Py app for the GAE, so those daemon-based frameworks aren't an option AFAIK.
I'm using api wrapper libraries for the various apis, so I'm wondering if there's an async solution that can be implemented at the thread level, rather than the http request level?

You can make async URL Fetch calls, if that's helpful: http://code.google.com/appengine/docs/python/urlfetch/asynchronousrequests.html — Amy U., Feb 29 '12 at 02:07
Amy- that may be the best/only answer- make it an answer so I can give you some cred.. — Yarin, Feb 29 '12 at 12:11

score 3 · Accepted Answer · answered Feb 29 '12 at 21:46

3

Python supports async URL Fetch calls, which may be helpful: http://code.google.com/appengine/docs/python/urlfetch/asynchronousrequests.html

answered Feb 29 '12 at 21:46

Amy U.

2,227
11
11

Assuming your async URL fetches will be against worker pages you have created, you probably need to enable Concurrent Requests to make sure they actually execute in parallel: http://code.google.com/appengine/docs/python/python27/newin27.html#Concurrent_Requests – Raven Mar 03 '12 at 18:37

score 1 · Answer 2 · answered Feb 28 '12 at 23:21

1

If you're brave, you might try the Experimental new DB api, NDB. It has async APIs for working with the datastore + URL fetch. If those are the things you hoped to do async-ly, then you're in luck.

answered Feb 28 '12 at 23:21

Larry Hosken

99
4

No this doesn't have anything to do w the datastore- just outside APIs – Yarin Feb 28 '12 at 23:31

score 1 · Answer 3 · answered Feb 29 '12 at 16:13

1

Task Queues might get you there. I don't see any promises about responsiveness, but you can definitely control how many tasks are started per second.

answered Feb 29 '12 at 16:13

Raven

1,264
1
12
22

Raven- In my case I need be able to perform an action on completion- I don't see any callback or listener functionality for task queues. However, they look like they could be useful elsewhere, and I didn't know about them, so thanks. – Yarin Mar 02 '12 at 12:58
@Yarin You could have your tasks write the results to the datastore. Then the main code could poll for completion. – Raven Mar 03 '12 at 18:34

GAE-ready asynchronous operations in Python?

3 Answers3