Channels and TaskQueues in Google App Engine (client debugging)

Question

I'm trying to build and debug my first GAE application and have already benefited from the awesome support of Stackoverflowers to get where I am in having tasks being processed on the default queue. Thanks!

However, I wanted to use Queues to demonstrate how you would do some 'long' work in the background. My idea was:

Receive a request to process a large file.
Store the file and enqueue a task.
Return the response.
Process the file on the background.
Let the client know, via a Channel, that the work is done!

I have all this working but for one problem. On my development server the Task Queue doesn't seem to process tasks in the background. To simulate long running work I just popped a sleep in there.

def post(self):
    time.sleep(60)
    #use a channel to let the client know we're done

It appears that the GAE development server is single threaded. It doesn't respond at all until the item has been processed off the queue? Is this assumption right? Any ideas?

Thanks

Adding code exanples:

#code to enqueue task
taskqueue.add(url='/processSubmission', params={'key': activity.key() }, transactional=False)

#class that processes queued work
class ProcessSubmission(webapp.RequestHandler):
  def post (self):
    time.sleep(60)
    activity = db.get(db.Key(encoded=self.request.get('key')))
    activity.approved = True
    activity.put()
    channel.send_message(activity.userid, 'Wahoo! we are done')

could you post the relevant part of your code? – systempuntoout Feb 07 '11 at 08:11 — systempuntoout, Feb 07 '11 at 08:11
Done - added the code. Hope that's enough. – ConfusedNoob Feb 07 '11 at 18:14 — ConfusedNoob, Feb 07 '11 at 18:14

score 2 · Answer 1 · answered Feb 07 '11 at 22:36

2

Yes, the App Engine dev_appserver is single-threaded, and only processes a single request at a time. Your user-facing request should return before it begins processing the task queue request, however.

answered Feb 07 '11 at 22:36

Nick Johnson

100,655
16
128
198

I've tested this numerous ways now and it really seems this isn't the case. Are you sure that it does return before completing the queued item? If you're right, then my sample with Channels should work (I should note, I have tried simpler setups) – ConfusedNoob Feb 08 '11 at 00:14
1

@ConfusedNoob Yes - tasks are run asynchronously to user requests. Note that the Channel API on the dev_appserver uses polling, though, and the polling requests won't go through until your task has finished executing - perhaps that's the cause of what you're observing? – Nick Johnson Feb 08 '11 at 01:57
That would explain why the whole scenario doesn't work properly, but not why it takes the response that enqueues the task 60 seconds to come through... – ConfusedNoob Feb 08 '11 at 03:16
OK, I've looked into this more and in all my experiments it seems that the page that enqueues the task doesn't respond until the task finished processing. This is even at the HTTP level (via a trace tool). The response only starts to come back after the task has been processed. This feels like a pretty poor development story - any workarounds or suggestions? – ConfusedNoob Feb 09 '11 at 06:17
Note, I've also deployed the app to appengine and it works fine there. – ConfusedNoob Feb 09 '11 at 06:59
1

@ConfusedNoob Are you sure you're running the latest SDK? Older versions executed tasks like that, if I recall correctly. Try reinstalling your SDK. – Nick Johnson Feb 09 '11 at 13:43

Channels and TaskQueues in Google App Engine (client debugging)

1 Answers1