26

I'm confused about Task execution using queues. I've read the documentation and I thought I understood bucket_size and rate, but when I send 20 Tasks to a queue set to 5/h, size 5, all 20 Tasks execute one after the other as quickly as possible, finishing in less than 1 minute.

deferred.defer(spam.cookEggs, 
               egg_keys, 
               _queue="tortoise")  

- name: tortoise
  rate: 5/h  
  bucket_size: 5

What I want is whether I create 10 or 100 Tasks, I only want 5 of them to run per hour. So it would take 20 Tasks approximately 4 hours to complete. I want their execution spread out.

UPDATE

The problem was I assumed that when running locally, that Task execution rate rules were followed, but that is not the case. You cannot test execution rates locally. When I deployed to production, the rate and bucket size I had set executed as I expected.

Will Curran
  • 6,959
  • 15
  • 59
  • 92
  • 1
    Does this help you at all? http://code.google.com/appengine/docs/python/config/cron.html#The_Schedule_Format – joslinm Feb 09 '11 at 21:46
  • So you're saying I should be using Scheduled Tasks instead? – Will Curran Feb 09 '11 at 21:51
  • I honestly couldn't tell you that one way or another. I've only recently begun working with the app engine, so I heard "tasks" then I heard "per hour" and thought scheduled tasks :] I think it's worth a design consideration -- especially if nothing else works.. – joslinm Feb 09 '11 at 22:11
  • 7
    Is this in production, or the dev_appserver? I don't believe the dev_appserver respects execution rates. – Nick Johnson Feb 14 '11 at 03:24
  • It's on dev_appserver, and I believe you're right. I created a "test" version for production and it executed as expected. – Will Curran Feb 19 '11 at 03:15
  • 4
    Can you answer your own question in a way that would help others? If you do, you can select yours as the correct answer. It may seem strange, but it is preferred to deleting in situations like this. –  Apr 08 '11 at 14:44
  • 2
    You should post your answer as an answer and mark it as accepted. So other SO users can tell you're question has already been answered. – Eduardo Aug 23 '11 at 09:06

2 Answers2

7

Execution rates are not honored by the app_devserver. This issue should not occur in production.

[Answer discovered by Nick Johnson and/or question author; posting here as community wiki so we have something that can get marked accepted]

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
-1

You want to set bucket_size to 1, or else you'll have "bursts" of queued activity like you saw there.

From the documentation:

bucket_size

Limits the burstiness of the queue's processing, i.e. a higher bucket size allows bigger spikes in the queue's execution rate. For example, consider a queue with a rate of 5/s and a bucket size of 10. If that queue has been inactive for some time (allowing its "token bucket" to fill up), and 20 tasks are suddenly enqueued, it will be allowed to execute 10 tasks immediately. But in the following second, only 5 more tasks will be able to be executed because the token bucket has been depleted and is refilling at the specified rate of 5/s.

rmmh
  • 6,997
  • 26
  • 37
  • I did read that, several times, and to me that says that if I have rate at 5/h, and a bucket size of 5, then 5 Tasks will run immediately, but then it will take an hour for 5 more tokens to fill up the bucket, so 5 more Tasks would run over that hour, and so on. But what I am seeing is all 20 Tasks are running immediately, one after the other. – Will Curran Feb 09 '11 at 21:53
  • And when I set bucket_size to 1, it executes at the same rate as far as I can tell. – Will Curran Feb 09 '11 at 22:07