Google App Engine: Modifying 1000 entities using TaskQueue

Question

I am hoping to modify 1000 entities using task queue, as suggested Zig Mandel in my original question here: Google App Engine: Modifying 1000 entities

I have a UserAccount kind like this:

class UserAccount(ndb.Model):
    email = ndb.StringProperty()

Some of the UserAccount emails contain uppercases (example: JohnathanDough@email.com), and I would like to apply email.lower() to every entity's email.

So I've set up a task queue like this:

class LowerEmailQueue(BaseHandler):

    def get(self):
        all_accounts = UserAccount.query().fetch()
        for a in all_accounts:
            taskqueue.add(url = '/lower-email', params = {'account_id': a.key.id()})


class LowerEmail(BaseHandler):

    def post(self):
        account_id = self.request.get('account_id')
        account = UserAccount.get_by_id(int(account_id))
        account.email = account.email.lower()
        account.put()

app = webapp2.WSGIApplication([
    ('/', MainPage),
    ('/lower-email-queue', LowerEmailQueue),
    ('/lower-email', LowerEmail),


], debug=True)

I have not run this yet, because I want to prevent causing disastrous damage to my data. Should this work?

score 2 · Answer 1 · answered Jul 15 '14 at 15:38

2

No, in fact this will not do anything at all, because you don't do anything with the lowered email address. You need to actually assign it back to the entity.

account.email = account.email.lower()

answered Jul 15 '14 at 15:38

Daniel Roseman

588,541
66
880
895

Good spotting Daniel. Thanks you. Assuming this change, will it work? I've been told that cursors should be involved. – hyang123 Jul 15 '14 at 15:59
Will work now, but its overkill to use one task queue per item. Do 50 per task queue at least. In reality you should look at cursors – Zig Mandel Jul 15 '14 at 18:06
At a minimum, your loop could build small arrays (say 20 items), stringify the array and pass as a task queue param. I mentioned cursors before because that would be the optimal way, the query being made by each task queue and not the loop. With more items that loop would timeout from a frontend instance. – Zig Mandel Jul 15 '14 at 18:23
@ZigMandel, I am not too familiar with task queue and its ability to set 50 per task queue. If it is not too troublesome, can you please post a code to exemplify what you are suggesting? I think I can learn most from seeing the code. Thank you. – hyang123 Aug 18 '14 at 17:00

Google App Engine: Modifying 1000 entities using TaskQueue

1 Answers1

Linked