KeyError during python GAE push task queue with cron

Question

I'm very close to finishing up a project that uses push task queues in GAE to send out follow-up emails to users. However, I keep getting a KeyError and don't know why. I been looking for good models to base my project on but haven't found any decent examples that use multiple parameters. The GAE documentation has improved in the last month but still leaves a lot to be desired.

I've checked many pieces of the code using the interactive console in the development server but still I don't know what I am doing wrong. My best guess is that the parameters are not getting passed along to the next part of the script (class pushQueue).

app.yaml:

application: gae-project
version: 1
runtime: python27
api_version: 1
threadsafe: true

handlers:
- url: /cron/sendfu
  script: main.app
  login: admin

- url: /emailworker
  script: main.app
  login: admin

- url: /worker
  script: main.app
  login: admin

- url: /.*
  script: main.app
  login: required

cron.yaml:

cron:
- description: sends follow-up emails
  url: /cron/sendfu
  schedule: every day 20:00

queue.yaml:

total_storage_limit: 120M
queue:
- name: emailworker
  rate: 1/s
  bucket_size: 50
  retry_parameters:
    task_retry_limit: 5
    task_age_limit: 6h
    min_backoff_seconds: 10
    max_backoff_seconds: 60

main.py:

import webapp2
import datetime
from google.appengine.ext import db
from google.appengine.api import users
from google.appengine.api import taskqueue
import jinja2
import os

jinja_environment = jinja2.Environment(
loader=jinja2.FileSystemLoader(os.path.dirname(__file__)))

class emailJobs(db.Model):
    """ Models an a list of email jobs for each user """
    triggerid = db.StringProperty()  #Trig id
    recipientid_po = db.StringProperty() # id
    recipientlang = db.StringProperty()  #Language
    fu_email_sent = db.DateTimeProperty() 
    fuperiod = db.IntegerProperty() # (0 - 13)
    fu1 = db.DateTimeProperty() 
    fu2 = db.DateTimeProperty()
    
    @classmethod
    def update_fusent(cls, key_name, senddate):
        """ Class method that updates fu messages sent in the GAE Datastore """
        emailsjobs = cls.get_by_key_name(key_name)
        if emailsjobs is None:
            emailsjobs = cls(key_name=key_name)
        emailsjobs.fu_email_sent = senddate
        emailsjobs.put()

def timeStampFM(now):
    d = now.date()
    year = d.year
    month = d.month
    day = d.day
    t = now.time()
    hour = t.hour
    minute = t.minute + 5
    second = t.second
    today_datetime = datetime.datetime(year, month, day, hour, minute, second)
    return today_datetime


class MainPage(webapp2.RequestHandler):
    """ Main admin login page """
    def get(self):
        if users.get_current_user():
            url = users.create_logout_url(self.request.uri)
            url_linktext = 'Logout'
            urla = '/'
            url_admin = ""
            if users.is_current_user_admin():
                url = users.create_logout_url(self.request.uri)
                urla = "_ah/admin/"
                url_admin = 'Go to admin pages'
                url_linktext = 'Logout'
             
        else:
            url = users.create_login_url(self.request.uri)
            url_linktext = 'Login'

        template_values = {
            'url': url,
            'url_linktext': url_linktext,
            'url_admin': url_admin,
            'urla': urla,
            }

        template = jinja_environment.get_template('index.html')
        self.response.out.write(template.render(template_values))


class sendFollowUp(webapp2.RequestHandler):
    """ Queries Datastore for fu dates that match today's date, then adds them to a task queue """
    def get(self):
    
        now = datetime.datetime.now()
        now_dt = now.date() #today's date to compare with fu dates
   
        q = emailJobs.all()
        q.filter('fuperiod >', 0)
        q.filter('fuperiod <', 99)

        for part in q:
            guid = str(part.recipientid_po)
            lang = str(part.recipientlang)
            trigid = str(part.triggerid)

            if part.fuperiod == 1:
                fu1rawdt = part.fu1
                fu1dt = fu1rawdt.date()
                if fu1dt == now_dt:
                    follow_up = '1'
                
            if part.fuperiod == 2:
                fu2rawdt = part.fu2
                fu2dt = fu2rawdt.date()
                if fu2dt == now_dt:
                    follow_up = '2'
                
            if follow_up != None:
                taskqueue.add(queue_name='emailworker', url='/emailworker', params={'guid': guid,
                                                                                'fu': follow_up,
                                                                                'lang': lang,
                                                                                'trigid': trigid,
                                                                                })
        self.redirect('/emailworker')


class pushQueue(webapp2.RequestHandler):
    """ Sends fu emails, updates the Datastore with datetime sent """

    def store_emails(self, trigid, senddate):
        db.run_in_transaction(emailJobs.update_fusent, trigid, senddate)
        
    def get(self):
        fu_messages = {'1': 'MS_x01', 
                       '2': 'MS_x02',
                       }
        langs = {'EN': 'English subject',
                 'ES': 'Spanish subject',
                 }
    
        fu = str(self.request.get('fu'))
        messageid = fu_messages[fu]

        lang = str(self.request.get('lang'))
        subject = langs[lang]
    
        now = datetime.datetime.now()
        senddate = timeStampFM(now)
     
        guid = str(self.request.get('guid'))
        trigid = str(self.request.get('trigid'))
    
        data = {}
        data['Subject'] = subject
        data['MessageID'] = messageid
        data['SendDate'] = senddate
        data['RecipientID'] = guid
        # Here I do something with data = {}
    
        self.store_emails(trigid, senddate)
    
app = webapp2.WSGIApplication([('/', MainPage),
                           ('/cron/sendfu', sendFollowUp),
                           ('/emailworker', pushQueue)],
                           debug=True)

When I test the cron job at: localhost:8086/cron/sendfu

It redirects to: localhost:8086/emailworker

and I get the following error message:

Internal Server Error

The server has either erred or is incapable of performing the requested operation.

Traceback (most recent call last):
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
  File "/Users/me/Documents/workspace/gae-project/src/main.py", line 478, in get
messageid = fu_messages[fu]
KeyError: ''

from the logs:

INFO     2013-03-05 03:03:22,337 dev_appserver.py:3104] "GET /cron/sendfu HTTP/1.1" 302 -
ERROR    2013-03-05 03:03:22,348 webapp2.py:1552] ''
Traceback (most recent call last):
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
  File "/Users/me/Documents/workspace/gae-project/src/main.py", line 478, in get
messageid = fu_messages[fu]
KeyError: ''
INFO     2013-03-05 03:03:22,355 dev_appserver.py:3104] "GET /emailworker HTTP/1.1" 500 -
INFO     2013-03-05 03:03:22,509 dev_appserver.py:3104] "GET /favicon.ico HTTP/1.1" 404 -

lines:

469    def get(self):
470        fu_messages = {'1': 'MS_x01', 
471                       '2': 'MS_x02',
472                       }
473        langs = {'EN': 'English subject',
474                 'ES': 'Spanish subject',
475                 }
476
477        fu = str(self.request.get('fu'))
478        messageid = fu_messages[fu]

Can you try to pinpoint this to a usable stacktrace, like where line 478 occurs in `main.py`? Also, what request causes this to occur? Just running the server? — bossylobster, Mar 05 '13 at 02:19
Hi bl, sorry, I'm a python and GAE newbie, and don't understand all the lingo. The only thing on line 478 is: messageid = fu_messages[fu]. The request is the cron job that I am running in the development server. I hope that answers your question. Does my code look alright? — 655321, Mar 05 '13 at 02:26
Can you include the logs from the development server for the failed request? Can you include the code around line 478? — bossylobster, Mar 05 '13 at 02:59
Hi bl, I hope that's what you were asking for. The part of the code that I truncated is after data['RecipientID'] = guid, but the problem comes way before. — 655321, Mar 05 '13 at 03:15

bossylobster · Accepted Answer · 2013-03-05T05:08:38.357

2

When you call

fu = str(self.request.get('fu'))

if there is no 'fu' in the request, self.request.get will return the empty string (''). So when you try

messageid = fu_messages[fu]

it looks up the empty string in

fu_messages = {'1': 'MS_x01', 
               '2': 'MS_x02',
               }

which only has '1' and '2' as keys.

The reason your pushQueue handler is not seeing the params you send via

params = {
    'guid': guid,
    'fu': follow_up,
    'lang': lang,
    'trigid': trigid,
}
taskqueue.add(queue_name='emailworker', url='/emailworker', 
              params=params)

is because you are using a GET handler instead of a POST or PUT handler. As the documentation states:

Params are encoded as application/x-www-form-urlencoded and set to the payload.

So the payload of the request has your 'fu' param in it, but since it is a GET request, the payload is dropped (this is how HTTP works, not specific to App Engine). If you use POST as your handler, the payload will come through as expected.

I noticed your code is very similar to the documented sample, but simply uses get where the sample uses post.

edited Mar 05 '13 at 05:08

answered Mar 05 '13 at 03:27

bossylobster

9,993
1
42
61

I only have complete data in the Datastore for the fu field. All the entities have values. I tried the query in the interactive console and it returned values. What could be wrong? – 655321 Mar 05 '13 at 03:30
When you use `self.request.get`, `'fu'` comes from the request not the datastore. You should brush up on accessing data from the datastore (https://developers.google.com/appengine/docs/python/ndb/overview) and revisit your code after doing so. – bossylobster Mar 05 '13 at 03:32
What starts off the script is the query of the Datastore but then it passes it on to the task queue and that's where the problem starts. It's the taskqueue.add(queue_name='emailworker'... and the retrieval of those parameters that I don't get. The documentation on task queues is confusing. – 655321 Mar 05 '13 at 03:38
How do you mean "what starts off the script is the query of the Datastore"? Why are you using the task queue to query the datastore? The request that is failing is from the cron and is to your handler. A request to a handler is not the same as a datastore request. What part of the documentation are you confused by? – bossylobster Mar 05 '13 at 04:07
Sorry, I'm not trained as a programmer, so the lingo is not my forte. I'm querying the Datastore with a cron job because the Datastore is being updated by another process, so I'm looking up the new values on a schedule. I then pass on these new values via the task queue which I ultimately use in a REST API call to another service to send an email. The part I don't understand is what is going wrong with the task queue, handler. To answer your question, the task queue documentation is confusing. It doesn't have complete or several examples. I guess as a neophyte I'm expecting too much. – 655321 Mar 05 '13 at 04:16
1

I updated my answer after reading more of your code. No one wants to read that much code and only after our conversation in the comments could I get an idea of what to look for, rather than doing a full code review of your application. The key to answering questions here is narrowing down your problem. This easily could've been two SO questions: 1) Why is `self.request.get('foo')` the empty string? and 2) Why isn't the payload being sent when I use `params=` with `taskqueue.add`? – bossylobster Mar 05 '13 at 04:47
Thank you! I changed the get to a post request but got an error message saying that it wasn't allowed. So would I need to have both a get and a post request in the sendFollowUp handler? One to first get the values from the Datastore query and then one to post them to the task queue? If so, how do you transition from get to post in the same handler (sendFollowUp)? Thanks again. – 655321 Mar 05 '13 at 22:51
Please start a new question and we can address this next issue there. – bossylobster Mar 05 '13 at 23:31

KeyError during python GAE push task queue with cron

Internal Server Error

1 Answers1

Linked