Google App Engine Python - Protorpc && Taskqueue

Question

How to use Task Queue (Push Queue) with Protorpc.

I have a landing page form that do multiple actions when sending it:

• Save the fields in the DataStore
• Send an email to the form's sender
• Send the fields to a third party application (let's say a CRM)

The form send is implemented in the server side with protorpc.

class FormRequest(messages.Message)
  field1 = messages.StringField(1, required=True)
  field2 = messages.StringField(2, required=True)

...

class FormApi(remote.Service):
  @remote.method(TravelRequest, message_types.VoidMessage)
  def insert(self, request):
    # Save the form in the DataStore
    travel = FormModel(field1=request.field1, field2=request.field2)
    travel.put()

    # Send an email to the client
    ...

    # Send the data to a third party
    ...

    return message_types.VoidMessage()

This solution is stuck because the user need to wait all this request time. (In this case it is only 2-3s but it is a lot for a landing page form)

A good solution will be to use taskqueue to minimise the time the user need to wait:

(As an example)

class ...
  @remote ...
  def ...
    # Save the form in the DataStore
    taskqueue.add(url='/api/worker/save_to_db', params={'field1': request.field1, 'field2': request.field2})
    # Send an email to the client
    taskqueue.add(url='/api/worker/send_email', params={'field1': request.field1, 'field2': request.field2})
    # Send the data to a third party (CRM)
    taskqueue.add(url='/api/worker/send_to_crm', params={'field1': request.field1, 'field2': request.field2})

The "problem" is that protorpc get only json object as request. How to do this with TaskQueue(Push) ?

The default behavior of TaskQueue is to send params as a string of urlencoded and it's not conveniant to protorpc.

Answer 1

Let's define a Worker service for the taskqueue:

class WorkersApi(remote.Service):
  @remote.method(TravelRequest, message_types.VoidMessage)
  def save_to_db(self, request):
    # Instead of write each parameter, I am using this "cheat"
    params = {}
    for field in request.all_fields():
      params[field.name] = getattr(request, field.name)

    # Save data in the datastore
    form_model = FormModel(**params)
    form_model.put()

    return message_types.VoidMessage()

Pay attention that I am using the same message object for the real request and for the taskqueue request (It is a big advantage to need not create and different message object for each request) The question is how to use taskqueue with this protorpc function.

As I say in the question, the default behavior of taskqueue is not conveniant.

The solution is to convert the orignal request/message (in our example the FormRequest) object back to string and set a header to taskqueue that the payload is application/json.

Here's the code:

# This format string is take from the util file in the protorpc folder in Google App Engine source code
format_string = '%Y-%m-%dT%H:%M:%S.%f'

params = {}
for field in request.all_fields():
  value = getattr(request, field.name)
  if (isinstance(value, datetime.datetime)):
    value = value.strftime(format_string)
  params[field.name] = value

taskqueue.add(url='/api/workers.save_to_db', payload=json.dumps(params), headers={'content-type':'application/json'})

Do the same for the "email" and the "crm".

Answer 2

you can used put_async() for write with no time : Asynchronously writes the entity's data to the Datastore.

for example:

travel = FormModel(field1=request.field1, field2=request.field2)
travel.put_async()
# next action