0

Encounter an error "RequestTooLargeError: The request to API call datastore_v3.Put() was too large.".

After looking through the code, it happens on the place where it is using task queue.

So how can I split a large queue task into several smaller ones?

Weiming
  • 37
  • 6

2 Answers2

0

Check the size of the payload (arguments) you are sending to the task queue.

If it's more than a few KB in size you need to store it in the datastore and send the key of the object holding the data to the task queue

Tim Hoffman
  • 12,976
  • 1
  • 17
  • 29
  • How can I check the size of it? I don't want to create a datastore for this. Is it possible that I split the data into smaller one and then send to the queue to process? – Weiming Jan 24 '16 at 13:11
  • you can check the size of it because it was you that created it and passed it to the task! If you want to split the data across tasks then you can also do that, provided the thing that is processing tasks can deal with that! – Paul Collingwood Jan 24 '16 at 14:15
  • @Weiming as you can see from your error message you're using the datastore either way. Doesn't make much sense to avoid it. Sure you can split your job into two smaller jobs but you can't split the payload of a single job. That would be like an HTTP POST Request split into multiple requests. – konqi Jan 24 '16 at 15:20
0

The maximum size of a task is 100KB. That's a lot of data. It's hard to give specific advice without looking at your code, but I would mention this:

  1. If you pass a collection to be processed in a task in a loop, than the obvious solution is to split the entire collection into smaller chunks, e.g. instead of passing 1000 entities to one task, pass 100 entities to 10 tasks.

  2. If you pass a collection to a task that cannot be split into chunks (e.g. you need to calculate totals, averages, etc.), then don't pass this collection, but query/retrieve it in the task itself. Every task is saved back to the datastore, so you don't win much by passing the collection to the task - it has to be retrieved from the datastore anyway.

  3. If you pass a very large object to a task, pass only data that the task actually needs. For example, if your task sends an email message, you may want to pass Email, Name, and Message, instead of passing the entire User entity which may include a lot of other properties.

Again, 100KB is a lot of data. If you are not using a loop to process many entities in your task, the problem with the task queue may indicate that there is a bigger problem with your data model in general if you have to push around so much data every time. You may want to consider splitting huge entities into several smaller entities.

Andrei Volgin
  • 40,755
  • 6
  • 49
  • 58