0

I am trying to upload 1744 small files to the blobstore (total size of all the files is 4 MB) and get HTTP/1.1 503 Service Unavailable error.

This is 100% reproducible.

Is it a bug, I do I violate any constraints? I don't see any constraints in the documentation about number of blobs submitted in one call.

user431640
  • 61
  • 7

2 Answers2

2

The answer that claims that create_upload_url can only accept one file per upload above is wrong. You can upload multiple files in a single upload and this is the way you should be approaching your problem.

That being said, there was a reliability problem when doing a batch upload that was worked on and fixed around a year or so ago. If possible I would suggest keeping the batch sizes a little smaller (say 100 or so files in a batch). Each file in the batch results in a write to the datastore to record the blob key so 1744 files == 1744 writes and if one of them fails then your entire upload will fail.

If you give me the app_id I can take a look at what might be going wrong with your uploads.

Stuart Langley
  • 7,044
  • 1
  • 20
  • 20
  • the app_id is cl-test-grid, it would be great if you take a look – user431640 Aug 27 '12 at 07:13
  • I deleted my answer.. but I wasn't entirely wrong :) I couldn't find where I read about the fact that something was broken.. thanks for the insight :) – Lipis Aug 27 '12 at 07:29
  • Stuart, I've registered it as a ticket: https://code.google.com/p/googleappengine/issues/detail?id=8032 – user431640 Aug 28 '12 at 18:32
  • BTW, unfortunately splitting the batch into smaller ones is not an immediately available alternative, it would take redesign of the application logic, because now every upload results in sending an email to admin, and if we have split it, the number of emails will increase by order of 10. – user431640 Aug 28 '12 at 18:34
  • sure i understand - you could always move your e-mail sending to a task queue based solution where you delay sending the mail until all of the blobs have uploaded. – Stuart Langley Aug 28 '12 at 22:30
  • FYI I updates the issue you opened - but currently you cannot upload > 500 blobs in a single request due to how we're batching the writes to the datastore. We will fix this but it will not be for 1 or 2 releases. – Stuart Langley Aug 30 '12 at 05:21
0

So, the answer. Currently only < 500 files may be submitted in one request. This is going to be fixed in the scope of the ticket http://code.google.com/p/googleappengine/issues/detail?id=8032 so that unlimited number of files may be submitted. But it may take a GAE release or 2 before the fix is deployed.

user431640
  • 61
  • 7