3

I'm having a really hard time finding documentation on a very basic question about AppEngine:

Does the BlobStore support Chunked Transfer Encoding for uploads?

I'm using an HttpURLConnection object in Java with setChunkedStreamingMode to upload a file in a multipart/form-data type request using the following code to set up the connection:

HttpURLConnection cxn = (HttpURLConnection) new URL(uploadUrl).openConnection();
cxn.setRequestMethod("POST");
cxn.setChunkedStreamingMode(9999);
cxn.setRequestProperty("Content-Type", "multipart/form-data; boundary=-");
cxn.setDoOutput(true);
cxn.connect();

The dev-server answers my request with Status 411: Length required. Does this mean that chunked transfer mode is not supported, or am I initializing the connection incorrectly? Does the production server act differently here? Is this behavior a consequence of specifying a max upload size when generating the upload url?

EDIT:

If I simply comment out the line cxn.setChunkedStreamingMode(9999);, everything works perfectly, but I'd rather not do this, so I don't have to buffer hundreds of MB in memory before sending the request...

Markus A.
  • 12,349
  • 8
  • 52
  • 116
  • The production server should handle this - have you tried it? – Stuart Langley Oct 30 '12 at 09:34
  • If you're trying this against production and it's returning that response, I would say the answer is no, it does not support it. However, you shouldn't have to buffer the whole file - you just need to know its length. – Nick Johnson Oct 30 '12 at 10:57
  • @NickJohnson I am actually compressing and encrypting data on the fly as I upload it and my processing functions only have a throughput of a few MB/s. This is still much faster than most people's upload, so if I can stream the upload as the data becomes available, the processing will not lead to any slow down at all, but if I need to do this step in advance, it becomes very noticeable. Unfortunately, since I don't know how much compression I will get, these are the only two choices I have (chunked streaming or full processing in advance to determine the upload length before the request). – Markus A. Oct 30 '12 at 15:51
  • 1
    @StuartLangley Just read that you are working on the AppEngine team and wanted to thank you for being involved in such an awesome product! Do you know if there are any plans for implementing chunked transfer in the dev server as well? It would make testing complex code using this feature (which may become increasingly frequent as more and more Android apps use AppEngine) just that much easier. Thanks! – Markus A. Oct 30 '12 at 17:27
  • Filing a feature request in the issue tracker would be the way to get this added, if enough people star it. – Stuart Langley Oct 31 '12 at 00:44

1 Answers1

5

Here are the result of my tests regarding the questions above:

  1. The development server does not support chunked transfer encoding for uploads.
  2. The production server does support it, as indicated by Stuart. (phew)
  3. I'll report back on any dependence on specifying an upload size limit, but I doubt it matters.

It's unfortunate, that I have to write two versions of my code, one for testing on the dev server and one for running on the production server, but this is definitely an acceptable scenario.

Markus A.
  • 12,349
  • 8
  • 52
  • 116