creating a request from bufio.Reader

Question

I am trying to implement a batch handler that accepts multipart mixed.

My currently somewhat naive implementation looks like the below. Later I will try to aggregate the responses and send a multipart response.

My current issue is that I am not able to parse the body of the individual parts into a new request.

func handleBatchPost(w http.ResponseWriter, r *http.Request) {
  // read the multipart body
  reader, err := r.MultipartReader()
  if err != nil {
    http.Error(w, fmt.Sprintf("could not read multipart %v\n", err), http.StatusBadRequest)
  }

  // read each part
  for {
    part, err := reader.NextPart()
    if err == io.EOF {
      break
    } else if err != nil {
      http.Error(w, fmt.Sprintf("could not read next part %v\n", err), http.StatusBadRequest)
      return
    }

    // check if content type is http
    if part.Header.Get("Content-Type") != "application/http" {
      http.Error(w, fmt.Sprintf("part has wrong content type: %s\n", part.Header.Get("Content-Type")), http.StatusBadRequest)
      return
    }

    // parse the body of the part into a request
    req, err := http.ReadRequest(bufio.NewReader(part))
    if err != nil {
      http.Error(w, fmt.Sprintf("could not create request: %s\n", err), http.StatusBadRequest)
      return
    }

    // handle the request
    router.ServeHTTP(w, req)
  }
}

func handleItemPost(w http.ResponseWriter, r *http.Request) {
  var item map[string]interface{}
  if err := json.NewDecoder(r.Body).Decode(&item); err != nil {
    http.Error(w, fmt.Sprintf("could not decode item json: %v\n", err), http.StatusBadRequest)
    return
  }
  w.Write([]byte(`{"success": true}`))
}

I am getting an error response from the server. It seems like ReadRequest is not reading the body but only the headers (method, url, etc).

could not decode item json: EOF

This is the payload I am sending.

POST /batch  HTTP/1.1
Host: localhost:8080
Content-Type: multipart/mixed; boundary=boundary

--boundary
Content-Type: application/http
Content-ID: <item1>

POST /items HTTP/1.1
Content-Type: application/json

{ "name": "batch1", "description": "batch1 description" }

--boundary
Content-Type: application/http
Content-ID: <item2>

POST /items HTTP/1.1
Content-Type: application/json

{ "name": "batch2", "description": "batch2 description" }

--boundary--

I found this pattern on the gmail api docs https://developers.google.com/gmail/api/guides/batch.

Answer 1

The main problem is that your payload does not specify Content-Length header for the sub-requests. In case of a missing Content-Length header, http.ReadRequest() will assume no body, will not read and present the actual body, this is why you get EOF errors.

So first provide the missing Content-Length headers:

POST /batch  HTTP/1.1
Host: localhost:8080
Content-Type: multipart/mixed; boundary=boundary

--boundary
Content-Type: application/http
Content-ID: <item1>

POST /items HTTP/1.1
Content-Type: application/json
Content-length: 58

{ "name": "batch1", "description": "batch1 description" }

--boundary
Content-Type: application/http
Content-ID: <item2>

POST /items HTTP/1.1
Content-Type: application/json
Content-length: 58

{ "name": "batch2", "description": "batch2 description" }

--boundary--

With this it should work, but note that since you are processing parts in the same loop, and calling router.ServeHTTP(w, req) in the end, you're reusing the w writer. What does this mean? If handleItemPost() writes anything to the output, subsequent calls to handleItemPost() can't take that back.

E.g. if a handleItemPost() fails, it responds with an HTTP error (this implies setting response status and writing the body). A subsequent handleItemPost() can't report an error again (headers are already committed), and also if it would report success, the error header is already sent and could only write further message to the error body.

So for example if we modify handleItemPost() to this:

func handleItemPost(w http.ResponseWriter, r *http.Request) {
    var item map[string]interface{}
    if err := json.NewDecoder(r.Body).Decode(&item); err != nil {
        fmt.Printf("JSON decode error: %v\n", err)
        return
    }
    fmt.Printf("Success, item: %v\n", item)
}

And execute the following curl command:

curl localhost:8080/batch -X POST \
    -H "Content-Type: multipart/mixed; boundary=boundary" \
    -d '--boundary
Content-Type: application/http
Content-ID: <item1>

POST /items HTTP/1.1
Content-Type: application/json
Content-length: 58

{ "name": "batch1", "description": "batch1 description" }

--boundary
Content-Type: application/http
Content-ID: <item2>

POST /items HTTP/1.1
Content-Type: application/json
Content-length: 58

{ "name": "batch2", "description": "batch2 description" }

--boundary--'

We will see the following output:

Success, item: map[description:batch1 description name:batch1]
Success, item: map[description:batch2 description name:batch2]

Note that if handleItemPost() needs to remain fully functional and callable on its own (to process the request and produce response), you can't use the same http.ResponseWriter for all of its calls.

In this case you may create and use a separate http.ResponseWriter for each of its invocation. The standard lib has an httptest.ResponseRecorder type that implements http.ResponseWriter. It's primarily for testing purposes, but you may use it here too. It records the written response, so you may inspect it after the call.

For example:

w2 := httptest.NewRecorder()
router.ServeHTTP(w2, req)
if w2.Code != http.StatusOK {
    fmt.Printf("handleItemPost returned non-OK status: %v\n", w2.Code)
    fmt.Printf("\terror body: %v\n", w2.Body.String())
}

Running this with your original request (without specifying Content-Length), the output will be:

handleItemPost returned non-OK status: 400
        error body: could not decode item json: EOF


handleItemPost returned non-OK status: 400
        error body: could not decode item json: EOF

But when you specify the Content-Length of the sub-requests, no output (error) is printed.