1

In HTTP server I'm trying to pipe a file from a io.Writer to io.Reader. I'm using io.Pipe to transfer the data.

The common pattern for this seems to be something like this:

// Some handler func ..
pr, pw := io.Pipe()
go func() {
    defer pw.Close()
    if err := fetchFile(pw); err != nil {
        pw.CloseWithError(err)
    }
}()

writer.WriteHeader(http.StatusOK)
io.Copy(responseWriter, pr)

The issue I'm having is that if sometimes the writer returns an error(e.g file not found), but at this point the HTTP status OK header and Content-Type has already been written.

What I came up with was using a bufio.Reader to peek into the buffer before sending anything down to the client.

Something like this:

pr, pw := io.Pipe()
go func() {
    defer pw.Close()
    if err := fetchFile(pw); err != nil {
        _ = pw.CloseWithError(err)
    }
}()

// Do a peek to determine if we have the data flowing or we have encountered some error
bufReader := bufio.NewReader(pr)
_, err := bufReader.Peek(1)
if err != nil {
    pr.CloseWithError(err)
    writer.WriteHeader(http.StatusInternalServerError)
    writer.Write([]byte("This is error"))
    return
}

writer.WriteHeader(http.StatusOK)
io.Copy(responseWriter, bufReader)

That works, but looks rather hacky to me and I'm not also sure about all the bufio corner cases.

Is there some better/simpler way to achieve this ?

Note: The above examples are simplified a bit. The actual implementation uses GIN framework and instead of io.Copy the ctx.DataFromReader() is used to transfer data.

Paperclip
  • 615
  • 5
  • 14
  • 1
    You can't start writing the body until the http header has been sent, that's how the http protocol works. If you want to ensure that there are no errors from another read operation, then you must wait and buffer the entire operation. – JimB Jun 22 '23 at 16:17
  • Not sure what you mean before headers. In my examples I'm sending the headers. Also responseWriter calls WriteHeader() before Write if necessary. I agree that in the middle of transfer there's nothing to do but I'm more interested about the initialization phase - (eg file is not found, some network drop, token expired etc..) (Due to the storage solution and performance reasons I' cant do a pre-flight to the storage.) I might get away with some validation to be extra careful, but still. .what if fetchFile imeediatly returns error, then OK has already been sent anyway.. – Paperclip Jun 22 '23 at 16:23
  • 1
    You should change the `fetchFile` function. The way it looks, it either fails immediately, or writes the whole file to the passed in writer. Instead, make it return an (io.Reader, error) – Burak Serdar Jun 22 '23 at 16:26
  • `HTTP/1.1 200` (or `HTTP/2 200` etc) is the response header, and it must come first in the response. In order to call `Write` on the response that header (plus any others you have set) must already be sent to the client, which means you can never change the response code after the `Write` call. If you only want to check for an error from the first read, then using the buffered reader is fine. If you can check for errors from `fetchFile`, then you would not need to rely on passing them through the pipe at all. – JimB Jun 22 '23 at 16:26
  • @JimB, I don't want to change the status after write. I want be sure the file retrieval has sucessfully started before starting to stream the OK response. But I may need to look into the option provided by Burak – Paperclip Jun 22 '23 at 16:30
  • I know you don't _want_ to change the status after write, but that is what you are trying to do and why it's failing ;). Yes, the key is to do as much as possible before calling Write, either by restructuring `fetchFile` or buffering the reads. – JimB Jun 22 '23 at 16:33
  • Its not failing. It's just confusing when you expect a JPEG, but get an empty file instead because there was error, or the JPEG containing a JSON error. I want to write a API where people wont tear their hair out :D – Paperclip Jun 22 '23 at 16:35
  • @BurakSerdar Thanks. I guess I have been looking at it for too long to realize a better pattern. I think I wanted to avoid the caller being worried about Close(). Used io.ReadCloser to imply that it needs to be closed. I guess it's relatively common (e.g client response body). – Paperclip Jun 22 '23 at 16:52

0 Answers0