Golang Channel Streams

Question

I am relatively new to Golang and don't fully understand streams. I have a function (circuit breaker function) that is making Rest calls. I have it working but it is only streaming the "responseBody" back. I would actually like to stream back the entire request of stream back both the Body and the Header together.

When I try to use a similar approach on the "header" then I get an error that the header is not streamable.

Is there a good/best way to accomplish this? Thank you. Below is my function.

func CallWithRetries(req *http.Request, output chan []byte) error {
    r := retrier.New(retrier.ConstantBackoff(RETRIES, 100 * time.Millisecond), nil)
    attempt := 0
    err := r.Run(func() error {
        attempt++
        resp, err := Client.Do(req)
        if err == nil && resp.StatusCode < 299 {
            responseBody, err := ioutil.ReadAll(resp.Body)
            if err == nil {
                output <- responseBody
                return err
            }
            return err
        } else if err == nil {
            customLogger.LogDebug("Status code was: " , transactionId)
            err = fmt.Errorf("Status was %v", resp.StatusCode)
        }
        return err
    })
    return err
}

Because your channel expects values of type `[]byte`, and the HTTP headers is of type `[][]string`. Find a way to serialize your headers to `[]byte`. — icza, Aug 27 '18 at 15:05
`output` is a `chan []byte` in your example, and a response's headers aren't stored in a `[]byte`, which is why you can't send it through your channel. Simply convert them to a `[]byte` format (by for example marshalling them into a JSON object) and you'll be able to send them. — Ullaakut, Aug 27 '18 at 15:05
Just for reference in your future searches, this isn't commonly referred to as streaming. Streaming typically refers to processing (reading or writing) a data stream, in Go typically represented as a `io.Reader`/`io.Writer`. Use of channels is just standard Go concurrency. — Adrian, Aug 27 '18 at 15:14

score 2 · Accepted Answer · answered Aug 27 '18 at 17:17

You are looking for the httputil.DumpResponse function.

The code might be changed to something similar to

func CallWithRetries(req *http.Request, output chan []byte) error {
    r := retrier.New(retrier.ConstantBackoff(RETRIES, 100*time.Millisecond), nil)
    attempt := 0
    err := r.Run(func() error {
        attempt++
        resp, err := Client.Do(req)
        if err == nil && resp.StatusCode < 299 {
            dump, err := httputil.DumpResponse(resp, true)
            if err == nil {
                output <- dump
                return err
            }
            return err
        } else if err == nil {
            customLogger.LogDebug("Status code was: ", transactionId)
            err = fmt.Errorf("Status was %v", resp.StatusCode)
        }
        return err
    })
    return err
}

Side notes,

you might want to consider to close the response body as mentioned in the documentation https://golang.org/pkg/net/http/#Client.Get
Looks likes the err variable is shadowed, this should be modified to avoid any surprises.

This version of the code attempts to return errors early, and to close the response body. It was not tested, only written on the fly, to use with care.

func CallWithRetries(req *http.Request, output chan []byte) error {
    r := retrier.New(retrier.ConstantBackoff(RETRIES, 100*time.Millisecond), nil)
    attempt := 0
    return r.Run(func() error {
        attempt++
        var resp *http.Response
        {
            r, err := Client.Do(req)
            if err != nil {
                return err
            }
            defer r.Body.Close()
            if resp.StatusCode > 299 {
                customLogger.LogDebug("Status code was: ", transactionId)
                return fmt.Errorf("Status was %v", resp.StatusCode)
            }
            resp = r
        }
        var out []byte
        {
            x, err := httputil.DumpResponse(resp, true)
            if err != nil {
                return err
            }
            out = x
        }
        output <- out
        return nil
    })
}

Thank you @mh-cbon - your answer was very helpful and clear. This is exactly what I needed. I really appreciate it (and the extra assistance you provided). — mornindew, Aug 28 '18 at 15:14

Golang Channel Streams

1 Answers1