How to read file content from an http.Request's response

Question

I use below code to send a request to an http server.

The server sends a response that contains those http headers

Content-Disposition:[attachment;filename=somefilename.csv] 
Content-Type:[text/csv; charset=UTF-8]

How do i proceed to retrieve the content of the file attached with the response ?

baseUrl := "Some url that i call to fetch csv file"

client := http.Client{}

resp, _ := client.Get(baseUrl)
defer resp.Body.Close()

fmt.Println(resp)

// &{200 OK 200 HTTP/2.0 2 0 map[Content-Disposition:[attachment;filename=somefilename.csv] Content-Type:[text/csv; charset=UTF-8] Date:[Mon, 30 Sep 2019 09:54:08 GMT] Server:[Jetty(9.2.z-SNAPSHOT)] Vary:[Accept]] {0xc000530280} -1 [] false false map[] 0xc000156200 0xc0000c26e0}

he does not have a clue what he is doing. Unless it can be marked as duplicate, please help to improve the question / response instead of throwing close votes. — , Sep 30 '19 at 10:20

score 6 · Accepted Answer · 2019-09-30T11:07:34.883

you have to consume the body of the request.

baseUrl := "Some url that i call to fetch csv file"

client := http.Client{}

resp, _ := client.Get(baseUrl)
defer resp.Body.Close()
io.Copy(os.Stdout, resp.Body) // this line.

fmt.Println(resp)

if you have to deal with a multipart form data https://golang.org/pkg/net/http/#Request.FormFile

Given following comment,

i see now after printing resp that there is a csv text but type is http.Response i have to deal with golang.org/pkg/encoding/csv/#Reader how to turn resp to string in order to be able reader to read it, or i miss something else ?

OP has to understand that an http response Body implements the io.Reader interface. When the http response come back from the server, the body is not read directly into memory as slice of bytes []byte.

OP should note also that a csv.Reader is an implementation to decode a CSV encoded content that consumes an io.Reader. Under the hood it does not hold the entire content of the file in memory, it reads whats needed to decode one line and proceed further.

As a consequence to those two important properties of golang implementation, is that it is easy and natural to connect the response body reader to the csv reader.

To the question, what is an io.Reader, OP has to figure out it is anything capable of reading a byte stream by chunks of max len p. This is illustrated by the signature of the unique method of this interface Read([]byte) (int, error)

This interface is designed in such way that it minimizes consumed memory and allocations.

ref links

All that being said, the final code is trivially written,

package main

import (
    "encoding/csv"
    "fmt"
    "io"
    "log"
    "net/http"
)

func main() {
    baseUrl := "https://geolite.maxmind.com/download/geoip/misc/region_codes.csv"

    client := http.Client{}

    resp, err := client.Get(baseUrl)
    if err != nil {
        log.Fatal(err)
    }
    defer resp.Body.Close()
    fmt.Println(resp)

    r := csv.NewReader(resp.Body)

    for {
        record, err := r.Read()
        if err == io.EOF {
            break
        }
        if err != nil {
            log.Fatal(err)
        }

        fmt.Println(record)
    }
}

i see now after printing resp that there is a csv text but type is http.Response i have to deal with https://golang.org/pkg/encoding/csv/#Reader how to turn resp to string in order to be able reader to read it, or i miss something else ? — pshx, Sep 30 '19 at 10:35

score 1 · Answer 2 · answered Sep 30 '19 at 10:36

The easiest way is to use ioutil.ReadAll method to read the entire body. The body implements the io.Reader interface, so everything you can do with a reader can be done with the body. That includes piping it to stdout as mh-cbon has shown in his answer.

baseUrl := "Some url that i call to fetch csv file"

resp, err := http.Get(baseUrl)
if err != nil {
    // do something with the error
}
defer resp.Body.Close()

content, err := ioutil.ReadAll(resp.Body)
if err != nil {
    // do something with the error
}

fmt.Println(string(content))

Note that I removed the client. It is not needed for simple GET calls. If you need more configuration use the http.Client as you did in your example.

Also note that content is of type []byte but it can easily be converted to string as done above in the print statement.

i was unhappy you proposed a string conversion so i updated my answer. — , Sep 30 '19 at 11:08
Yes, very nice! I tried to give the simplest answer but also point out that the body is a `io.Reader`. Reading the body at once is a bad idea with large files and streaming it is way superior... but a lot more complex -- especially for beginners. — TehSphinX, Sep 30 '19 at 11:24

How to read file content from an http.Request's response

2 Answers2