Sending chunked data a grpc ServerStream

Question

I have this RPC defined

rpc Download(DownloadRequest) returns (stream google.api.HttpBody)

I'd eventually like to use this to serve both CSV and binary data, but I'm starting with CSV.

In my handler I am doing the following to repeatedly grab the data from storage and send it to the stream:

if err := stream.SetHeader(metadata.Pairs("content-disposition", "attachment")); err != nil {
    return err
}
chunkSize := uint64(100) // arbitrary chunk size
for offset := uint64(0); offset < objLen; offset += chunkSize {
    contents, err := getContent(offset, chunkSize)
    if err != nil {
        return err
    }
    err = stream.Send(&httpbody.HttpBody{
        ContentType: "text/csv",
        Data:        contents,
    })
    if err != nil {
        return err
    }
}

The problem with this implementation is that the CSV result ends up with line breaks every chunkSize characters. How can I avoid the newline after every chunk in my output?

I am using grpc-gateway to serve this via REST, in case that matters.

I believe its because grpc-gateway, as mentioned in [docs](https://grpc-ecosystem.github.io/grpc-gateway/docs/mapping/customizing_your_gateway/#stream-error-handler) > When the method has a streaming response, gRPC-Gateway handles that by emitting a newline-separated stream of “chunks”. — medasx, Jul 15 '22 at 08:48

wowzaman · Answer 1 · 2023-06-17T02:29:55.057

When the method has a streaming response, gRPC-Gateway handles that by emitting a newline-separated stream of “chunks”. as mentioned in docs.

you can use your custom marshaler for specific content-type (e.g. 'text/csv') to change delimiter (by default is []byte('\n'))

package marshaler

import (
    "github.com/grpc-ecosystem/grpc-gateway/v2/runtime"
)

type marshaler struct {
    mime      string
    delimiter []byte
    runtime.Marshaler
}

func NewMarshaler(mime string, delimiter []byte) runtime.Marshaler {
    return &marshaler{mime: mime, delimiter: delimiter}
}

func (m *marshaler) Delimiter() []byte {
    return m.delimiter
}

func (m *marshaler) ContentType(v interface{}) string {
    return m.mime
}

then in new mux server (gateway)

mux := runtime.NewServeMux(
        ....

        // allow chunked stream responses to work with "Content-Type: text/csv" requests
        runtime.WithMarshalerOption("text/csv", marshaler.NewMarshaler("text/csv", nil)),
    )

and send request with "Content-Type: text/csv"

curl --location 'https://your-api-gateway/download' \
--header 'Content-Type: text/csv'

nothing to do with your rpc server

csvbyte := getCSV()
chunkSize := 1024 * 1024 * 1 // 1MB
    for i := 0; i < len(csvbyte); i += chunkSize {
        end := i + chunkSize
        if end > len(csvbyte) {
            end = len(csvbyte)
        }
        if err := stream.Send(&httpbody.HttpBody{
            ContentType: "text/csv",
            Data:        csvbyte[i:end],
        }); err != nil {
            return err
        }
    }

Sending chunked data a grpc ServerStream

1 Answers1