How can I use gzip on a string?

Question

I want to use Go to read out a chunk from a file, treat it as a string and gzip this chunk. I know how to read from the file and treat it as a string, but when it comes to compress/gzip I am lost.

Should I create an io.writer, which writes to a buf (byte slice), use gzip.NewWriter(io.writer) to get a w *gzip.Writer and then use w.Write(chunk_of_file) to write the chunk of file to buf? Then I would need to treat the string as a byte slice.

possible duplicate of [How can I use golang's "compress/gzip" package to gzip a file?](http://stackoverflow.com/questions/16890648/how-can-i-use-golangs-compress-gzip-package-to-gzip-a-file) — Erik Kaplun, Oct 05 '13 at 12:50
I'm confused by your question, in Go a string *is* a []byte so if you want it to be one, just convert it as such []byte(mystringvar) — mzimmerman, Oct 08 '13 at 15:30

score 37 · Accepted Answer · edited Aug 12 '19 at 12:16

37

You can just write using gzip.Writer as it implements io.Writer.

Example:

package main

import (
    "bytes"
    "compress/gzip"
    "fmt"
    "log"
)

func main() {
    var b bytes.Buffer
    gz := gzip.NewWriter(&b)
    if _, err := gz.Write([]byte("YourDataHere")); err != nil {
        log.Fatal(err)
    }
    if err := gz.Close(); err != nil {
        log.Fatal(err)
    }
    fmt.Println(b.Bytes())
}

Go Playground

If you want to set the compression level (Default is -1 from compress/flate) you can use gzip.NewWriterLevel.

edited Aug 12 '19 at 12:16

Dave C

7,729
4
49
65

answered Oct 09 '13 at 08:51

Intermernet

18,604
4
49
61

1

I feel like it should be noted that the Writer will not actually write the compressed bytes to the buffer until `gz.Flush()` is called. The flush is also called with `gz.Close()`, but the playground example never gets that far. – Nenoco Jan 18 '15 at 22:54
3

One more thing is that Flush() only writes the _current_ data to the buffer. It doesn't finish off the whole GZIP format. So, in this case, it's pretty useless, since what's written on the last line is not a valid GZIP structure. You need to call Close() _before_ you do anything with the buffer. – Phil Kulak Jan 29 '15 at 19:47
1

@PhilKulak Thanks for the correction. I've changed the example code to close the writer before printing the buffer contents. – Intermernet Jan 29 '15 at 23:23
1

You can just call Close(). From the golang documentation: func (z *Writer) Close() error Close closes the Writer, _flushing any unwritten data to the underlying io.Writer_, but does not close the underlying io.Writer. https://golang.org/pkg/compress/gzip/#Writer.Close – Aaron Oct 20 '15 at 00:15
2

Excellent tip @PhilKulak. Just to add a bit, using _defer_ to close the compressed Writer can lead to subtle bugs where the buffer is being used before being closed. This can result in unexpected EOF errors when reading the compressed data. Watch out! – Dec 03 '16 at 05:30

Zombo · Answer 2 · 2021-10-21T18:21:57.063

If the result is not going back into a file, then you could just use Flate directly. You save a bit of overhead from Gzip. Another option is Brotli. Examples:

package main

import (
   "bytes"
   "compress/flate"
   "github.com/andybalholm/brotli"
)

func compressFlate(data []byte) ([]byte, error) {
   var b bytes.Buffer
   w, err := flate.NewWriter(&b, 9)
   if err != nil {
      return nil, err
   }
   w.Write(data)
   w.Close()
   return b.Bytes(), nil
}

func compressBrotli(data []byte) []byte {
   var b bytes.Buffer
   w := brotli.NewWriterLevel(&b, brotli.BestCompression)
   w.Write(data)
   w.Close()
   return b.Bytes()
}

Result:

package main

import (
   "bytes"
   "fmt"
)

func main() {
   data := bytes.Repeat([]byte("hello world"), 999_999)
   f, err := compressFlate(data)
   if err != nil {
      panic(err)
   }
   b := compressBrotli(data)
   fmt.Println(len(f) == 21379, len(b) == 40)
}

How can I use gzip on a string?

2 Answers2