How to calculate sha256 file checksum in Go

Question

I need utility for Windows that calculates sha256 file checksum so that when I download fedora I can verify checksum from here: https://fedoraproject.org/static/checksums/Fedora-18-i386-CHECKSUM

Microsoft utility from http://support.microsoft.com/kb/889768 does only md5 and sha1.

I don't want to use other downloadable tools that are not signed and not available from https or from sources that I don't know about, because it does not make any sense to download unsigned code over unencrypted connection or from untrusted source to verify signature of another code to trust it.

Luckily google provides possibility to use https for all downloads so I can download Go over secure connection and start from there.

Here is simple code that does that for a small file, but it's not very good for big files because it's not streaming.

package main

import (
    "io/ioutil"
    "crypto/sha256"
    "os"
    "log"
    "encoding/hex"
)

func main() {
    hasher := sha256.New()
    s, err := ioutil.ReadFile(os.Args[1])    
    hasher.Write(s)
    if err != nil {
        log.Fatal(err)
    }

    os.Stdout.WriteString(hex.EncodeToString(hasher.Sum(nil)))
}

How to make it to use streams so that it works on any file size.

Possible duplicate of [Generating the SHA hash of a string using golang](https://stackoverflow.com/questions/10701874/generating-the-sha-hash-of-a-string-using-golang) — 030, May 13 '19 at 19:15

score 43 · Answer 1 · answered Jul 16 '17 at 00:48

43

The crypto/sha256 godoc actually has a snippet that shows how to do that (it's basically the same code as James):

package main

import (
  "crypto/sha256"
  "fmt"
  "io"
  "log"
  "os"
)

func main() {
  f, err := os.Open("file.txt")
  if err != nil {
    log.Fatal(err)
  }
  defer f.Close()

  h := sha256.New()
  if _, err := io.Copy(h, f); err != nil {
    log.Fatal(err)
  }

  fmt.Printf("%x", h.Sum(nil))
}

answered Jul 16 '17 at 00:48

3lil636wm

7,580
3
24
26

3

How does *io.Copy* work? Is the file read entirely in memory or is it processed by chunks... of size? – Massimo Oct 28 '19 at 16:28
6

It's processed by chunks of 32KB: https://golang.org/src/io/io.go#L380 You can change that by providing your own buffer using CopyBuffer – 3lil636wm Nov 01 '19 at 22:07
1

should I generate the sha256 hash only once as a global var or is it needed a new one before each comparison? – tuxErrante Nov 23 '22 at 10:24

score 37 · Accepted Answer · answered Apr 09 '13 at 03:20

37

The SHA256 hasher implements the io.Writer interface, so one option would be to use the io.Copy() function to copy the data from an appropriate io.Reader in blocks. Something like this should do:

f, err := os.Open(os.Args[1])
if err != nil {
    log.Fatal(err)
}
defer f.Close()
if _, err := io.Copy(hasher, f); err != nil {
    log.Fatal(err)
}

answered Apr 09 '13 at 03:20

James Henstridge

42,244
6
132
114

1

do I need to do `sha256.New()` if I'm doing this in a loop? – majidarif Mar 19 '19 at 14:56
2

@majidarif: you can reuse the hasher by calling its `Reset` method. If you don't reset and start calculating the hash of a second file, you'll instead be calculating the hash of the two files concatenated. With that said, your code will probably be more readable if you don't try to reuse the hashers. – James Henstridge Mar 20 '19 at 03:13
1

How does *io.Copy* work? Is the file read entirely in memory or is it processed by chunks... of size? – Massimo Oct 28 '19 at 16:28
2

@Massimo: in the generic case, it will use a 32 kB buffer to copy data in chunks. As the documentation states, it will defer to the `src.WriteTo` or `dest.ReadFrom` methods if they exist. Those may have different buffering behaviours, but shouldn't read the entire file. – James Henstridge Nov 01 '19 at 11:07
there is also an entire library dedicated to this, even if it seems an overkill to me https://pkg.go.dev/github.com/udhos/equalfile#pkg-functions – tuxErrante Nov 23 '22 at 13:23

score 10 · Answer 3 · edited Nov 07 '20 at 22:06

Full example of md5sum:

func md5sum(filePath string) (string, error) {
    file, err := os.Open(filePath)
    if err != nil {
        return "", err
    }
    defer file.Close()

    hash := md5.New()
    if _, err := io.Copy(hash, file); err != nil {
        return "", err
    }
    return hex.EncodeToString(hash.Sum(nil)), nil
}

EncodeToString does not omits leading 0 bytes, so fmt.Println(hex.EncodeToString([]byte{0x00, 0x00, 0xA, 0xB, 0xC})) gives 00000a0b0c

How to calculate sha256 file checksum in Go

3 Answers3