3

I'm in the process of writing a utility in Go which can be used to calculate CRC32 checksum of an input string. A similar utility exists in Java which we are using extensively and has been working well for us.

The Java Utility uses java.util.zip.CRC32 to calculate the checksum. Pseudo code is as follows : 

 public static void main(String[] args) {
        final Checksum checksum = new CRC32();
        byte[] input1Bytes = "input1".getBytes(StandardCharsets.UTF_8);
        checksum.update(input1Bytes, 0, input1Bytes.length);
        final byte[] input2Bytes = "input2".getBytes(StandardCharsets.UTF_8);
        checksum.update(input2Bytes, 0, input2Bytes.length);
        final byte[] input3Bytes = "input3".getBytes(StandardCharsets.UTF_8);
        checksum.update(input3Bytes, 0, input3Bytes.length);
        System.out.println("Checksum in Java : " + checksum.getValue());
    }

The utility in Go uses the crc32 from the hash package of the Go SDK (version 1.13.6) (import "hash/crc32") Pseudo code for generating the checksum in Go is as follows :

    table := crc32.MakeTable(0)
    checksum := crc32.Checksum([]byte("input1"), table)
    checksum = crc32.Update(checksum, table, []byte("input2"))
    checksum = crc32.Update(checksum, table, []byte("input3"))
    log.Printf("Checksum in go : %v", checksum)

The output from the Java code is :

Checksum in Java : 2705255531

The output from the Go code is :

Checksum in go : 4294967295

I've also compared the byte arrays being generated in the 3 steps and they have the same values.

What am i missing and how do I ensure this mismatch is handled?

Appreciate the help!

saarthak gupta
  • 173
  • 1
  • 11
  • Is not 4294967295 = 0xFFFF_FFFF? Then golang seems off. – Joop Eggen Jan 20 '20 at 09:32
  • 4
    The call `crc32.MakeTable(0)` is clearly wrong: you're setting the polynomial to zero. Which polynomial did you want? (Zero is never right.) See also https://stackoverflow.com/q/9004016/1256452 – torek Jan 20 '20 at 09:32
  • Thanks a lot @JoopEggen and @torek for your response. You're right @torek, `crc32.MakeTable(0)` is the issue. I changed it to `table := crc32.IEEETable` and it seems to be working. Will be running tests on this to ensure this covers all the aspects. – saarthak gupta Jan 20 '20 at 09:35
  • 1
    Yes, that looks like it is the right one. Note that the `hash/crc32` code uses a little-endian (or bit-reversed) polynomial, hence `0x04C11DB7` is not right. Bit-reversing `0x04C11DB7`, however, produces `0xedb88320` which is `crc32.IEEE`. – torek Jan 20 '20 at 09:44

1 Answers1

5

Use the same polynomial as your Java code, and you get the same results.

The most common CRC-32 polynomial is there in the crc32 package as a constant: crc32.IEEE = 0xedb88320:

table := crc32.MakeTable(crc32.IEEE)

With this change you get the same result. Try it on the Go Playground.

icza
  • 389,944
  • 63
  • 907
  • 827