Hash protobuf object to a string as the key for redis database

Question

I have some sort of complex protobuf object. It's a request sent to my GRPC endpoint. I want to just process it if I didn't before. So I want to hash the object to some string and store it in my Redis database. I used ObjectHash-Proto but with new versions of protobuf-compiler I got this error:

got an unexpected struct of type 'impl.MessageState' for field {Name:state PkgPath:... Type:impl.MessageState Tag: Offset:0 Index:[0] Anonymous:false}

Seems it does not support structs and new version of protobuf-compiler generates a code containing struct.

I can't generate some sort of ID for each request. The ID is actually the hash of the whole of the object.

Answer 1

If you have a proto.Message [1], then you get a Marshal function [2] for free. So after you Marshal the message, you can just pass the bytes to base64, or md5 or whatever you want:

package main

import (
   "encoding/base64"
   "google.golang.org/protobuf/proto"
   "google.golang.org/protobuf/types/known/structpb"
)

func hash(m proto.Message) (string, error) {
   b, err := proto.Marshal(m)
   if err != nil {
      return "", err
   }
   return base64.StdEncoding.EncodeToString(b), nil
}

func main() {
   m, err := structpb.NewStruct(map[string]interface{}{
      "month": 12, "day": 31,
   })
   if err != nil {
      panic(err)
   }
   s, err := hash(m)
   if err != nil {
      panic(err)
   }
   println(s) // ChIKBW1vbnRoEgkRAAAAAAAAKEAKEAoDZGF5EgkRAAAAAAAAP0A=
}

1. https://godocs.io/google.golang.org/protobuf/proto#Message
2. https://godocs.io/google.golang.org/protobuf/proto#Marshal

Answer 2

Proto serialization is not stable, thus you can't rely on marshalling and hashing the output to produce the same hash for the same message.

From https://developers.google.com/protocol-buffers/docs/reference/go/faq#hash

How do I use a protocol buffer message as a hash key?

You need canonical serialization, where the marshaled output of a protocol buffer message is guaranteed to be stable over time. Unfortunately, no specification for canonical serialization exists at this time. You'll need to write your own or find a way to avoid needing one.

The closest solution I could find is deepmind objecthash-proto but there have been no contributions in last 4 years so I assume it might be outdated