Range over elements of a protobuf array in Go

Question

I have a Protobuf structure defined as so in my .proto file:

message Msg{
  message SubMsg {
    string SubVariable1 = 1;
    int32 SubVariable2 = 2;
    ...
  }
  string Variable1 = 1;
  repeated SubMsg Variable2 = 2;
  ...
}

I pull data into this structure using the https://godoc.org/google.golang.org/protobuf/encoding/protojson package when consuming data from a JSON API, as so:

Response, err := Client.Do(Request)
if err != nil {
    log.Error(err)
}
DataByte, err := ioutil.ReadAll(Response.Body)
if err != nil {
    log.Error(err)
}
DataProto := Msg{}
err = protojson.Unmarshal(DataByte, &DataProto)
if err != nil {
    log.Error(err)
}

What I want to be able to do is to range over the elements of Variable2 to be able to access the SubVariables using the protoreflect API, for which I have tried both:

Array := DataProto.GetVariable2()
for i := range Array {
  Element := Array[i]
}

and also:

DataProto.GetVariable2().ProtoReflect().Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) {
      …
      return true})

The first of which fails with error message:

cannot range over DataProto.GetVariable2() (type *SubMsg)

despite the fact DataProto.GetVariable2() returns a variable of type []*Msg_SubMsg.

The second of which fails with:

DataProto.GetVariable2.ProtoReflect undefined (type []*SubMsg has no field or method ProtoReflect)

which suggests that DataProto.GetVariable2() does indeed return an array unlike what is suggested in the error returned in my first approach. This makes sense to me as the protoreflect API only allows this method to be called on a defined message, not an array of those messages. There therefore must be another way of accessing the elements of these arrays to be able to make use of the protoreflect API (for which I have been unsuccessful in finding and answer to on the web thus far).

Could someone help me make sense of these seemingly conflicting error messages? Has anyone had any success iterating over a Protobuf array themselves?

Thanks in advance.

Is there a compelling reason to use the `protoreflect` package? The `Range` method is used to iterate over all the *fields* of a message struct - it's not meant for a single field that happens to be a (`repeated`) slice. — colm.anseo, Jul 24 '20 at 12:36
I have maybe misinterpreted what the protoreflect Range function brings to the table. I thought given I was struggling to iterate over an array that it would be my silver bullet, as it gave the ability to access the protoreflect.Value in a way that calling DataProto.Variable2 would not. Even approaching it without the protoreflect package (trying to range over DataProto.Variable2) gives the same error. Are you aware of a way I can iterate over a protobuf array of unknown length? — Michael Pyle, Jul 24 '20 at 12:44
You seem to be doing other things with your code. Please include a [minimal, reproducible example](https://stackoverflow.com/help/minimal-reproducible-example). At the very least, show something that can be run, including the generated protobuf code. — Marc, Jul 24 '20 at 13:14

score 3 · Answer 1 · answered Oct 06 '20 at 17:01

You'll want to treat your Array variable as a List, which means you can't use Range() as in your second attempt. It's close though. Here is a functional example of iterating through and inspecting nested messages:


import (
    "testing"

    "google.golang.org/protobuf/reflect/protoreflect"
)

func TestVariable2(t *testing.T) {
    pb := &Msg{
        Variable2: []*Msg_SubMsg{
            {
                SubVariable1: "string",
                SubVariable2: 1,
            },
        },
    }

    pbreflect := pb.ProtoReflect()
    fd := pbreflect.Descriptor().Fields().ByJSONName("Variable2")
    if !fd.IsList() {
        t.Fatal("expected a list")
    }
    l := pbreflect.Get(fd).List()
    for i := 0; i < l.Len(); i++ {
        // should test that we are now inspecting a message type
        li := l.Get(i).Message()
        li.Range(func(lifd protoreflect.FieldDescriptor, liv protoreflect.Value) bool {
            t.Logf("%v: %v", lifd.Name(), liv)
            return true
        })
    }
}

Run with go test -v ./... if you want to see output

Range over elements of a protobuf array in Go

1 Answers1