1

We want to unmarshal (in golang) a GRPC message and transform it into a map[string]interface{} to further process it. After using this code:

err := ptypes.UnmarshalAny(resource, config)
configMarshal, err := json.Marshal(config)
var configInterface map[string]interface{}
err = json.Unmarshal(configMarshal, &configInterface)

we get the following structure:

{
    "name": "envoy.filters.network.tcp_proxy",
    "ConfigType": {
        "TypedConfig": {
            "type_url": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
            "value": "ChBCbGFja0hvbGVDbHVzdGVyEhBCbGFja0hvbGVDbHVzdGVy"
        }
    }
}

Where the TypedConfig field remains encoded. How can we decode the TypedConfig field? We know the type_url and we know the value, but to unmarshal the field, it needs to be of the pbany.Any type. But because the TypedConfig structure is a map[string] interface {}, our program either fails to compile, or it crashes, complaining that it is expecting a pbany.Any type, but instead it is getting a map[string] interface {}.

We have the following questions:

  1. Is there a way to turn the structure under TypedConfig into a pbany.Any type that can be subsequently unmarshalled?
  2. Is there a way to recursively unmarshal the entire GRPC message?

Edit (provide more information about the code, schemas/packages used)

We are looking at the code of xds_proxy.go here: https://github.com/istio/istio/blob/master/pkg/istio-agent/xds_proxy.go

This code uses a *discovery.DiscoveryResponse structure in this function:

func forwardToEnvoy(con *ProxyConnection, resp *discovery.DiscoveryResponse) {

The protobuf schema for discovery.DiscoveryResponse (and every other structure used in the code) is in the https://github.com/envoyproxy/go-control-plane/ repository in this file: https://github.com/envoyproxy/go-control-plane/blob/main/envoy/service/discovery/v3/discovery.pb.go

We added code to the forwardToEnvoy function to see the entire unmarshalled contents of the *discovery.DiscoveryResponse structure:

var config proto.Message
switch resp.TypeUrl {
case "type.googleapis.com/envoy.config.route.v3.RouteConfiguration":
    config = &route.RouteConfiguration{}
case "type.googleapis.com/envoy.config.listener.v3.Listener":
    config = &listener.Listener{}
// Six more cases here, truncated to save space
}

for _, resource := range resp.Resources {
    err := ptypes.UnmarshalAny(resource, config)
    if err != nil {
        proxyLog.Infof("UnmarshalAny err %v", err)
        return false
    }
    configMarshal, err := json.Marshal(config)
    if err != nil {
        proxyLog.Infof("Marshal err %v", err)
        return false
    }

    var configInterface map[string]interface{}
    err = json.Unmarshal(configMarshal, &configInterface)
    if err != nil {
        proxyLog.Infof("Unmarshal err %v", err)
        return false
    }
}

And this works well, except that now we have these TypedConfig fields that are still encoded:

{
    "name": "virtualOutbound",
    "address": {
        "Address": {
            "SocketAddress": {
                "address": "0.0.0.0",
                "PortSpecifier": {
                    "PortValue": 15001
                }
            }
        }
    },
    "filter_chains": [
        {
            "filter_chain_match": {
            "destination_port": {
                "value": 15001
            }
        },
        "filters": [
            {
                "name": "istio.stats",
                "ConfigType": {
                    "TypedConfig": {
                        "type_url": "type.googleapis.com/udpa.type.v1.TypedStruct",
                        "value": "CkF0eXBlLmdvb2dsZWFwaXMuY29tL2Vudm95LmV4dGVuc2lvbnMuZmlsdG"
                     }
                }
             },

One way to visualize the contents of the TypedConfig fields is to use this code:

for index1, filterChain := range listenerConfig.FilterChains {
    for index2, filter := range filterChain.Filters {
        proxyLog.Infof("Listener %d: Handling filter chain %d, filter %d", i, index1, index2)
        switch filter.ConfigType.(type) {
            case *listener.Filter_TypedConfig:
            proxyLog.Infof("Found TypedConfig")
            typedConfig := filter.GetTypedConfig()
            proxyLog.Infof("typedConfig.TypeUrl = %s", typedConfig.TypeUrl)
            switch typedConfig.TypeUrl {
                case "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy":
                    tcpProxyConfig := &tcp_proxy.TcpProxy{}
                    err := typedConfig.UnmarshalTo(tcpProxyConfig)
                    if err != nil {
                        proxyLog.Errorf("Failed to unmarshal TCP proxy configuration")
                    } else {
                        proxyLog.Infof("TcpProxy Config for filter chain %d filter %d: %s", index1, index2, prettyPrint(tcpProxyConfig))
                    }

But then the code becomes very complex, as we have a large number of structures, and these structures can occur in different order in the messages. So we wanted to get a generic way of unmarshalling these TypedConfig message by using pbAny, and hence our questions.

c m adam
  • 11
  • 3
  • I would ask this question by filing an issue in the protobuf repo: https://github.com/golang/protobuf/issues/new?assignees=&labels=&template=question.md – Doug Fawley Sep 22 '21 at 17:46

0 Answers0