I am trying to debug connectivity issues with Thanos Sidecar.
I have Prometheus & Sidecar set up on a rack. On another rack I have Thanos Query to query the remote Sidecar. Actually I have about 30 remote racks. The query works to some racks some times and they come and go from the list of available query endpoints. Often they seem to timeout eg. context deadline exceeded.
I am able to use grpcurl <rack-url>:10901 grpc.health.v1.Health.Check
to check the endpoints and I get status SERVING. However I can't figure out how to get more data. grpcurl <rack-url>:10901 list
returns
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
thanos.Exemplars
thanos.Metadata
thanos.Rules
thanos.Store
thanos.Targets
thanos.info.Info
However if I try to list more on any of the thanos categories, I get Symbol not found
grpcurl <rack-url>:10901 list thanos.info.Info
Failed to list methods for service "thanos.info.Info": Symbol not found: thanos.info.Info
My question is, What grpcurl call can I do that replicates what Thanos Query is sending when Query is trying to get information to add the Sidecar to the list of available endpoints?
Thanks
Also any good resources on how grpc works with Thanos/Prometheus queries would be appreciated.