How to debug opencensus metrics agent?

Question

I am having a hard time debugging why my metrics don't reach my prometheus. I am trying to send simple metrics to oc agent and from ther pull them with prometheus, but my code does not fail and I can't see the metrics in prometheus. So, how can I debug it? Can I see if a metric arrived in the oc agent?

cAgentMetricsExporter.createAndRegister(
            OcAgentMetricsExporterConfiguration.builder()
                    .setEndPoint("IP:55678")
                    .setServiceName("my-service-name")
                    .setUseInsecure(true)
                    .build());

    Aggregation latencyDistribution = Aggregation.Distribution.create(BucketBoundaries.create(
            Arrays.asList(
                    0.0, 25.0, 100.0, 200.0, 400.0, 800.0, 10000.0)));


    final Measure.MeasureDouble M_LATENCY_MS = Measure.MeasureDouble.create("latency", "The latency in milliseconds", "ms");
    final Measure.MeasureLong M_LINES = Measure.MeasureLong.create("lines_in", "The number of lines processed", "1");
    final Measure.MeasureLong M_BYTES_IN = Measure.MeasureLong.create("bytes_in", "The number of bytes received", "By");

    View[] views = new View[]{
            View.create(View.Name.create("myapp/latency"),
                    "The distribution of the latencies",
                    M_LATENCY_MS,
                    latencyDistribution,
                    Collections.emptyList()),

            View.create(View.Name.create("myapp/lines_in"),
                    "The number of lines that were received",
                    M_LATENCY_MS,
                    Aggregation.Count.create(),
                    Collections.emptyList()),
    };

    // Ensure that they are registered so
    // that measurements won't be dropped.
    ViewManager manager = Stats.getViewManager();
    for (View view : views)
        manager.registerView(view);


    final StatsRecorder STATS_RECORDER = Stats.getStatsRecorder();

    STATS_RECORDER.newMeasureMap()
            .put(M_LATENCY_MS, 17.0)
            .put(M_LINES, 238)
            .put(M_BYTES_IN, 7000)
            .record();

score 1 · Answer 1 · answered Oct 10 '19 at 17:23

A good question; I've also struggled with this.

The Agent should provide better logging.

A Prometheus Exporter has been my general-purpose solution to this. Once configured, you can hit the agent's metrics endpoint to confirm that it's originating metrics.

If you're getting metrics, then the issue is with your Prometheus Server's target confguration. You can check the server's targets endpoint to ensure it's receiving metrics from the agent.

A second step is to configure zPages and then check /debug/rpcz and /debug/tracez to ensure that gRPC calls are hitting the agent.

E.g.

receivers:
  opencensus:
    address: ":55678"

exporters:
  prometheus:
    address: ":9100"

zpages:
  port: 9999

You should get data on:

http://[AGENT]:9100/metrics
http://[AGENT]:9999/debug/rpcz
http://[AGENT]:9999/debug/tracez

Another hacky solution is to circumvent the Agent and configure e.g. a Prometheus Exporter directly in your code, run it and check that it's shipping metrics (via /metrics).

Oftentimes you need to flush exporters.

How to debug opencensus metrics agent?

1 Answers1