2

Background (skip to "The Problem" for end goal)

Take an example endpoint /process-batch that processes a batch of items given a single request. Upon receiving that request, it logs the following line:

{"endpoint_type": "process-batch", "batch_size": <integer>, "are_there_warnings": <boolean>}

I can configure a logs-based metric for this via:

enter image description here

Then I can create a line chart (changing the "Resource & Metric" to be the logs-based metric created above, specifying that it can be emitted from anywhere globally):

enter image description here

Resulting dashboard looks like this:

enter image description here

The issue is that batch_size and are_there_warnings don't automatically appear as fields that I can use to manipulate the line graph. See "Group By" and "Add Filter" below:

enter image description here enter image description here

To enable those fields, I can add a label for each of them (let's call this the "explicit labeling" step for future reference):

enter image description here

enter image description here

After confirming those labels are added:

enter image description here

I can go back to the dashboard and see that "Group By" and "Add Filter" have been updated accordingly:

enter image description here

And can manipulate the dashboard to use those fields:

enter image description here

The Problem

When trying to improve observability for an existing codebase, I have to add all the existing fields already logged as explicit labels on the GCP side, instead of GCP inferring that the logged fields are labels based on their name and type. And for the rare cases when I add a new field (such as cost to the log lines emitted by /process-batch), I have to do explicit labeling on the GCP Monitoring and Logging side instead of GCP inferring that the newly added field should be a new implicitly added label in group by and filter.

Is it possible for GCP to automatically treat all fields part of jsonPayload as labels? Here are some reasonable defaults I had in mind:

  • "Log metric name" takes on the name of the field itself.
  • "Description" would be left empty.
  • "Label type" would be determined from the type of the field.
  • "Field name" would be jsonPayload.<log-metric-name>.

If this is not possible, I'd also appreciate any reasoning if implicitly treating jsonPayload fields as labels is a bad idea. In my eyes, it would significantly reduce the effort to have functional dashboards.

Mario Ishac
  • 5,060
  • 3
  • 21
  • 52

2 Answers2

1

Upon checking, This kind of feature is not yet available. However we already raise this request. This request does not have an ETA but we can monitor this using below link.

You can star the public issue tracker feature requests and add ‘Me too’ in the thread. This will bring more attention to the request as more users request support for it.

Feature Request Tracker : Autodetect Labels from JSON Payload

As of now below are some guides that can help you.

Google Cloud fluentd output plugin configuration

Configuring the agent

Processing payloads

Special fields in structured payloads

JaysonM
  • 596
  • 1
  • 10
0

I am from the Cloud Logging Engineering team.

Auto propagating jsonPayload fields from logs into logs-based metric (LBM) labels is currently not technically possible. LBMs are metrics that are created with certain customizable labels. As logs are ingested, the metric counters are incremented with the label values. There are limits on the number of labels and their cardinality in logs-based metrics, whereas jsonPayload in logs have unbounded cardinality and evolving schemas.

For a use-case like this, Log Analytics is a better fit. We are adding support for charting and dashboarding in Log Analytics such that you can directly add a chart with any jsonPayload values as dimensions or breakdowns by directly querying the logs using SQL. Stay tuned for this feature in the coming weeks.

Summit Raj
  • 820
  • 5
  • 10