How do I correlate request logs in Cloud Run?

Question

I am using self structured JSON payloads for logging from my Node.js/Express based Cloud Run service and I am unable to get logs from the same request to correlate using the trace method.

The documentation says:

Container logs are not automatically correlated to request logs unless you use a Stackdriver Logging client library. If you want this correlation without using a client library, use a structured JSON log line that contains a trace field with the content of the incoming X-Cloud-Trace-Context header. Then your logs will be correctly correlated to the request log.

I know that my structured JSON logs are working as the level/severity and message are being extracted and displayed as expected.

I am passing as the value of trace exactly what is passed by the X-Cloud-Trace-Context header which I get using req.get method provided by Express: req.get('X-Cloud-Trace-Context').

Here is the JSON that is being logged:

{
    "message": "Create Query",
    "level": "debug",
    "severity": "DEBUG",
    "trace": "40f...........................cc/131...............23;o=1"
}

Below is an example of how that log line is present in Stackdriver Logging.

I have also tried using the logging.googleapis.com/trace property as mentioned in the Special fields in structured payloads documentation. I'm fairly certain the value of the X-Cloud-Trace-Context header is not valid for this property but I'm not sure how to format the header value to match the value as documented on this page.

Given the above, my questions are:

Which is the correct property name to use for trace?
How do I correctly format the value of this property based on the value of the X-Cloud-Trace-Context header?

Here is an example of a complete log message as presented (with IDs removed) in Stackdriver Logging:

{
 insertId:  "..."  
 jsonPayload: {
  level:  "debug"   
  message:  "Create Query"   
  trace:  "40f...........................cc/131...............23;o=1"   
 }
 labels: {
  instanceId:  "0.........................................2"   
 }
 logName:  "projects/b.............0/logs/run.googleapis.com%2Fstdout"  
 receiveTimestamp:  "2019-08-16T18:05:58.816240093Z"  
 resource: {
  labels: {
   configuration_name:  "a..................ing"    
   location:  "..."    
   project_id:  "b.............0"    
   revision_name:  "a..................ing-01987"    
   service_name:  "a..................ing"    
  }
  type:  "cloud_run_revision"   
 }
 severity:  "DEBUG"  
 timestamp:  "2019-08-16T18:05:58.479527Z"  
}

Jim Fielding · Accepted Answer · 2019-12-24T17:42:36.320

Which is the correct property name to use for trace?

The property name required in your JSON string is logging.googleapis.com/trace. This is pulled out of the jsonpayload and into the trace property that you can see example usage of on "INFO" logs.

How do I correctly format the value of this property based on the value of the X-Cloud-Trace-Context header?

The format required is as follows: projects/[project]/traces/[trace] where [project] is your Google Cloud Project i.e. b.......0 and [trace] as from your example is 40f...........................cc. The base and query param needs to be omitted (this isn't particular well documented).

Below is a snippet of this performed in JS from the Google documentation found here: https://cloud.google.com/run/docs/logging

// Build structured log messages as an object.
const globalLogFields = {};

// Add log correlation to nest all log messages beneath request log in Log Viewer.
const traceHeader = req.header('X-Cloud-Trace-Context');
if (traceHeader && project) {
  const [trace] = traceHeader.split('/');
  globalLogFields[
    'logging.googleapis.com/trace'
  ] = `projects/${project}/traces/${trace}`;
}

Below is a Golang function I have had success with in terms of pulling the information out and formatting it correctly:

func extractTracePath(r *http.Request, app *application) string {
    s := r.Header.Get("X-Cloud-Trace-Context")
    traceURL, err := url.Parse(s)
    if err != nil {
        app.infoLog.Fatal("Invalid trace URL")
    }
    tracePath := traceURL.Path
    trace := strings.Split(tracePath, "/")[0]
    project := os.Getenv("GCLOUD_PROJECT")
    partialTracePath, err := url.Parse("projects/" + project + "/traces/")
    tracePath = partialTracePath.Path
    tracePath = path.Join(tracePath, trace)
    return tracePath
}

How do I correlate request logs in Cloud Run?

1 Answers1

Linked