Unable to get Stackdriver Trace information in Google Cloud Console

Question

I have a few microservices running in Container Engine (GKE) and I'm trying to get trace information in Google Console, but something goes wrong.

Here is my checklist:

• Stackdriver Trace API is enabled in API Manager.
• API Manager dashboard shows 99.98% error ratio.
• GKE has permissions:
  • Stackdriver Trace: Write Only
  • Stackdriver Logging API: Write Only
  • Stackdriver Monitoring API: Full
  • Service Control: Enabled
• There are no errors in logs
• I used the following manuals to integrate Trace API:
  • ASP.NET Core
  • Go

Did I miss something? Thanks in advance.

---

Update: I was able to query trace api manually via curl from GKE pod:

kubectl exec -it POD -- /bin/bash

curl "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" -H "Metadata-Flavor: Google"


curl --verbose -d '{"traces": [{"projectId":"xxxx","traceId":"12345678901234567890123456789053","spans":[{"spanId":3,"name":"test"}]}]}' -X PATCH https://cloudtrace.googleapis.com/v1/projects/xxxx/traces -H "Content-Type: application/json" -H "Authorization":"Bearer TOKEN"

curl output:

• Hostname was NOT found in DNS cache
• Trying 173.194.202.95...
• Connected to cloudtrace.googleapis.com (173.194.202.95) port 443 (#0)
• successfully set certificate verify locations:
• CAfile: none CApath: /etc/ssl/certs
• SSLv3, TLS handshake, Client hello (1):
• SSLv3, TLS handshake, Server hello (2):
• SSLv3, TLS handshake, CERT (11):
• SSLv3, TLS handshake, Server key exchange (12):
• SSLv3, TLS handshake, Server finished (14):
• SSLv3, TLS handshake, Client key exchange (16):
• SSLv3, TLS change cipher, Client hello (1):
• SSLv3, TLS handshake, Finished (20):
• SSLv3, TLS change cipher, Client hello (1):
• SSLv3, TLS handshake, Finished (20):
• SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
• Server certificate:
• subject: C=US; ST=California; L=Mountain View; O=Google Inc; CN=*.googleapis.com
• start date: 2017-07-05 08:20:33 GMT
• expire date: 2017-09-27 08:09:00 GMT
• subjectAltName: cloudtrace.googleapis.com matched
• issuer: C=US; O=Google Inc; CN=Google Internet Authority G2
• SSL certificate verify ok.

  PATCH /v1/projects/line-b/traces HTTP/1.1 User-Agent: curl/7.38.0 Host: cloudtrace.googleapis.com Accept: / Content-Type: application/json Authorization:Bearer TOKEN Content-Length: 118

• upload completely sent off: 118 out of 118 bytes < HTTP/1.1 200 OK < Content-Type: application/json; charset=UTF-8 ...

Dashboard also updated:

Answer 1

You can look directly at the API reporting graphs to see that your calls to google.devtools.cloudtrace.v1.TraceService.PatchTraces fail with 403 Forbidden errors.

403 Forbidden is caused when a request to a server is not authorized. Therefore, your calls from your Container Engine (GKE) cluster to 'cloudtrace.PatchTraces' are not authorized.

This could be due to the limited Stackdriver Trace 'Write Only' permission you set. Also, ensure that you added the 'trace.append' scope when creating the cluster.

---

Concerning the missing logs in Stackdriver. Once Stackdriver Logging is enabled for your cluster you should be able to simply write to 'STDOUT' and 'STDERR'. Fluentd should take care of sending this to Stackdriver and the output should be available in the Log Viewer under the 'GKE Container' dropdown.

Alternatively you can always use the Stackdriver Logging Client Library to directly write to Stackdriver, just as you are doing with the Stackdriver Trace API. You should also set the Log Severity to make filtering easier in the Log Viewer to find specific error logs (like the 403s you are seeing).