3

I'd like to get notifications for all standard error logs sent to Google Cloud Logging. Preferably, I'd like to get the notifications through Google Cloud Error Reporting, so I can easily get notifications on my phone through the GCP mobile app.

I've deployed applications to Google Kubernetes Engine that are writing logs to standard error, and GKE is nicely forwarding all the stderr logs to Google Cloud Logging with logName: "projects/projectName/logs/stderr"

I see the logs show up in Google Cloud Logging, but Error Reporting does not pick up on them.

I've tried troubleshooting as described here: https://cloud.google.com/error-reporting/docs/troubleshooting. But the proposed solutions revolve around formatting the logs in a certain way. What if I've deployed applications for which I can't control the log messages?

A (totally ridiculous) option could be to create a "logs-based metric" based on any log sent to stderr, then get notified whenever that metric exceeds 1.

What's the recommended way to get notified for stderr logs?

Eric Green
  • 31
  • 2

2 Answers2

0

  • If Error Reporting is not recognizing the stderr logs from your container it means they are not displayed in the correct format for this API to be able to detect them.

  • Take a look at this guide on how to setup error reporting for GKE API

  • There are other ways to do this with third party products like gSlack where basically you will export the stackdriver logs to pub/sub and then send them into the Slack channel with Cloud Functions.

  • You can also try to do it by using Cloud Build and try to integrate it with GKE container logs.

Still, I think the best and easiest option will be by using the Monitoring Alert.

Jujosiga
  • 409
  • 2
  • 7
  • 2
    That second link says "modify your application." My problem is that I can't modify the application. It's logging to stderr, but not in the exact format that Error Reporting would like. Is there another solution? – Eric Green Feb 12 '21 at 09:12
  • This answer doesn't address the case where you don't control the log message format, such as when deploying a third-party container (say, Grafana) and want alerts for any output on stderr – Benjamin Egelund-Müller Mar 05 '21 at 15:29
0

You can force the error by setting the @type in the context as shown in the docs. For some reason, even if this is the Google library, and it has code to detect that an Exception is thrown, with its stack trace, it won't recognize it as an error worth reporting.

I also added the service array to be able to identify my service in the error reporting.

enter image description here

JorgeeFG
  • 5,651
  • 12
  • 59
  • 92