FATAL log is marked as INFO on Google Cloud Kubernetes

Question

I have an app using Rails 2.5.1 which is deployed on Google Cloud Kubernetes but the logs are not working good.The FATAL errors are displayed as INFO logs instead of ERROR or CRITICAL.

Extra context:

Rails 5.2.1
This view is from Logs Viewer
I'm using RAILS_LOG_TO_STDOUT=true. If I remove this, logs are not displayed at all

Thanks.

[UPDATE]

I updated my cluster to use the newest Stackdriver API but it stills not working and Rails.logger.error is displayed as INFO.

Looks like this topic https://github.com/googleapis/google-cloud-ruby/issues/3990 is the same issue — Serhii Rohoza, Mar 05 '20 at 13:25
Do you use Legacy Logging and Monitoring for your GKE cluster? If yes, here you can find some explanation https://cloud.google.com/monitoring/kubernetes-engine/legacy-stackdriver/logging#best_practices "By default, logs written to the standard output are on the INFO level and logs written to the standard error are on the ERROR level. Structured logs can include a severity field, which defines the log's severity." — Serhii Rohoza, Mar 05 '20 at 13:47
Okay. Do you use Legacy Logging and Monitoring for your GKE cluster? — Serhii Rohoza, Mar 05 '20 at 14:08
So, we found the cause of this "issue" and it's intended behavior. — Serhii Rohoza, Mar 05 '20 at 14:20

score 3 · Accepted Answer · answered Mar 05 '20 at 19:17

I got this working changing the log format to use JSON.

config.log_formatter = proc do |severity, datetime, progname, msg|
    message = msg
    message << " from #{progname}" if progname.present?
    content = JSON.dump(timestamp: datetime.to_s, severity: severity, message: message)
    content << "\n"
    content
  end

Serhii Rohoza · Answer 2 · 2020-03-05T15:55:31.280

Have a look at the documentation Cloud Logging for Legacy Logging and Monitoring section Best practices:

Severities: By default, logs written to the standard output are on the INFO level and logs written to the standard error are on the ERROR level. Structured logs can include a severity field, which defines the log's severity.

and because you're using RAILS_LOG_TO_STDOUT=true log events generated by Ruby you can see with severity INFO.

Keep in mind that you should migrate to Kubernetes Engine Monitoring:

Warning: Legacy Logging and Monitoring support for Google Kubernetes Engine is deprecated. If you are using Legacy Logging and Monitoring, then you must migrate to Kubernetes Engine Monitoring before support for Legacy Logging and Monitoring is removed.

it's better to return to this "issue" after migration.

EDIT Have a look at the documentation Writing Logs section Writing log entries where you can find an example for Ruby:

Here is some sample code to write a single log entry to mylog. The service, region, labels, and other content will change depending on the entry and the application doing the writing.

require "google/cloud/logging"

logging = Google::Cloud::Logging.new project: "my-gcp-project-id"

entry = logging.entry entry.log_name = "my_application_log" 
entry.payload  = "Log message" 
entry.severity = :NOTICE 
entry.resource.type = "gae_app" 
entry.resource.labels[:module_id] = "default" 
entry.resource.labels[:version_id] = "20160101t163030"

logging.write_entries entry

I updated the Stack Driver but I still getting the same issue. `Rails.logger.error` is printing as INFO. — Bruno Quaresma, Mar 05 '20 at 15:30
I think rails is not sending logs with the right severity. I'm basing on this: https://cloud.google.com/logging/docs/setup/ruby#run-gae — Bruno Quaresma, Mar 05 '20 at 15:40
In addition to your link, I've updated my answer with link to documentation **Writing log entries**. — Serhii Rohoza, Mar 05 '20 at 15:56

FATAL log is marked as INFO on Google Cloud Kubernetes

2 Answers2