0

I am running a nodeJS server on Ubuntu 14.04 in Google Compute Engine. I want to use google cloud logging for my application so I installed google fluentd logging agent as per https://cloud.google.com/logging/docs/agent/installation

I used winston and winston-syslog for writing logs. Here is the code. 

var winston = require('winston');
var winstonsyslog = require('winston-syslog').Syslog;

var options = {
    json : true
};

winston.add(winston.transports.Syslog, options);

When I am writing a log using 

winston.log('info', "27", { anything: 'This is metadata' });

I am getting 

{
  metadata: {…}    
  textPayload: "May 14 10:47:44 localhost node[7633]: {"anything":"This is metadata","level":"info","message":"27"}"    
  insertId: "..."    
  log: "syslog.local0.info"    
}

How to get structPayload instead of textPayload which displays log as JSON instead of String.

tacticurv
  • 498
  • 4
  • 18
  • I'm also having the same issue.. I gave up in the end.. The [docs](https://cloud.google.com/appengine/articles/logging#cloud_logging_and_the_flexible_environment) state that you should name your logs with the suffix `.log.json` and that you should have an inline valid JSON object on each line.. though even with this I was unable to read it through Googles cloud logging.... FYI: Your log, as in "testPayload" is not a JSON object – farridav May 18 '16 at 12:51
  • I am using gcloud nodeJS client now for logging to google cloud logging service. Somehow it works https://googlecloudplatform.github.io/gcloud-node/#/docs/v0.34.0/logging/log – tacticurv May 19 '16 at 12:02
  • I think that's because it taps into Googles underlying fork of fluentd... I was able to get logs working by imitating the same format found [here](https://cloud.google.com/logging/docs/view/logs_index#common) .. although i think the right way to do it is using the [logging agent](https://cloud.google.com/logging/docs/agent/) – farridav May 19 '16 at 12:27

1 Answers1

0

Logging agent has it's own configuration files, and most of them have format none (see https://github.com/GoogleCloudPlatform/fluentd-catch-all-config). Thus, all log lines go to textPayload.

The solution is to write your own fluentd configuration file and use fluent-plugin-google-cloud as output. fluent-plugin-google-cloud should be installed as a gem directly, not using Logging Agent As long as your entry is a valid JSON, you will see that entry on Stackdriver having structPayload set properly. Key/value filters work as well.

I have never used winston, but here is sample configuration:

<source>
    @type tail
    format apache
    path /var/log/apache/access.log
    tag apache.access
</source>

<match **>
    @type google_cloud
    buffer_chunk_limit 10k
</match>

Notice the buffer_chunk_limit 10k, setting it to values too big or leaving it at default may result in Google::Apis::ClientError badRequest: Request payload size exceeds the limit.

prog893
  • 1
  • 3