0

Stackdriver logging export exports the entire json object, not just the jsonPayload I sent it.

I am sending JSON data to stackdriver via the Python SDK.

The data get ingested into Stackdriver fine, in the below format 

{ 
insertId: ""
jsonPaload: [what I sent]
labels: []
logname: ""
receiveTimestamp: ""
resource: {}
timestamp: ""
}

When I export the log to GCS, that entire payload is sent, not just my jsonPload and there appears to be no way to just send my data to GCS.

The issue this causes is that I have strip out my jsonPload data if I want to use it again, e.g. mydata=json.loads(export_line)['jsonPayload'].

Example: A vendor wants my bro logs. I can't just give them the GCS logs because the bro data are nested in jsonPayload. So, I have to either pre-parse the data and manually send to GCS or pull it from GCS, parse it, and give it to the vendor.

I would expect that there should be a way for me to export ONLY the jsonPaload that I sent it. i.e. I send stackdriver data, tell it to export my data, and only my data are exported, not its internal telemetry/additions.

MSven
  • 36
  • 3

1 Answers1

0

The problem is that every log entry in Stackdriver Logging is an object of type LogEntry, and the logging export will export the actual object to GCS. You can't isolate the payload even if you parse before writing the export, as the LogEntry object will always have at least the logName and resource fields as part of their structure.

If you want to make only the [*]Payload field available to someone else from your GCS log export, you would need to parse it before sharing. You could also export the logs to a BigQuery dataset, query against it, and share the output directly or through a view instead of parsing the LogEntry object from GCS.

Pablo
  • 66
  • 3
  • Thanks. I was kinda afraid of that. I'm exporting and manually parsing data, just hope/hoped there would be a checkbox on export to say "only export payload" BigQuery also exports the entire object so queries are select * from dataset.table where jsonPayload.field = "something" so, an additional field attribute. – MSven Sep 04 '19 at 15:53