Recently had to upgrade our GKE logging settings to Cloud Operations for GKE (per https://cloud.google.com/stackdriver/docs/solutions/gke/migration#what-is-changing). From the document, an interesting change in the logName field occurred, where it was previously based on the container name, but now is just "projects/{PROJECTID}/logs/stdout".
This normally would not be a problem but we rely heavily on Logging to BigQuery sinks to be able to analyze our log data. Since BigQuery log sinks use the logName for table it generates, every table we produce now is "stdout_*" instead of the container name. This is very confusing and makes it more difficult to use shared datasets and generally bad from a naming point of view. I have already filed a feature request with google to be able to customize the BQ sink table name, but that does not help our use case right now.
If we can change the logName then that would be able to change the BQ table name as well. I have seen google's documentation for their logging agent but have not found a way to edit this logName field.
Options I have considered:
- cron job to copy the tables to a new table name (downside on maintenance and small increase in cost)
- Disable default logging and use a custom logging solution (large time investment and not clear if that would still help here)
- Biting the bullet and just using the stdout_* table name (very close to choosing this option)
Is there anyway to customize the logName from a k8s container?
