How to add missing metadata to Google logging agent

Question

Using a docker image based on this I've created a pod on GKE. The agent will listen for fluentd events via TCP (my other application pods will send events), then forward those logs on to Google Cloud Logging. As is these events are missing some metadata. How can I add this missing information?

(symfony app)--[monolog]-->(google-fluentd-agent)-->(Cloud Logging)

google-fluentd.conf:

<match fluent.**>
  type null
</match>

# TCP Connections for fluentd aware applications.
<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match **>
  type google_cloud
  # Set the chunk limit conservatively to avoid exceeding the GCL limit
  # of 10MiB per write request.
  buffer_chunk_limit 2M
  # Cap the combined memory usage of this buffer and the one below to
  # 2MiB/chunk * (24 + 8) chunks = 64 MiB
  buffer_queue_limit 24
  # Never wait more than 5 seconds before flushing logs in the non-error case.
  flush_interval 5s
  # Never wait longer than 30 seconds between retries.
  max_retry_wait 30
  # Disable the limit on the number of retries (retry forever).
  disable_retry_limit
</match>

Event in Google Logging with missing data:

{
  metadata: {
    projectId: "my-project"      
    serviceName: "container.googleapis.com"      
    zone: "us-central1-a"      
    labels: {
      container.googleapis.com/cluster_name: "app-staging-a"        
      compute.googleapis.com/resource_type: "instance"        
      compute.googleapis.com/resource_name: "cluster-fluentd-1dom0"        
      container.googleapis.com/instance_id: "296757089355968949"        
      container.googleapis.com/pod_name: ""        
      compute.googleapis.com/resource_id: "296757089355968949"        
      container.googleapis.com/namespace_name: ""        
      container.googleapis.com/container_name: ""        
    }
    timestamp: "2016-05-16T00:25:37.000Z"      
    projectNumber: "10568438715"      
  }
  insertId: "94dadf6548d"    
  log: "symfony.php"    
  structPayload: {
    context: {
      stack: [33]       
      file: "classes.php"        
      type: 16384        
      line: 4156        
      level: 28928        
    }
    level: "INFO"      
    message: "Using an instance of "This_Function_Method" for function "some_stuff" is deprecated."      
  }
}

Answer 1

The google_cloud output plugin attempts to parse the namespace, pod, and container names out of the name of the log stream coming into it. In the normal setup, this works because the files on disk coming from each container's stdout/stderr are named that way. In order to get similar parsing behavior, you'll have to similarly craft the log stream name that you send to fluentd (or implement your own logic in a custom fluentd plugin).