2

Is there a way to automatically ship the logs from EKS pods to cloudwatch using fluent bit whenever new pods come up ? I am able to send the logs by adding INPUT, FILTER, OUTPUT section in fluent-bit configmap whenever we need to deploy a new service. But this process is cumbersome. Is there a way to automate this part ? like when we do a new deployment , fluent bit sees the new pods and ships the log to specific log group in cloudwatch.

Thanks in advance.

Satyaki
  • 31
  • 5
  • Your fluent-bit not running as daemonset? – gohm'c Dec 08 '21 at 07:09
  • yes , it is running as a daemon set. But I dont see the new pod's log getting shipped until I put the INPUT and OUTPUT section. I have a feeling that , there is something missing in my config. – Satyaki Dec 08 '21 at 09:59
  • Your logs write to stdout or custom location on the host? – gohm'c Dec 08 '21 at 10:59
  • I have both . Few service writes log in stdout and few writes log to custom locations. Those custom locations anyway have mounted in the flunetbit daemon set . – Satyaki Dec 09 '21 at 08:07
  • 1
    Here is the config file for flunet-bit I am using. https://github.com/satyaki88/fluent-bit/blob/main/fluent-bit.yaml – Satyaki Dec 09 '21 at 08:14

2 Answers2

2

Digging into the documentation here is the solution, there are two cloud watch output plugins mainly used and both of them supports what is called log group templating:

For example if your aggregated logs will be in this form:

{
  "kubernetes": {
      "container_name": "sample-api"
      "namespace_name": "sample-namespace",
      "pod_name": "sample-api-597d7449b5-wbcmq"
  },
  "log": "sample logs \n"
}
  1. cloudwatch created by aws (most probably if you deployed fluentbit following aws documentation)
  [OUTPUT]
        Name                cloudwatch
        Match               application.*
        region              ${AWS_REGION}
        log_group_name      /aws/myKubeCluster/$(kubernetes['container_name'])
        log_stream_name     $(kubernetes['container_name']).$(kubernetes['container_name'])
        auto_create_group   true
        extra_user_agent    container-insights
  1. cloudwatch_logs which is the new plugin that is provided by fluentBit and exist in their official documentation
  [OUTPUT]
        Name                cloudwatch_logs
        Match               application.*
        region              ${AWS_REGION}
        log_group_name      fallback_group
        log_stream_name     fallback_stream_name
        log_group_template  /aws/myKubeCluster/$kubernetes['container_name']
        auto_create_group   true
        extra_user_agent    container-insights
  • The difference is that, log_group_template is separate field and in the way that you call your vars (record_accessor)

Resources:

Ahmed Elsherbiny
  • 56
  • 6
0

Your input paths have specific expression that capture only certain logs:

/var/log/containers/grafana.log

/var/log/containers/memsql.log,/var/log/containers/studio.log

No custom location seen in your config. Anyway, you can follow EKS container insights (with fluent-bit) here which you do not need to change your config every time a new workload is introduced.

gohm'c
  • 13,492
  • 1
  • 9
  • 16
  • 1
    by default all the logs will go in /aws/containerinsights/Cluster_Name/application . But I need new workload's log will create a separate Log-group. How can we achieve this , without adding the OUTPUT section in the configmap ? – Satyaki Dec 09 '21 at 12:03