Unable to collect all kubernetes container/pod logs via fluentd/elasticsearch

Question

I'm new with fluentd/elasticsearch stack and I'm trying to deploy it on kubernetes. While I've managed to do that, I'm having a problem that not all pod/container logs are showing up on elasticsearch (I'm using Kibana for data visualisation). In other words, I'm able to see logs from "default" kubernetes pods like weave-net and elasticsearch related pod logs (es-data, es-master...etc.) but not from "custom" pods that I'm trying to deploy.

As a simple test, I've deployed redis in the same kube namespace where fluentd/elasticsearch resides and redis service/deployment looks like this:

---
apiVersion: v1
kind: Service
metadata:
  name: redis-master
  labels:
    app: redis
    role: master
    tier: backend
spec:
  ports:
  - port: 6379
    targetPort: 6379
  selector:
    app: redis
    role: master
    tier: backend
---
apiVersion: apps/v1 #  for k8s versions before 1.9.0 use apps/v1beta2  and before 1.8.0 use extensions/v1beta1
kind: Deployment
metadata:
  name: redis-master
spec:
  selector:
    matchLabels:
      app: redis
      role: master
      tier: backend
  replicas: 1
  template:
    metadata:
      labels:
        app: redis
        role: master
        tier: backend
    spec:
      containers:
      - name: master
        image: k8s.gcr.io/redis:e2e  # or just image: redis
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 6379

When I check logs from fluentd daemonpods, I see following:

2018-07-03 11:17:05 +0000 [info]: following tail of /var/log/containers/redis-master-585798d8ff-b5p5g_default_master-4c934d19a8e2b2d6143b662425fd8fc238df98433d1c0c32bf328c281ef593ad.log

which, if I'm correct, should give me an info that fluentd is picking up redis container logs. However, I'm unable to see any redis related documents stored in elasticsearch.

This is how part of the configuration for fluentd looks like (kubernetes.conf):

<source>
  @type tail
  @id in_tail_container_logs
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kubernetes.*
  read_from_head true
  format json
  time_format %Y-%m-%dT%H:%M:%S.%NZ
</source>

and fluent.conf:

<match **>
   @type elasticsearch
   @id out_es
   log_level info
   include_tag_key true
   host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
   port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
   scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
   ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"
   user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
   password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
   reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'true'}"
   logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'}"
   logstash_format true
   buffer_chunk_limit 2M
   buffer_queue_limit 32
   flush_interval 5s
   max_retry_wait 30
   disable_retry_limit
   num_threads 8
</match>

Any hint would be very helpful. Thanks in advance.

score 2 · Accepted Answer · answered Jul 03 '18 at 20:28

I am using fluent bit for the same purpose and I met exactly the same problem quite a few days back. Fluent bit is a light weight version of fluentd, and what worked for me might work for you as well.

What was wrong with my fluent bit was the input configuration. For the tail plugins that tail into large log files, there was some issue with the log rotation. So I lowered my refresh_interval to something like 5 secs (time period over which list of watched files are updated). Then I lowered the mem_buf_limit to something like 5MB (the total size of logs fluent bit takes into memory before flushing that out to the output plugin).

By these changes I was able to get more logs which were earlier not being collected for god knows reason.

I have asked this as an issue. Will update my answer if I get to know the reason.

Hope this helps in anyway. Mainly I suggest you to tweak your input configurations and then see the changes.

Unable to collect all kubernetes container/pod logs via fluentd/elasticsearch

1 Answers1