I'm running elasticsearch, fluentd, and Kibana in an EKS Kubernetes cluster. I have 3 pods in the same cluster that generate logs which are written to a volume mount, which is mounted in the fluentd pod. I have provided all 3 log paths in the fluent config file, like so:
path /first-pod/*.log,/second-pod/*.log,/third-pod/*.log
However, I only see the logs from the first-pod
showing up. If reorder the path like so:
path /second-pod/*.log,/third-pod/*.log,/first-pod/*.log
I will only see the logs of second-pod
.
I also tried putting everything in a single folder
/path/to/logs/
∟first-pod
∟foo.log
∟bar.log
∟second-pod
∟foo.log
∟bar.log
∟third-pod
∟foo.log
∟bar.log
And setting the path like this:
path /path/to/logs/**/*.log
But then it only took the logs of third-pod
(likely selecting the alphabetically last).
So any ideas on how I can get logs from all directories pushed into elasticsearch?
Full fluentd conf file:
<source>
@type tail
@id in_tail_container_logs
path /first-pod/*.log,/second-pod/*.log,/third-pod/*.log
pos_file /var/log/fluentd-containers.log.pos
tag inc.log
exclude_path ["/var/log/containers/fluent*"]
read_from_head true
<parse>
@type grok
grok_pattern %{TIMESTAMP_ISO8601:timestamp} \| %{GREEDYDATA:jsonbody} \|
</parse>
</source>
<match inc.log>
@type elasticsearch
<buffer>
flush_thread_count 8
</buffer>
host elasticsearch.kube-logging.svc.cluster.local
port 9200
logstash_format true
</match>