3

I am running one filebeat (version - 6.4.1) per node in kubernetes cluster with 1 master node and 3 worker nodes.

And a single logstash, elastic and Kibana for the entire cluster.

While the pods are up and running successfully, filebeat is unable to pull/send the logs to the logstash.

If I restart the filebeat pods then the logs can be seen from Kibana.

The error I see from filebeat logs are:

ERROR kubernetes/watcher.go:154 kubernetes: Watching API error EOF

Found a similar issue in ELK forums

https://discuss.elastic.co/t/kubernetes-filebeat-stops-sending-picking-up-logs/128578. It is said that filebeat of version 6.3.0 has a fix for this.

Component Versions:

cluster - 4 nodes (1 master & 3 workers)

master - 4 core & 8 GB RAM

worker - 16 core & 32 GB RAM

host OS - Centos: 7

container OS - alpine: 3.9.4

k8s - v1.13.1

docker - 18.09.0

filebeat - 6.4.1

logstash - 6.3.1

elasticsearch - 6.5.4

kibana - 6.5.4

I am facing the same issue in filebeat 6.4.1 > 6.3.0

Please suggest me if I need to make any changes in the ELK configurations.

Bhavani Prasad
  • 1,079
  • 1
  • 9
  • 26
  • Please provide more details about your cluster configuration and component version – Mark Aug 12 '19 at 12:00
  • Updated the details in the question. Please let me know if I miss anything – Bhavani Prasad Aug 13 '19 at 09:18
  • It looks like unresolved issue with filebeat, did you try the one of the new releases ? – Mark Aug 14 '19 at 08:29
  • did you resolve this @BhavaniPrasad? – Gewure Oct 08 '20 at 16:57
  • Fixed by setting up "selinux to permissive" in all the nodes. First the filebeat pods should run in root mode to access the logs from the nodes. Exec into any filebeat pod and check if this "/var/lib/docker/conainers//-json.log" exists. – Bhavani Prasad Oct 09 '20 at 10:38

0 Answers0