2

I'm trying to gather logs from all my running docker containers and send them into the ELK stack. I'd like to use filebeat to do this so I'm following a similar approach to what is described in https://logz.io/blog/docker-logging/.

My filebeat.yml

filebeat:
  inputs:
  - paths:
    - /var/jenkins_home/jobs/*/branches/*/builds/*/log
    document_type: jenkinslog
  - type: docker
    containers.ids: '*'
    document_type: docker     
output:
  logstash:
    hosts: ["logstash:5044"]

My Dockerfile:

FROM docker.elastic.co/beats/filebeat:6.5.2
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
USER root
RUN chown root:filebeat /usr/share/filebeat/filebeat.yml

My command to run this image:

docker run --rm --name filebeat_container --volumes-from jenkins_container:ro -v /var/lib/docker:/var/lib/docker --network=ttydinternproject_default filebeatimage

I'm mounting the /var/lib/docker folder as a volume and have to be root to access it as according to this https://github.com/hashicorp/vagrant/issues/6822#issuecomment-169826764 "/var/lib/docker is for the docker daemon and no one else.".

There must be a better way to get docker container logs into filebeat than to leave the filebeat container running root to access folders to apparently shouldn't even be accessed. Do I need to switch from the default docker logging driver to one of the other ones supported?

yoomtah
  • 33
  • 4
  • you can change docker log driver to fluent-driver. and it can makes send logs to configured destination. – GNOKOHEAT Jan 17 '20 at 05:07

1 Answers1

1

You can reconfigure your Jenkins container to publish its log files to a host directory (use docker run -v to provide some host directory for the /var/jenkins_home/jobs tree; this is probably a good idea regardless since you don't want to lose all of your job history if you ever need to update the underlying Jenkins code). You can then either use docker run -v to inject that same directory into the Filebeat container, or just run Filebeat directly on the host (if its principal job is reading host-system files...).

If you have the option and are in a more productiony setup, switching log drivers to point at your logstash is also a good idea, but that will only collect the main process's stdout and stderr (instead of having to run docker logs that data will show up on your central log server). That won't collect the per-Jenkins-job log files, though.

My experience agrees with the Vagrant bug you quote: never look inside /var/lib/docker, and especially don't try to mount Docker's internal state into a Docker container. (You probably won't get a kernel panic.)

David Maze
  • 130,717
  • 29
  • 175
  • 215