Best Docker logging architecture using ELK stack

Question

Recently I am trying to find out best Docker logging mechanism using ELK stack. I am having some questions regarding the best work flow that companies use in production. Our system has typical software stack including Tomcat, PostgreSQL, MongoDB, Nginx, RabbitMQ, Couchbase etc. As of now, our stack runs in CoreOS cluster. Please find my questions below

1. With ELK stack, what is the best methodology to do the log forwarding - Should I use Lumberjack ?. I am asking this because I have seen workflows where people use Syslog/Rsyslog to forward the logs to logstash.
2. Since all of our software pieces are containerized, should I include Log-forwarder in all my containers ? I am planning to do this as most of my containers switch nodes based on health so I am not keen on mounting the file system from the container to host.
3. Should I use redis as a broker in forwarding the logs ? If yes why ?
4. How difficult is it to write log-config files that defines the log format to be forwarded to log-stash ?

This is a subjective questions but I am sure that this is a problem that people have solved long ago and I am not keen on re-inventing the wheel.

Answer 1

Good questions and the answer like in many other cases are - "it depends".

1. Shipping Logs - we use rsyslog as docker containers internally and logstash-forwarder in some cases - the advantage of logstash-forwarder is that it encrypts the logs and compresses them so in some cases that's important. I find rsyslog to be very stable and low on resources so we use it as a default shipper. The full logstash might be heavy for small machines (some more data about logstash - http://logz.io/blog/5-logstash-pitfalls-and-how-to-avoid-them/)

2. We're also fully dockerized and use a separate Docker for each rsyslog/lumberjack. Easy to maintain, update versions and move around when needed.

3. Yes, definitely use Redis. I wrote a blog about how to build production ELK (http://logz.io/blog/deploy-elk-production/) - I spoke about what I find to be the right architecture to deploy ELK in production

4. Not sure what exactly are you trying to achieve with that.

HTH

Answer 2

Docker as of Aug 2015, has "Logging Driver", so that you can ship logs into other places. These are the supported way to ship the logs remotely.

• syslog
• fluentd
• journald
• gelf
• etc..

Answer 3

I would recommend against putting the logging forwarder into each Docker image. That adds unneeded complexity and bloat to your Docker containers. A cleaner solution is to put the log forwarder (the latest log forwarder from Elastic being FileBeat, which replaces logstash forwarder) into its own container and mount the host machine's /var/lib/docker directory as a volume for that container.

docker run --detach --name=docker-filebeat -v /var/lib/docker:/var/lib/docker

/var/lib/docker contains all the logs for every container running on the host's Docker daemon. The data of the log files in this directory is the same data you would get from running docker logs <container_id> on each container.

Then in the filebeat.yml configuration file, put:

filebeat:
  prospectors:
    -
      paths:
        - /var/lib/docker/containers/*/*.log

Then config Filebeat to forward to the rest of your ELK stack and start the container. All the Docker container logs on that machine will be forwarded to your ELK stack automatically.

The cool thing about this approach is that it allows you to forward the rest of the host systems logs as well if you want to. Simply add another volume pointing to the host system log files you want to forward and add that path to your filebeat.yml config as well.

I find this method cleaner and more flexible than other methods such as using the Docker logging drivers because the rest of the your Docker setup stays the same. You don't have to add the logging driver flags to each Docker run command (or to the Docker daemon parameters).