best way to send docker container logs to logstash

Question

Let say I have Nginx running inside a container (docker).
The access log and error logs are sent through STDOUT, in the Dockerfile :

# forward request and error logs to docker log collector
RUN ln -sf /dev/stdout /var/log/nginx/access.log
RUN ln -sf /dev/stderr /var/log/nginx/error.log

Logspout seems an elegant solution to send STDOUT of your container inside logstash (configured with a syslog input)

input {
    syslog {
        type => syslog
        port => 5514
    }
}

But logspout have no idea about the format of the log sent through STDOUT (Or am I missing something ?)

So do I have to do something like :

input {
    syslog {
        type => nginx-access
        port => 5514
    }
}

But then what about nginx error log ? And what if I send php-fpm log through STDOUT too ? How does logspout manage this ?

Another solution is to run rsyslog indose the container and send the collected logs to the input of logstatsh ...

As you can see it is not really clear for me ... I would like to be able to send nginx and php-fpm logs to logstash so they can be interpreted as what they are ... but I don't find a "good practice" ...

Can you help me please

womble · Answer 1 · 2019-03-20T20:29:34.673

11

You can do this, now, with a bit of Italian-style plumbing.

In newer versions of Docker, there is a GELF output driver, which you can configure to send the logs. Since logstash has a GELF input plugin, you can configure logstash to receive those same log messages, and do something useful with them.

Another option, if you'd prefer to avoid the GELF translation round-trip, is to use logspout-logstash, a logstash output plugin for logspout, which reads log entries as they come out of Docker.

Addendum: I've been active in this space since this answer was originally written, and have built mobystash as a more modern and manageable replacement for logspout-logstash, as well as syslogstash for taking syslog messages direct from the /dev/log socket and putting them in logstash.

edited Mar 20 '19 at 20:29

answered Nov 12 '15 at 00:49

womble

12,033
5
52
66

1

It's important to notice that the GELF output driver only uses UDP (as of 20.04.2017) so packages might be lost if the logstash is on some other machine. – herm Apr 20 '17 at 14:49
1

Also important to note that the Docker GELF driver is not supporting multiline messages, which is super annoying e.g. with Exception logging: https://github.com/moby/moby/issues/22920 – derFunk May 19 '17 at 08:45

score 1 · Answer 2 · edited May 19 '17 at 13:40

I just noticed that recent versions of nginx can be configured to log to network syslog daemons.

Logging to syslog is available since version 1.7.1.

Using this you can configure a syslog input filter on a dedicated UDP port in logstash and you are off to the races.

Unfortunately, v1.7.1 is a long way off in official repos. You'll need a log shipper somewhere.

In the nginx container.
In a parallel container using '--volumes-from' to access the nginx container.
On the docker host with volumes bound to the host filesystem.

You have many log shippers to pick from and a free chapter of The Logstash Book dedicated to the subject.

best way to send docker container logs to logstash

2 Answers2