0

I have logentries coming into logstash from different containers. Each logentry contains a field ContainerID and a field Message. Any messages with the same ContainerID I assume are ordered properly.

I want to merge the contents of messages of multiple logentries with the same containerId like the following:

  1. PREFIX | String1
  2. String2
  3. String3
  4. PREFIX | String4

So that I end up with logentries:

  1. PREFIX | String1 \n String2 \n String 3
  2. PREFIX | String4

Can this be done with the logstash aggregate filter plugin?

Specifically, will there be issues if messages with a different containerID are received in between 1., 2., 3., and 4. from above?

Background:

I have multiple docker containers generating log entries which are sent to logstash using the docker gelf log driver from multiple docker deamons (in a swarm).

All of this works and logstash receives and filters the logentries.

Docker views each line as a separate logentry and forwards it to logstash. In case of multiline logentries this is a problem. I want to "merge" many logentries which correspond to one multiline message into one. I know when a "logical" logentry ends because the next entry will start with PREFIX which is why I suggest the solution above.

herm
  • 14,613
  • 7
  • 41
  • 62

1 Answers1

1

I suggest to use Filebeat for that. It has multiline support and with the latest release you can make use of the Autodiscover functionality for Docker.

I recommend to read the discussion in: https://github.com/elastic/beats/issues/918

Dennis Kieselhorst
  • 1,280
  • 1
  • 13
  • 23
  • I read that the "autodiscover" functionality enriches the logentries with fields from docker (such as servicename). If this information contains all (or the most of the usefull) fields that the gelf driver delivers, then this would indeed be a nice solution – herm Jan 12 '18 at 09:03
  • Nevertheless do you think the aggregate filter could be used to merge such messages? – herm Jan 12 '18 at 16:26