Filebeats doesn't foward Docker compose logs, why?

Question

I am following this tutorial to set up a ELK stack (VPS B) that will receive some Docker/docker compose images logs (VPS A) using Beatfile as forwarder, my diagram is as shown below

So far, I have managed to have all the interfaces with green ticks working. However, there are still remaining some issues in that I am not able to understand. Thus, I would appreciate if someone could help me out a bit with it.

My main issue is that I don't get any Docker/docker-compose log from the VPSA into the Filebeat Server of VPSB; nevertheless, I got other logs from VPSA such as rsyslog, authentication log and so on on the Filebeat Server of VPSB. I have configured my docker-compose file to forward the logs using rsyslog as logging driver, and then filebeat is fowarding that syslog to the VPSB. Reaching this point, I do see logs from the docker daemon itself, such as virtual interfaces up/down, staring process and so, but not the "debug" logs of the containters themselves.

The configuration of Filebeat client in VPSA looks like this

root@VPSA:/etc/filebeat# cat filebeat.yml 
filebeat:
  prospectors:
    -
      paths:
        - /var/log/auth.log
        - /var/log/syslog
#        - /var/log/*.log


      input_type: log

      document_type: syslog

  registry_file: /var/lib/filebeat/registry

output:
  logstash:
    hosts: ["ipVPSB:5044"]
    bulk_max_size: 2048

    tls:
      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

shipper:

logging:
  files:
    rotateeverybytes: 10485760 # = 10MB
    level: debug

One of the docker-compose logging driver looks like this

redis:
    logging: 
       driver: syslog
       options:
           syslog-facility: user

Finally I would like to ask, whether it is possible to forward natively from docker-composer the logs to Filebeat client in VPSA, red arrow in the diagram, so that it can forward them to my VPSB.

Thank you very much,

REgards!!

score 0 · Accepted Answer · answered Oct 20 '16 at 14:49

The issue seemed to be in FileBeat VPSA, since it has to collect data from the syslog, it has to be run before that syslog!

Updating rc.d made it work

sudo update-rc.d filebeat defaults 95 10

My filebeats.yml if someone needs it

root@VPSA:/etc/filebeat# cat filebeat.yml 
filebeat:
  prospectors:
    -
      paths:
#        - /var/log/auth.log
        - /var/log/syslog
#        - /var/log/*.log


      input_type: log
      ignore_older: 24h
      scan_frequency: 10s      
      document_type: syslog

  registry_file: /var/lib/filebeat/registry

output:
  logstash:
    hosts: ["ipVPSB:5044"]
    bulk_max_size: 2048

    tls:
      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

shipper:

logging:
  level: debug
  to_files: true
  to_syslog: false
  files:
    path: /var/log/mybeat
    name: mybeat.log
    keepfiles: 7
    rotateeverybytes: 10485760 # = 10MB

Regards

Filebeats doesn't foward Docker compose logs, why?

1 Answers1