I have a number of servers that use CoreOS. CoreOS natively uses journald for all it's log files. The latest stable release incorporates docker 1.9 so it doesn't yet support docker log drivers.
After exploring multiple avenues, I think rsyslog might be the easiest way to get logs into logz as this is listed as a support log shipper in their documentation.
And I discovered that there is a module that allows you to link journal and rsyslog together. So, using a container I have given it a go.
According the to logz documentation I can ship logs to them using rsyslog by basically using the following config for rsyslogd.conf
# -------------------------------------------------------
# File Logging Directives for Logz.io
# -------------------------------------------------------
$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/spool/rsyslog
# File access file:
$InputFileName PATH_TO_FILE
$InputFileTag TYPE:
$InputFileStateFile stat-TYPE
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
$template logzFormatFileTagName,"[SPECIAL_KEY_HERE] <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [type=TYPE] %msg%\n"
if $programname == 'TYPE' then @@listener.logz.io:5000;logzFormatFileTagName
if $programname == 'TYPE' then ~
However, because I'm not using a log file for input, but the journald via modimuxsock I omitted all the logfile config and added:
$ModLoad imuxsock.so
$OmitLocalLogging off
I created a simple Docker image that runs rsyslogd. So in the end I have the following as a config file and I pass in the socket via docker's run -v hostpath:containerpath for the journald socket.
#
# http://www.rsyslog.com/doc/
#
# Input modules
$ModLoad immark.so # provide --MARK-- message capability
$ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command)
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf
$OmitLocalLogging off
$DebugFile /tmp/rsyslog-debug.log
$DebugLevel 2
$template logzFormatFileTagName,"[SPECIAL_KEY_HERE] <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [type=docker] %msg%\n"
if $programname == 'docker' then @@listener.logz.io:5000;logzFormatFileTagName
if $programname == 'docker' then ~
To be honest, I'm a little confused by the TYPE setting. I've assumed I should be setting it to dockerd. As when you run journalctl -u unit, Where unit is a unit that runs a service under docker, it comes through showing e.g.
Is this correct?
My docker image is being run with:
docker run --name=logzio-journald -v /run/systemd/journal/syslog:/run/systemd/journal/syslog --rm myregistry.com/logzio-journal-shipper
Last 30 odd lines of debug output: Note, when something gets logged by docker, no more debug appears. Nothing appears under my logz account either.
8562.569331781:main Q:Reg/w0 : processBATCH: next msg 1: warning: ~ action is deprecated, consider using the 'stop' statement instead [v8.9.0 try http://www.rsyslog.com/e/2307 ]
8562.569337108:main Q:Reg/w0 : IF
8562.569350044:main Q:Reg/w0 : var 'programname'
8562.569370397:main Q:Reg/w0 : ==
8562.569386822:main Q:Reg/w0 : string 'docker'
8562.569412534:main Q:Reg/w0 : eval expr 0x561c9dddcf80, type 'CMP_EQ'
8562.569419072:main Q:Reg/w0 : eval expr 0x561c9dddcf20, type 'V[86]'
8562.569425909:main Q:Reg/w0 : rainerscript: var 17: 'rsyslogd-2307'
8562.569454839:main Q:Reg/w0 : eval expr 0x561c9dddcf20, return datatype 'S'
8562.569463719:main Q:Reg/w0 : eval expr 0x561c9dddcf80, return datatype 'N'
8562.569469502:main Q:Reg/w0 : if condition result is 0
8562.569474698:main Q:Reg/w0 : IF
8562.569487271:main Q:Reg/w0 : var 'programname'
8562.569507498:main Q:Reg/w0 : ==
8562.569523822:main Q:Reg/w0 : string 'docker'
8562.569549617:main Q:Reg/w0 : eval expr 0x561c9dddf780, type 'CMP_EQ'
8562.569556377:main Q:Reg/w0 : eval expr 0x561c9dddefc0, type 'V[86]'
8562.569562533:main Q:Reg/w0 : rainerscript: var 17: 'rsyslogd-2307'
8562.569568453:main Q:Reg/w0 : eval expr 0x561c9dddefc0, return datatype 'S'
8562.569574734:main Q:Reg/w0 : eval expr 0x561c9dddf780, return datatype 'N'
8562.569580401:main Q:Reg/w0 : if condition result is 0
8562.569586028:main Q:Reg/w0 : END batch execution phase, entering to commit phase
8562.569592015:main Q:Reg/w0 : processBATCH: batch of 2 elements has been processed
8562.569598799:main Q:Reg/w0 : regular consumer finished, iret=0, szlog 0 sz phys 2
8562.569606353:main Q:Reg/w0 : DeleteProcessedBatch: we deleted 2 objects and enqueued 0 objects
8562.569612733:main Q:Reg/w0 : doDeleteBatch: delete batch from store, new sizes: log 0, phys 0
8562.569619238:main Q:Reg/w0 : regular consumer finished, iret=4, szlog 0 sz phys 0
8562.569624860:main Q:Reg/w0 : main Q:Reg/w0: worker IDLE, waiting for work.
8562.569640696:7f075a82dab0: thread created, tid 7, name 'in:imuxsock'
8562.569649200:7f075a82dab0: set thread name to 'in:imuxsock'
8562.569665945:imuxsock.c : --------imuxsock calling select, active file descriptors (max 4): 4
8562.569689335:7f075a843ab0: thread created, tid 6, name 'in:immark'
8562.569697227:7f075a843ab0: set thread name to 'in:immark'
