Does rsyslogd send new information or the entire log file upon restart

Question

I'm trying to use rsyslog on a Debian machine to send log data to an external server.

The log files that I'm interested in have around 10GB of historical data.

When I started on configuring rsyslog I was under the impression that it's going to start sending all the new log entries to the server but I was told by the maintainer of the destination server that I have sent more than 10GB of data. I restarted rsyslogd a few times.

I'm wondering if by default it sends all the logs from the beginning of time instead just the new ones?

Here's what my config file looks like

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/spool/rsyslog

####################
# Nginx Access Log #
####################
# Input for Nginx Access Log
$InputFileName /var/log/nginx/myapp.access.log
$InputFileTag nginx-access
$InputFileStateFile stat-nginx-access #this must be unique for each file being polled
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Add a tag for file events
$template NginxAccessFormat,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% [SecretKey@41058 tag=\"myapp-nginx-access\"] %msg%\n"

# Send to Loggly then discard
if $programname == 'nginx-access' then @@server.loggly.com:514;NginxAccessFormat
if $programname == 'nginx-access' then ~

######################
# Nginx Access Error #
######################
# Input for Nginx Error Log
#$InputFileName /var/log/nginx/myapp.error.log
#$InputFileTag nginx-error
#$InputFileStateFile stat-nginx-error #this must be unique for each file being polled
#$InputFileSeverity info
#$InputFilePersistStateInterval 20000
#$InputRunFileMonitor

# Add a tag for file events
#$template NginxErrorFormat,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% [SecretKey@41058 tag=\"myapp-nginx-error\"] %msg%\n"

# Send to Loggly then discard
#if $programname == 'nginx-error' then @@server.loggly.com:514;NginxErrorFormat
#if $programname == 'nginx-error' then ~

#############
# PHP Error #
#############
# Input for PHP Error Log
#$InputFileName /var/log/hhvm/error.log
#$InputFileTag php-error
#$InputFileStateFile stat-php-error #this must be unique for each file being polled
#$InputFileSeverity info
#$InputFilePersistStateInterval 20000
#$InputRunFileMonitor

# Add a tag for file events
#$template PhpErrorFormat,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% [SecretKey@41058 tag=\"myapp-hhvm-error\"] %msg%\n"

# Send to Loggly then discard
#if $programname == 'php-error' then @@server.loggly.com:514;PhpErrorFormat
#if $programname == 'php-error' then ~

Here's the version info of rsyslogd

rsyslogd -v
rsyslogd 8.4.2, compiled with:
    FEATURE_REGEXP:             Yes
    GSSAPI Kerberos 5 support:      Yes
    FEATURE_DEBUG (debug build, slow code): No
    32bit Atomic operations supported:  Yes
    64bit Atomic operations supported:  Yes
    memory allocator:           system default
    Runtime Instrumentation (slow code):    No
    uuid support:               Yes
    Number of Bits in RainerScript integers: 64

See http://www.rsyslog.com for more information.

Is your `$InputFileStateFile` getting updated properly? I have never used this functionality before. But my reading of the docs http://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html seems to suggest that it will read it all once, then save the state, and then always continue from the last processed point. But if your state file isn't being saved/created properly, then it would probably be resending everything at each restart. — Zoredache, Nov 07 '15 at 00:32
I couldn't figure this out because it has to be watched while sending the data and that'll cause an incident but now I'm more confident that it is the case that rsyslog does send whatever it hasn't sent before which means the entire file for the first time. — user893730, Nov 09 '15 at 19:20

score 1 · Accepted Answer · answered Nov 09 '15 at 12:57

Yes, by default it will start getting logs from the beginning, though it will write a state file to remember where it left off. I assume you can remove the state file, rotate the file in question and start fresh if you only want the new logs.

Also, I would suggest to upgrade rsyslog to the latest stable (now 8.14.0) as a lot of imfile-related fixed have been added since 8.4.2 (see changelog for more details).

Does rsyslogd send new information or the entire log file upon restart

1 Answers1