I have use case where my daily batch processing application (spring batch java application using remote partitioning) is deployed to 4 servers and application creates log in log file in daily batch folder.
e.g. batch with batch id 2014-07-15 stores log in /var/log/myapp/2014-07-15/batch.log
I want to collect logs from all servers and collect in one single log file on master (master is one of the 4 servers).
I am trying to use Rsyslog for this purpose. But if there is any other better solution, kindly suggest.
1) How do I read logs from /var/log/myapp/2014-07-15/batch.log file whose path is dynamic and send to master ?
2) How to collect logs coming from all servers of myapp application and store it in log file /var/log/myapp/2014-07-15/batch.log on master server (log file path will be same on all server for the batch).
I have reffered to the documentation and guides here
http://www.rsyslog.com/guides-for-rsyslog/
but can't understand how to send log read from file to other server ? How to use dynamic paths though I found
$template DynFile,"/var/log/%HOSTNAME%/%programname%.log"
I am new to rsyslog, so its bit difficult to put together all this info to achieve my use case. It would be great help if someone could guide me to achieve this.
Something like this ?
#On master server
$ModLoad imtcp
$InputTCPServerRun 10514
$template DynFile,"/var/log/$now/batch.log"
# where $now The current date stamp in the format YYYY-MM-DD
# referred this - http://ftp.ics.uci.edu/pub/centos0/ics-custom-build/BUILD/rsyslog-3.19.7/doc/property_replacer.html
if syslogtag isequal "myapp" then ?DynFile
#On slaves machins
$template DynFile,"/var/log/$now/batch.log"
module(load="imfile" PollingInterval="10")
input(type="imfile" File=?DynFile Tag="myapp" StateFile="/var/spool/rsyslog/statefile1")
. @@[SERVER_IP]
Also how should be StateFile configuration as there will be different batch log files for different batches and so they should use different StateFile for each batch log file ? How do I configure Dynamic StateFile ?
