logrotate, syslog-ng and loosing log

Question

Background

1. Apache server running on a machine and producing logs into /var/log/httpd/error_log
2. Using syslog-ng to send log to a port 5140
3. Eventually it will be consumed by kafka producer to be send to a topic
4. logrotate rotates /var/log/httpd/error_log every hour

Settings

syslog-ng.conf

options {                                                                                                                             
    flush_lines (0);                                                                                                                
    time_reopen (10);                                                                                                               
    log_fifo_size (1000);                                                                                                          
    long_hostnames (off);                                                                                                           
    use_dns (no);                                                                                                                   
    use_fqdn (no);                                                                                                                  
    create_dirs (no);                                                                                                               
    keep_hostname (no);                                                                                                             
};

source s_apache2 {
    file("/var/log/httpd/error_log" flags(no-parse));
}


destination loghost {
    tcp("*.*.*.*" port(5140) template("$MSG\n")); 
}

logrotate.hourly.conf

/var/log/httpd/error_log {
  copytruncate
  rotate 4
  dateext
  missingok
  ifempty
  .
  .
  .
}

Log Consumption

 nc -k -l 5140 | grep "STATS:" | java program

This java program is essentially a kafka-producer and sends data to kafka cluster

Problem

Between the logrotate initiating copytruncate and syslog-ng reading data from /var/log/httpd/error_log, some data is sent to this file but is not captured by syslog-ng and hence is not pushed to kafka-cluster

Is there any sane way to read logs continuously without losing from a file which is rotated by logrotate periodically ?

Answer 1

Does logrotate reload syslog-ng on rotating the log? Without it syslog-ng learns about the truncation of the file slower.