1

I'm trying to pipe syslogs to a perl script via syslog-ng but not all of the syslog entries make it through - maybe 1 in 3 actually happen.

I've looked all over the place and can't find anyone remotely having the problem that I have. It seems so simple but I can't find the answer!

Here's my syslog-ng setup:

source s_1 { tcp(port(514)); };

destination d_zen { program("/tmp/zen.pl"); }; 

log { source(s_1); destination(d_zen); }; 

and here's my perl script:

#!/usr/bin/perl

use strict;
use warnings;

$|=1

my $filename = "/tmp/zen.log";
open(my $fh, '>>', $filename) or die "could not open file '$filename' $!";

while ( <STDIN> ) {

    print $fh <STDIN>."\n";

};

any thoughts?

ticontas
  • 113
  • 1
  • 5
  • fyi i've tried flush_lines (1) and it doesn't help. I'm also concurrently logging to other places and logs are happening appropriately, only this destination is spotty. – ticontas Jun 04 '15 at 03:30
  • Is this possible for you to tell how frequent are logs getting generated ? If logs are frequent then i will suggest using unix domain socket in between. It worked for me very well in my use case. – shivams Jun 04 '15 at 05:21
  • It's actually not very frequent, but can come in blocks of 5 at a time. Maybe once every 10 seconds I'll get 1-5 log entries. I'm not sure I understand domain unix sockets totally is this something I would use inside of my perl script or somewhere else? – ticontas Jun 05 '15 at 19:25
  • Can you please place your code with while loop here. Also i do not think you need any kind of socket to work with. Socket is placed between syslog and perl script so flow is like `syslog ==> socket ==> perl script`. Rate of logs in your case is not very much. – shivams Jun 05 '15 at 20:23
  • No relation with rate limiting in the conf file ? –  Jun 16 '15 at 22:41
  • @igael - nope,it works with `flush_lines(1)` and `flush_lines(0)` (which is default) just the same. Also, I found no need for `$|=1` either - this is probably because I'm not passing that much data at a time. – ticontas Jun 18 '15 at 01:54
  • The key was the `$_` variable. I found out that perl in its awesomeness pipes each line in a `` string into `$_` automagically which worked perfectly for my case. Since my `` wasn't coming from a file but a pipe/stream I used `<>` in my while loop. – ticontas Jun 18 '15 at 02:00

2 Answers2

0

Is your Perl line buffer disabled?

According to the syslog-ng manual, it can cause some problems:

"Certain external applications buffer the log messages, which might cause unexpected latency and other problems. For example, if you send the log messages to an external Perl script, Perl uses a line buffer for terminal output and block buffer otherwise. You might want to disable buffering in the external application."

Also, I don't know how your script reads the incoming messages (I don't now perl), but I think it should use a loop to keep reading incoming messages. So syslog-ng should start the script once at startup, and it should keep running and processing messages.

HTH,

Regards, Robert Fekete

Robert Fekete
  • 557
  • 3
  • 5
  • I've tried turning off buffering with `$|=1` also I actually had a `while ( ) { }` loop but no dice! Thanks for the insights. – ticontas Jun 05 '15 at 19:19
  • Stupid idea, using while ( 1 ) instead of while ( )? (Or something that is always true, to avoid that the script exits) – Robert Fekete Jun 08 '15 at 07:17
0

I figured out the problem. My while loop wasn't built properly:

#!/usr/bin/perl

$|=1;

use strict;
use warnings;

my $filename = "/tmp/zen.log";
open(my $fh, '>', $filename) or die "could not open file '$filename' $!";

my $my_string;

while( <> ) {
    $my_string .= $_;
    print $fh "$my_string\n";
};
ticontas
  • 113
  • 1
  • 5