0

I am working on migrating a large syslog farm from syslog-ng 2.0.9 to 3.8.1.

I have incoming datagrams that look like this:

<14>Aug 14 21:28:49 pa01.foo.com 1,2017/08/14 21:28:49,009401031978,THREAT,url,...

The actual message starts just after the pa01.foo.com hostname.

On 2.0.9, I just used "$MSG" as my output template, and the output file would get:

1,2017/08/14 21:28:49,009401031978,THREAT,url,...

which is what I want.

On 3.8.1, it's deciding that the actual message starts is in a different place, my output looks like:

21:28:49,009401031978,THREAT,url,...

I lost the leading 1,2017/08/14.

I discovered that setting flags(no-parse) at the source and adding the syslog parser back into the pipeline with parser { syslog-parser(); }; works correctly. In summary:

source net_src1 { network( transport("udp") port(4514) flags(no-parse)); };
source net_src2 { network( transport("udp") port(5514)); };
source net_src3 { syslog( transport("udp") port(6514)); };

filter f_test { netmask(127.0.0.1/32); } ;

destination d_test1 { file("/data/syslog/test/1" template("$MSG\n")); };
log { source(net_src1); filter(f_test); parser { syslog-parser(); }; destination(d_test1); flags(final); };

destination d_test2 { file("/data/syslog/test/2" template("$MSG\n")); };
log { source(net_src2); filter(f_test); destination(d_test2); flags(final); };

destination d_test3 { file("/data/syslog/test/3" template("$MSG\n")); };
log { source(net_src3); filter(f_test); destination(d_test3); flags(final); };

If I send to port 4514 which disables the parse with flags(no-parse) and then adds it back in with parser { syslog-parser(); };, all is well. Traffic sent to ports 5514 or 6514 comes out missing the leading 1,2017/08/14.

It seems the syslog-parser is parsing differently based on whether it is implicitly or explicitly put into the pipeline.

Is there a way to get the desired behaviour without have to use flags(no-parse) and explicitly adding the parser back in?

Douglas Wegscheid
  • 3
  • 2

1 Answers1

0

it seems to me that the message does not comply to the syslog RFC, and the program field is missing, I'd say that's the root cause. If such a thing happens, syslog-ng uses some heuristics to try and parse the message. The missing 1,2017/08/14 part of your message is probably parsed into the $PROGRAM field. As a workaround, check if using the following template solves the problem: template("$PROGRAM $MSG\n"))

Alternatively, if the above does not work, then check if you can use a filter to separate the problematic messages (for example, based on the sender host), and you can use a junction to use the no-parse flag only on these messages. You can find an example for a similar case in the docs: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/junctions.html

Nevertheless, it is strange that the parser behaves differently when used explicitly and implicitly, AFAIK the same code is running in both cases. Please open an issue about it on the syslog-ng github page

Regards, Robert

Robert Fekete
  • 552
  • 1
  • 3
  • 6
  • I want to remove the no-parse flag from the src (that will break a lot of other destinations).; I don't understand how https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/junctions.html can work without the no-parse in src. – Douglas Wegscheid Aug 17 '17 at 12:55
  • ...and you are correct. The missing text was getting parsed into $PROGRAM. I am reluctant to put a bug report in about the inconsistency; they may fix it so that neither way works! – Douglas Wegscheid Aug 17 '17 at 12:59
  • Hi, the junctions part does use the no-parse flag, it's just a trick to use a filter to handle messages that need no-parse and regular messages together. Anyway, I'll notify our developers about this inconsistency, so they can fix the problem to work as expected :) – Robert Fekete Aug 23 '17 at 08:48
  • thanks. I still don't understand how I can use a junction for this without setting no-parse on the src, which is what I am trying to avoid doing. – Douglas Wegscheid Aug 24 '17 at 13:10