syslog-ng configuration works in debug mode, but not without

Question

I am trying to run syslog-ng as an OCF resource in an HA cluster. I am encountering some terribly strange behavior - when I start a single instance in debug mode, the filters match and it forwards appropriately. However when I remove the debug flag, it only matches one of two filters. So, here's how it works (hostnames and IPs are redacted):

# pcs status
Cluster name: fwdr
Stack: corosync
Current DC: fwdr-secondary (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition with quorum
Last updated: Thu Sep  5 11:50:18 2019
Last change: Thu Sep  5 10:27:51 2019 by root via cibadmin on fwdr-primary

2 nodes configured
2 resources configured

Online: [ fwdr-primary fwdr-secondary ]

Full list of resources:

 virtual_ip     (ocf::heartbeat:IPaddr2):       Started fwdr-primary
 syslog-ng      (ocf::heartbeat-git:syslog-ng): Started fwdr-primary

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

syslog-ng.conf:

@version: 3.5

source incoming {
        udp(
            ip("VIP")
            port(514)
            flags(no-parse)
        );

        tcp(
            ip("VIP")
            port(514)
            flags(no-parse)
        );
};

filter pi_duplication {
    netmask("someip/32")
    or netmask("someip/32")
    or netmask("someip/32")

    ...a bunch of these...

    or netmask("someip/32")
};

destination dl {
    udp(
            "<REDACTED:dl hostname>"
            port(514)
            spoof_source(yes)
            template( "${MESSAGE}\n" )
    );
};

destination ci {
    tcp(
        "<REDACTED:ci hostname>"
        port(11468)
        template( "${MESSAGE}\n" )
    );
};

log {
    source(incoming);
    filter(pi);
    destination(ci);
};

Now, disable the syslog-ng resource:

# pcs resource disable syslog-ng

...
Full list of resources:

 virtual_ip     (ocf::heartbeat:IPaddr2):       Started fwdr-primary
 syslog-ng      (ocf::heartbeat-git:syslog-ng): Stopped (disabled)

Now start it in debug mode:

# syslog-ng -f /etc/syslog-ng/syslog-ng.conf --foreground --debug
Reading path for candidate modules; path='//usr/lib64/syslog-ng'

...

Compiling #unnamed sequence [log] at [/etc/syslog-ng/syslog-ng.conf:6]
  Compiling incoming reference [source] at [/etc/syslog-ng/syslog-ng.conf:6]
    Compiling incoming sequence [source] at [/etc/syslog-ng/syslog-ng.conf:3]
      Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:3]
        Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:4]
        Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:1]
  Compiling pi_duplication reference [filter] at [/etc/syslog-ng/syslog-ng.conf:6]
    Compiling pi_duplication sequence [filter] at [/etc/syslog-ng/syslog-ng.conf:1]
      Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:1]
  Compiling ci reference [destination] at [/etc/syslog-ng/syslog-ng.conf:6]
    Compiling ci sequence [destination] at [/etc/syslog-ng/syslog-ng.conf:5]
      Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:5]
        Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:5]
Compiling #unnamed sequence [log] at [/etc/syslog-ng/syslog-ng.conf:7]
  Compiling incoming reference [source] at [/etc/syslog-ng/syslog-ng.conf:7]
  Compiling dl reference [destination] at [/etc/syslog-ng/syslog-ng.conf:7]
    Compiling dl sequence [destination] at [/etc/syslog-ng/syslog-ng.conf:4]
      Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:4]
        Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:5]
Syslog connection established; fd='9', server='AF_INET(<REDACTED:dl host's IP>:514)', local='AF_INET(0.0.0.0:0)'
Running application hooks; hook='1'
Running application hooks; hook='3'
syslog-ng starting up; version='3.5.6'
Syslog connection established; fd='8', server='AF_INET(<REDACTED:is host's IP>:11468)', local='AF_INET(0.0.0.0:0)'
Syslog connection accepted; fd='16', client='AF_INET(<REDACTED:ci host's IP>:47876)', local='AF_INET(10.68.233.48:514)'
Incoming log entry; line='<REDACTED>'
Filter rule evaluation begins; rule='pi_duplication', location='/etc/syslog-ng/syslog-ng.conf:17:32'
Filter node evaluation result; result='not-match'
Filter node evaluation result; result='not-match'
Filter node evaluation result; result='not-match', type='OR'
Filter node evaluation result; result='not-match'
Filter node evaluation result; result='not-match', type='OR'
Filter node evaluation result; result='not-match'
Filter node evaluation result; result='not-match', type='OR'

...repeated...

Filter node evaluation result; result='not-match'
Filter node evaluation result; result='not-match', type='OR'
Filter node evaluation result; result='match'
Filter node evaluation result; result='match', type='OR'
Filter node evaluation result; result='match', type='OR'
Filter node evaluation result; result='match', type='OR'
Filter node evaluation result; result='match', type='OR'
Filter node evaluation result; result='match', type='OR'
Filter node evaluation result; result='match', type='OR'
Filter rule evaluation result; result='match', rule='pi_duplication', location='/etc/syslog-ng/syslog-ng.conf:17:32'

And so forth, every incoming line matches and is sent to both targets correctly. Traffic example, where source is the generating host, destination is the ci host, myvip is the VIP I listen on, and myrealip is the real IP of fwdr-primary:

# tcpdump -nn -i enp15s0f0 "port 514 or port 11468"                                       
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp15s0f0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:03:14.545138 IP source.48100 > myvip.514: Flags [P.], seq 2372587949:2372588369, ack 3533250116, win 29, length 420
12:03:14.545185 IP myvip.514 > source.48100: Flags [R], seq 3533250116, win 0, length 0
12:03:15.227043 IP source.48112 > myvip.514: Flags [S], seq 2965678208, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0
12:03:15.227107 IP myvip.514 > source.48112: Flags [S.], seq 280396112, ack 2965678209, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
12:03:15.260720 IP source.48112 > myvip.514: Flags [.], ack 1, win 29, length 0
12:03:15.260773 IP source.48112 > myvip.514: Flags [P.], seq 1:401, ack 1, win 29, length 400
12:03:15.260796 IP myvip.514 > source.48112: Flags [.], ack 401, win 237, length 0
12:03:15.262926 IP source.48112 > dlhost.514: SYSLOG local0.info, length: 400
12:03:15.263037 IP myrealip.41003 > destination.11468: Flags [P.], seq 2022253190:2022253590, ack 3273547315, win 229, options [nop,nop,TS val 3195491935 ecr 501321261], length 400
12:03:15.263175 IP destination.11468 > myrealip.41003: Flags [.], ack 400, win 235, options [nop,nop,TS val 501331496 ecr 3195491935], length 0

Now, re-enable the cluster resource:

# pcs resource enable syslog-ng

Now the network is silent:

12:08:24.610741 IP source.48240 > myvip.514: Flags [S], seq 3387574314, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0
12:08:24.610796 IP myvip.514 > source.48240: Flags [S.], seq 2754922833, ack 3387574315, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
12:08:24.644579 IP source.48240 > myvip.514: Flags [.], ack 1, win 29, length 0
12:09:01.941077 IP source.48240 > myvip.514: Flags [P.], seq 1:484, ack 1, win 29, length 483
12:09:01.941127 IP myvip.514 > source.48240: Flags [.], ack 484, win 237, length 0
12:09:01.942064 IP source.48240 > dlhost.514: SYSLOG local0.info, length: 483

(The packets directly from source > dlhost are where I spoof the source on the dl rule). In other words, the trace when running under the cluster shows that it only matches the dl rule, whereas when running in the foreground under debug mode matches both rules correctly! This is making it very hard to debug and I cannot figure out what's going on.

score 1 · Accepted Answer · answered Sep 05 '19 at 18:18

1

From your syslog-ng version I guess you are using syslog-ng from EPEL on RHEL/CentOS 7.

Can't verify it right now, but I have some distant memories, that when syslog-ng is started from systemd, then SELinux blocks network connections. You should check your audit logs, if there is anything related to syslog-ng.

My blog on the topic might help: https://www.syslog-ng.com/community/b/blog/posts/using-syslog-ng-with-selinux-in-enforcing-mode

answered Sep 05 '19 at 18:18

Peter Czanik

71
3

That was it! FYI, because of the port I was using `audit2allow` told me it was equivalent to setting the `nis_enabled` boolean, which allowed it out. Thanks! – matoro Sep 06 '19 at 12:52

syslog-ng configuration works in debug mode, but not without

1 Answers1