I have a server that is aggregating logs from various other servers. It mostly works great, however from time-to-time (right after a reboot, but not all reboots) it will decide that various UDP "connections" are in some weird state, and reject incoming packets.
Thing is, that doesn't make sense, because there are no "connections" when dealing with UDP!
Here's an example of what shows up in my firewall logs:
Shorewall:loc2fw:REJECT:IN=eth0 OUT= MAC=/*snip*/ SRC=10.x.x.4 DST=10.y.y.14 LEN=159 TOS=0x00 PREC=0x00 TTL=254 ID=37407 PROTO=UDP SPT=514 DPT=514 LEN=139
After running the command sudo conntrack -D -p udp to clear all UDP "connections" from conntrack, here's a log showing an incoming message from the same host:
Shorewall:loc_dnat:REDIRECT:IN=eth0 OUT= MAC=/*snip*/ SRC=10.x.x.4 DST=10.y.y.14 LEN=201 TOS=0x00 PREC=0x00 TTL=254 ID=50542 PROTO=UDP SPT=514 DPT=514 LEN=181
And here's what shows up in conntrack for this host before I ran the -D command:
udp 17 29 src=10.x.x.4 dst=10.y.y.14 sport=514 dport=514 [UNREPLIED] src=10.y.y.14 dst=10.x.x.4 sport=514 dport=514 mark=0 use=1
Here's the relevant bits from my Shorewall configuration for this port:
#ACTION SOURCE DEST PROTO DPT
SECTION NEW
REDIRECT:info loc 5000 tcp,udp 514
ACCEPT:info loc $FW tcp,udp 5000
And for completeness' sake, here's the resultant chain in iptables (this is after the conntrack -D -p udp command was run and the firewall had been properly accepting packets for a while):
Chain loc2fw (1 references)
pkts bytes target prot opt in out source destination
16328 1648K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
6428 386K ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 8 /* Ping */
1 64 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 /* SSH */
18 1080 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 /* HTTP */
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:5000 ctorigdstport 514
12M 6677M ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:5000 ctorigdstport 514
0 0 ~log0 tcp -- * * 0.0.0.0/0 0.0.0.0/0 [goto] tcp dpt:5000
0 0 ~log0 udp -- * * 0.0.0.0/0 0.0.0.0/0 [goto] udp dpt:5000
52 3088 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:9200
126M 32G Reject all -- * * 0.0.0.0/0 0.0.0.0/0
126M 32G LOG all -- * * 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 6 prefix "Shorewall:loc2fw:REJECT:"
126M 32G reject all -- * * 0.0.0.0/0 0.0.0.0/0 [goto]
As you can see, I'm using a REDIRECT rule to accept connections coming in on port 514 and send them instead to port 5000, where my log aggregator listens (was easier to accomplish this in the firewall than to jump through the hoops to allow an unprivileged user to open a privileged port).
I can't understand why UDP "connections" seem to get into this strange state where they get rejected by the default policy; I say this because they show up quite clearly in conntrack, and deleting them from there seems to "reset" everything and let the messages flow into my log aggregator again. At this point the only alternative I see is setting up cron to run my conntrack command shortly after reboot, but I'd much rather understand why this is happening and fix the cause rather than just implement this workaround.
