0

On a FreeBSD 12 server, I started to notice that pf is blocking out having Flags [FP.], the logs are flooded with something like this:

00:00:00.000004 rule 2/0(match): block out on bge0: 213.59.241.172.80 > 162.158.59.122.48894: Flags [FP.], seq 0:187, ack 1, win 65535, length 187: HTTP
00:00:00.000004 rule 2/0(match): block out on bge0: 213.59.241.172.80 > 141.101.98.92.16036: Flags [FP.], seq 0:187, ack 1, win 65535, length 187: HTTP
00:00:00.000004 rule 2/0(match): block out on bge0: 213.59.241.172.80 > 162.158.89.53.45136: Flags [FP.], seq 0:187, ack 1, win 65535, length 187: HTTP
00:00:00.000004 rule 2/0(match): block out on bge0: 213.59.241.172.80 > 162.158.88.60.43016: Flags [FP.], seq 0:187, ack 1, win 65535, length 187: HTTP
00:00:00.000004 rule 2/0(match): block out on bge0: 213.59.241.172.80 > 162.158.89.101.58320: Flags [FP.], seq 0:187, ack 1, win 65535, length 187: HTTP
00:00:00.000004 rule 2/0(match): block out on bge0: 213.59.241.172.80 > 162.158.179.50.21756: Flags [FP.], seq 0:187, ack 1, win 65535, length 187: HTTP
00:00:00.000004 rule 2/0(match): block out on bge0: 213.59.241.172.80 > 162.158.91.39.18516: Flags [FP.], seq 0:187, ack 1, win 65535, length 187: HTTP
00:00:00.000004 rule 2/0(match): block out on bge0: 213.59.241.172.80 > 162.158.90.202.25684: Flags [FP.], seq 0:187, ack 1, win 65535, length 187: HTTP
00:00:00.000004 rule 2/0(match): block out on bge0: 213.59.241.172.80 > 172.69.226.63.52316: Flags [FP.], seq 0:187, ack 1, win 65535, length 187: HTTP
00:00:00.000003 rule 2/0(match): block out on bge0: 213.59.241.172.80 > 162.158.90.202.25662: Flags [FP.], seq 0:187, ack 1, win 65535, length 187: HTTP
00:00:00.000005 rule 2/0(match): block out on bge0: 213.59.241.172.80 > 198.41.242.26.29508: Flags [FP.], seq 0:187, ack 1, win 65535, length 187: HTTP

Any idea of why this could be happening?

Rule 2 is

@2 block drop log all

Server running mainly HAproxy and these settings on /etc/sysctl.conf:

debug.debugger_on_panic=0
debug.trace_on_panic=1
kern.ipc.shmmax=2147483648
kern.ipc.somaxconn=32768
kern.panic_reboot_wait_time=0
net.inet.icmp.drop_redirect=1
net.inet.icmp.icmplim=10
net.inet.icmp.log_redirect=0
net.inet.icmp.maskrepl=0
net.inet.ip.accept_sourceroute=0
net.inet.ip.random_id=1
net.inet.ip.redirect=0
net.inet.ip.sourceroute=0
net.inet.tcp.blackhole=2
net.inet.tcp.drop_synfin=1
net.inet.tcp.fast_finwait2_recycle=1
net.inet.tcp.finwait2_timeout=1000
net.inet.tcp.msl=2500
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.recvbuf_inc=16384
net.inet.tcp.recvbuf_max=134217728
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.sendbuf_max=134217728
net.inet.udp.blackhole=1
security.bsd.see_other_gids=0
security.bsd.see_other_uids=0
security.bsd.see_jail_proc=0
security.bsd.stack_guard_page=1
security.bsd.unprivileged_proc_debug=0
security.bsd.unprivileged_read_msgbuf=0
net.inet.tcp.mssdflt=1460
net.inet.tcp.minmss=536
net.inet.tcp.syncache.rexmtlimit=0
net.inet.ip.maxfragpackets=0
net.inet.ip.maxfragsperpacket=0
net.inet.tcp.abc_l_var=44
net.inet.tcp.initcwnd_segments=44
kern.ipc.maxsockbuf=614400000
net.inet.tcp.syncookies=1
net.inet.tcp.tso=1
kern.random.fortuna.minpoolsize=256
net.inet.tcp.isn_reseed_interval=123

These are the pf rules after adding a pass before block (rule 2 now is rule 7)

scrub in all no-df max-mss 1440 fragment reassemble
block drop in log on ! bge0 inet6 from 2001:4ba0:85a3:105::/64 to any
block drop in log on bge0 inet6 from fe80::eeeb:b8ff:fe87:9514 to any
block drop in log inet6 from <__automatic_2bacaf44_0:7> to any
block drop in log on ! bge0 inet from 213.59.241.128/25 to any
block drop in log inet from 213.59.241.172 to any
pass quick from <allow:5> to any flags S/SA keep state (if-bound)
block drop log all
block drop quick from <banned:0> to any
block drop quick from <bruteforce:0> to any
block drop in log quick from no-route to any
block drop in quick on bge0 proto tcp all flags FPU/FSRPAUEW
block drop in quick on bge0 proto tcp all flags FSRPAUEW/FSRPAUEW
block drop in quick on bge0 proto tcp all flags FSRAU/FSRPAUEW
block drop in quick on bge0 proto tcp all flags /FSRPAUEW
block drop in quick on bge0 proto tcp all flags SR/SR
block drop in quick on bge0 proto tcp all flags FS/FS
pass in quick on bge0 proto tcp from any to any port = http flags S/SA keep state (if-bound)
pass in quick on bge0 proto tcp from any to any port = https flags S/SA keep state (if-bound)
pass in quick on bge0 proto tcp from any to any port = 2222 flags S/SA keep state (if-bound)
pass in quick proto ipencap all keep state (if-bound)
pass inet proto icmp all icmp-type echoreq keep state (if-bound)
pass inet proto icmp all icmp-type unreach keep state (if-bound)
pass proto ipv6-icmp all keep state (if-bound)
pass out quick proto tcp all flags S/SA keep state (if-bound)
pass out all flags S/SA keep state (if-bound)

For this line:

pass quick from <allow:5> to any flags S/SA keep state (if-bound)

Allow is a table, something like:

group1 = "10.0.0.4"
group2 = "5.19.152.31 95.116.0.173:"
table <allow> { $group1, $group2 }

Something that I notice is that even I have the pass before the block:

pass quick from <allow:5> to any flags S/SA keep state (if-bound)
block drop log all

Some "allowed" groups are beeing blocked, for example:

00:00:00.589761 rule 7/0(match): block out on bge0: 213.59.241.172.4567 > 5.19.152.31.61847: Flags [RP.], seq 2:81, ack 0, win 0, length 79

The rule 7 is the block drop log all, In this case, 5.19.152.31 is in the allow table but is in being blocked

Therefore wondering how to full allow/skip certain IP's, network ranges before getting block.

nbari
  • 558
  • 1
  • 9
  • 28
  • 1) Do you have there only one interface ? 2) does the state table have enough free entries ? – drookie Mar 09 '19 at 23:17
  • and I don’t like this MASSIVE sysctl tampering with default oid values around net.inet. Are you sure about what you’re doing here ? Because if you’re the author of these tweaks then you have to be experienced enough to deal with pf on your own; if you’re not the author of these then it’s a cargo cult. – drookie Mar 09 '19 at 23:20
  • the server has 4 interfaces, currently only using one `bge0`, regarding the sysctl settings any hint, I have being tunning `net.inet.tcp.msl` but don't think could justify the block out rule – nbari Mar 10 '19 at 06:20
  • Show ys traffic on other interfaces then, if they are up. – drookie Mar 10 '19 at 06:30
  • other interfaces are down – nbari Mar 10 '19 at 07:01
  • Show us the `pfctl --vvvs rules` output then. – drookie Mar 10 '19 at 07:46
  • @drookie, I updated the question also with some findings, for some reasons pf is blocking out traffic from an allowed defined table – nbari Mar 10 '19 at 08:29
  • What is the reason of `FPU/FSRPAUEW` and subsequent flags rules ? Are you trying to enhance the builtin pf sanity checks ? Why ? Do these rules have non-zero counts ? – drookie Mar 10 '19 at 09:20
  • If all of the interfaces except bge0 are down why using `if-bound` ? Where is the `set skip on lo0` rule (do you filter localhost ?) ? And so on. Your pf.conf looks acrually weird. Like extremely weird. The more I know about it the more questions I have. Rewrite it from scratch and see if it helps. I also recommend flushing out all the pseudo-tweaks from sysctl. Looks like you have absolutely no idea what they do. – drookie Mar 10 '19 at 09:23
  • `pfctl -vvvsr` don't show the skip, I indeed have it for the whole group `set skip on {lo}`, the `FPU/FSRPAUEW` help against some nmap scans but that blocks only incoming traffic, the question is more about the block in the outgoing traffic – nbari Mar 10 '19 at 09:29

2 Answers2

0

Looks like Pf's view of those TCP streams doesn't match the peers TCP/IP-stacks's.

I'd recommend …

  1. checking out whether state-mismatch is getting increased when you're seeing those entries in the logs, for e. g.:

    sudo pfctl -s info | fgrep -- state-mismatch

    of course, check other -s info counters as well, make sure you don't have states pruned due to mem. pressure, for e. g..

  2. getting rid of scrub altogether at least while you're debugging that issue.

If there's anything more than just log cluttering and-or you badly want that traffic to flow freely, not being ditched by Pf, you can add explicit pass … no state catching rule, tailored to your preferences of course.

poige
  • 9,448
  • 2
  • 25
  • 52
-1

I recommend starting with this configuration (yeah, I skipped mtu tweaks until your explanation why 1440 size is chosen), since yours is a total mess:

oif = "bge0"

# because you don't want to filter localhost
set skip on lo0

# because it's a default rule
block drop log all

# because you don't want to drop ICMP
pass proto icmp all

# or ICMP6
pass proto ipv6-icmp all

# okay then
block drop quick from <banned:0> to any
block drop quick from <bruteforce:0> to any

# because you need ssh (is the 2222 new port for ssh ? Jeezus, use blacklistd)
pass in on $oif proto tcp from any to any port 22

# because flags S/SA and keep state is the default
pass in on $oif proto tcp from any to any port { 80, 443, 2222 }

# because you need to configure both ends with ipinip, so why bother about states
pass in on $oif proto ipencap all

# because you want to allow all the locally-originated sessions
pass out all
drookie
  • 8,625
  • 1
  • 19
  • 29
  • What is the difference between using only `block in log all` vs `block log all` and need to use `pass out all`? – nbari Mar 10 '19 at 09:45
  • Well it's obvious - `block in log all` blocks everything inbound, while `block log all` blocks outbound too. `pass out all` passes everything out locally-originated. Is there anyone from technical personnel around and who let you to the server console ? – drookie Mar 10 '19 at 10:17
  • All the blocks I have are related to a proxy, HTTP request are accepted then the proxy tries to connect to the remote backends (this from my understanding may not be `locally-originated`) by using `block in` solve the problem but I would like to better understand if the why of this – nbari Mar 10 '19 at 10:31
  • Short answer - no. No, you don’t need all of these rules. Since you don’t understand how pf is working, you need to start with something fresh and simple. Yet equivalent to what you have and not interfering with your services. You can use the configuration proposed or keep searching for answers, it’s up to you. – drookie Mar 10 '19 at 10:34
  • I am indeed asking because with your configuration I still have the problem, that's why I post the `sysctl` settings since I thought that they may be affecting the `pf` rules – nbari Mar 10 '19 at 10:37
  • Show me the tcpdump output from pflog0 then. – drookie Mar 10 '19 at 10:38
  • Nah, this isn’t a paid techsupport. – drookie Mar 10 '19 at 10:43
  • Anyway, I see many `TCP retransmission` I have a guess is related to the `net.inet.tcp.msl` – nbari Mar 10 '19 at 11:00
  • Flush out that nonsense and reboot. I see only two reasonable settings - `kern.ipc.shmmax` and `kern.ipc.somaxconn` but I doubt you need these values. – drookie Mar 10 '19 at 11:04
  • From the `tcpdump` I see `[block bge0/0] 80 → 28650 [RST, PSH, ACK] Seq=1 Ack=1 Win=0 Len=193` (https://imgur.com/532Szwq) one backend was returning 504 but output was blocked – nbari Mar 10 '19 at 11:30