0

We have a problem where we set up a server running a service and it is capable of hundreds of simultaneous connections on port 3535 (arbitrarily assigned for this application). We have firewalld running on this near-end-host allowing connections from the far-end host and that is all working fine. The problem we ran into is the far-end-host is only able to establish a few connections at a time and it is taking upwards of 30 seconds to get those connections. The most we have seen on the near-end-receiving host is about 35 connections on average. We turned firewalld off and immediately it went to 850 connections and the far-end reported no problems and no delays when connecting and ran flawlessly for 15 minutes (until we turned firewalld back on).

We have a very simple rule set and are not doing any kind of throttling. Is there default throttling in firewalld that I need to disable or should I go to nftables and if so will it actually perform better or am I chasing a ghost? My ISP is not using VMWARE and so no external solution is available.

Thanks in advance. David

TekOps
  • 71
  • 4
  • Firewalld still uses nftables as backend (or it can be iptables). So unless you have a too heavy raw traffic that CPU can't withstand, you still need to understand what is happening, how are packets dropped when they should not etc. There's the *conntrack* part to also check. Bugs get corrected, so the kernel version also matters. – A.B Oct 16 '21 at 18:35
  • The CPU load when this is happening is .01 or .02 and responsive, etc. Even when the far-end-system is complaining of connection issues, we can telnet to the port from our remote site and it responds immediately. Regarding iptables vs nftables... the system has iptables on it. nftables package is not installed yet. I'm just trying to figure out if there is a default throttle or if iptables is too slow and nftables will solve the issue. We're not talking about insignificant performance variabtion. It basically went from 3% to 80% when we turned off the firewall. – TekOps Oct 16 '21 at 18:43
  • You'd need to supply your whole network settings (including firewalld, iptables (eg: iptables-save -c), interfaces, addresses, routes, routing rules in this question to get help. If it's something to keep secret, I'm not sure much can be told. + OS, kernel version iptables version etc. – A.B Oct 16 '21 at 19:02
  • I'm not asking for a specific "fix this". I'm asking for general knowledge. Because this is an extremely simple setup. Our entire firewalld ruleset is about 6 rules (allow this, this and this, block everything else). There is only one route, the default route. The primary questions are: (a) Is there some form of default throttling in firewalld/iptables and if so how do I turn off all throttling and (b) is nftables faster than iptables? Neither of those questions require specific detail. – TekOps Oct 16 '21 at 19:06
  • Oh ok. That's the first time I see this kind of issue reported around. I'm surprised this is related to firewall *performances*. I'd say there's something else, but you're the person knowning your setup. – A.B Oct 16 '21 at 19:09
  • It's definitely firewall. As soon as I turn it off, performance skyrockets. Most people think their systems are fast but they really have no idea. They also don't understand how to test effectively to isolate. Software firewalls require a *lot* of extra cpu horsepower to do packet filtering. We *always* opt for external hardware based firewalls and never have this problem. But this vendor insists on software firewalls local to the box. VERY MUCH appreciate everybody's input. – TekOps Oct 16 '21 at 19:19
  • Is this compatible with "The CPU load when this is happening is .01 or .02 and responsive"? – A.B Oct 16 '21 at 19:48
  • I'm not sure what you are asking. I think you're asking if the CPU Load is virtually zero when this is happening and the answer is yes. The server is extremely responsive so it's not a CPU load issue. That's why I suspected throttling. And I have ruled out the sending side and the recipient software because the issue goes away the install firewalld is disabled. – TekOps Oct 16 '21 at 19:54
  • I believe this is a performance limitation of firewalld. Others have also had the issue and it has gone unresolved: https://forums.centos.org/viewtopic.php?t=58673 – TekOps Oct 17 '21 at 03:13

3 Answers3

2

firewalld maintainer here.

Firewalld does not currently support acceleration (software fast path or hardware offload). However, I think that both could be added by using nftables flowtable infrastructure. Only a few NICs support flowtable offload though.

That being said, what you're describing sounds wrong. That's abysmal performance. Firewalld doesn't actually do the firewalling. It builds iptables/nftables rule sets and applies them. The iptables/nftables rule execution happens in the kernel/netfilter.

You may consider disabling some optional firewalld features. IPv6_rpfilter is known to have performance issues in scaled environments because it requires a FIB lookup. Also consider other things that may cause frequent rule updates or add a large amount of rules, e.g. fail2ban.

Edit 2023-06-07: firewalld now supports nftables flowtable. This may significantly improve forwarded traffic throughput.

erig
  • 21
  • 2
  • Thank you so much for the information and that makes sense. We ultimately went back to an external firewall solution. – TekOps Apr 13 '22 at 13:57
0

After much research it appears that nftables is not a "savior" here. It does not perform any better in this situation than iptables and so is not helpful.

The overall performance problem appears to be a limitation of the firewalld software firewall and must be addressed by an external firewall, etc. Other's have had the same issue and it has gone unresolved: https://forums.centos.org/viewtopic.php?t=58673

TekOps
  • 71
  • 4
0

You could try to copy the iptables / nftables rules generated by firewalld, disable firewalld, and then paste the iptables / nftables rules again, so you could infirm or confirm firewalld has some other limiting parameters.

If it is still capped, it is the generated rules and you could try to optimize the rules or build the from scratch with iptables / nftables.

If it isn't, you then it might be a firewalld limitation, or more probably a firewalld bug.

setenforce 1
  • 1,200
  • 6
  • 10
  • it's a limitation. these software firewalls are MASSIVELY slower than hardware firewalls. I know they are being implemented all over, but I think most users don't realize how slow they are. So we went back to hardware. – TekOps Apr 13 '22 at 13:53