1

I'm under continuous floods at time that are generally of lower magnitudes. I've observed my system often to become totally unresponsive on its network while I observed this in the logs,

kernel: [455951.513204] __ratelimit: 1771 callbacks suppressed
kernel: [455951.513207] dst cache overflow

I've researched a lot on this but did not find any suitable answer. I'm running the kernel version 2.6.32-5-amd64 by the way on a Debian 6 system.

John Elvy Adam
  • 23
  • 2
  • 7

1 Answers1

2

This page (http://bluecoat.force.com/knowledgebase/articles/Solution/CB-IPdestinationcacheoverflowdstcacheoverflowipdstcachemessage) has instructions for increasing the cache associated with this message - quoted below.

However, if you are "under continuous floods" increasing this cache may only make the flooding problem worse, and it is probably better to address the problem before it gets to your system. Perhaps by dropping offending traffic at your router/firewall.

Use the following procedure to assess the situation and change the size of the dest cache. All of the following commands assume that you are connected to the VAP console..

Check the current situation: cat /proc/slabinfo |grep ip_dst_cache

And settings: cat /proc/sys/net/ipv4/route/max_size

Set a new maximum value (ie. 2621440) and verify that it was accepted: echo 2621440 > /proc/sys/net/ipv4/route/max_size; cat /proc/sys/net/ipv4/route/max_size

Check the current situation again: cat /proc/slabinfo | grep ip_dst_cache

After a while, the CPU load should go down.

Another issue that a customer might experience is that the Firewall connections table is full. When this happens, more memory is required from the system.

If the firewall is not deployed in the core part of network this issue shouldn't occur. If the customer still sees this condition, it might be caused by somebody trying to spoof IP addresses in the internal network by doing a kind of nmap scan with spoofed IP addresses or something like that. For a typical data center in front of servers and/or a perimeter firewall it shouldn't be observed. The cause might be some kind of DoS/DDoS attack. Regardless of the source of the issue, the procedure above will solve the issue.

All sysctl parameters are loaded at boot time through /etc/init.d/network script. The command is:

sysctl -e -p /etc/sysctl.conf

This script starts before the Check Point process and that is why the changes don't survive a reboot.

When Check Point is installed this value is configured to 524288 when firewall starts by fwstart script. So, even though we change the parameter in the /etc/sysctl.conf file, and Linux configures it at boot time, when the firewall starts, this value is changed again. Then, if we just stop (cpstop) and start (cpstart) the firewall, these values would be changed again.

Check Point is changing this value - $ cd $FWDIR/bin $ grep -n max_size fwstart echo 524288 > /proc/sys/net/ipv4/route/max_size

To make sure the kernel will have the correct value after a reboot or firewall restart do the following:

  1. Configure files below to reflect the right value

    • /etc/sysctl.conf

  2. Disable line in fwstart script ($FWDIR/bin/fwstart)

    echo 524288 > /proc/sys/net/ipv4/route/max_size

NOTE: After applying the Check Point HFA or upgrading Check Point, the fwstart script might be overwritten.

To get real time changes, use this command:

$ sysctl -w net.ipv4.route.max_size = 2097152

Joe
  • 1,043
  • 8
  • 11
  • Thank you. I'll try that but how would continuous floods make the problem worse? – John Elvy Adam Apr 08 '15 at 02:33
  • There is a resource limit somewhere. With your dst cache as it stands, that is your resource limit, but as you increase that, it opens up the possibility that other resources will run out (RAM, CPU, bandwidth etc). I'm not saying don't raise this cache. Just that you want to be careful, and if you are being flooded with illegitimate traffic, you want to stop that traffic rather than give it more resources to eat up. – Joe Apr 08 '15 at 15:34
  • 1
    How do I stop this traffic from entering at all? I do not have DDOS protection so the traffic will enter the system at all times. However, I've blocked them using "iptables". Maybe I can stop connections from being tracked with some kind of pre-firewall? – John Elvy Adam Apr 08 '15 at 16:01
  • 1
    The only way would be by using another device in front of the server so that iptables doesn't have to handle it. In the ideal, this would be a DDOS protection device, but they are quite expensive. A hardware firewall would probably suffice. – Joe Apr 08 '15 at 16:12
  • Should I lower the size of my conntrack modules? I think the size is so big that it makes me run out of resources and memory that makes the system hangup in the event of an attack? How do I find out the optimal size for these parameters? – John Elvy Adam Apr 09 '15 at 09:58
  • It's hard to say. If you need the functionality offered by those modules, then you need it. If you don't, then get rid of it. If you are able, it might be better to just add more RAM to your system. It's relatively cheap to do so in most cases. – Joe Apr 09 '15 at 15:43
  • I also see very high CPU usage for processes such as "events/0" or "events/2" during these attacks. Other than "dst cache overflow" problem, I observe this in my logs as well, http://paste.ubuntu.com/10793330/ - I have made the size of the conntrack parameters smaller smaller now and the system seems to work better. When the size was large, I used to observe this in the logs http://paste.ubuntu.com/10793343/ – John Elvy Adam Apr 10 '15 at 18:06