0

I want to process all packets with their size going through our gateway server (running Debian 4.0).

My idea is to use tcpdump, but I have two questions.
The command I'm currently thinking of is tcpdump -i iface -n -t -q.

  1. Is it guaranteed that tcpdump will process all packets? What happens if the CPU is working to full capacity?
  2. The format of the output lines is IP ddd.ddd.ddd.ddd.port > ddd.ddd.ddd.ddd.port: tcp 1260. What exactly is 1260? I have the suspicion that it is the payload in bytes of the packet, which would be exactly what I need, but I'm not sure. It might be the TCP Window Size.

Or perhaps there is an even better way of doing this? I thought about a LOG rule in iptables, but tcpdump seems easier and I don't know whether iptables can log the packet lengths.

UPDATE:

It is now implemented in IpTables. Using a separate chain for every network segment I have something of a tree with depth 3.

The machine has a high soft interrupt load, especially in the evening (when most of our users are online), but up to now it was acceptable.

Thanks for your input.

Christoph Wurm
  • 299
  • 2
  • 12

3 Answers3

5

  1. tcpdump isn't guaranteed to process all packets. There's some buffering, but if the rate of packets crossing the network interface is faster than the rate at which your CPU can run them through tcpdump, the kernel starts dropping packets. The higher the demand on the CPU, and the higher the network traffic rate, the higher the propensity for dropping (it's impossible to be specific, you'll have to test this on your systems to find out where the drop threshold is).

  2. Offhand, I don't know.

As for better ways, the term you want is "traffic accounting". This is built into IPTables, so any modern Linux distro should support it out-of-the-box. In short, a few simple "pass-through" IPTables rules can give you the total bytes transfered, in real time, for just about any specified traffic types (broken down by proto, port, IP, etc., or not) that you want.

There's a great walk-through, with specific commands, here: http://www.catonmat.net/blog/traffic-accounting-with-iptables

This should be much more lightweight and reliable than tcpdump, since Netfilter handles it entirely in the kernel, and the kernel has the packet length info, anyway.

Ryan B. Lynch
  • 2,026
  • 1
  • 12
  • 13
  • In initial tests tcpdump causes the kernel to drop about 3-5 % of all packages. I initially thought of the iptables solution, the advantage I saw in tcpdump was the ability to process packets in real time. However, I guess it might be enough to use iptables chains and read the byte count every 60 seconds or similar. Thanks for your advice. – Christoph Wurm May 05 '10 at 17:35
  • It's not real time, but you should be able to poll the byte counts on a much shorter interval without using much CPU. Once you have your chains set up, try this in Bash: `while true; do ; sleep 0.1; done`, and then run 'top' in another terminal window to monitor how high the load gets. If it's too busy, try 'sleep 1' instead, and just keep increasing the sleep interval until you get to a tolerable level. I would be interested in hearing what your hardware is, and what it can tolerate. – Ryan B. Lynch May 07 '10 at 00:36
  • I load-tested it today. When I create a chain with over 9000 rules (over 2000 users, each with a subnet and a VPN IP, two rules - one for Source, one for Destination IP checking - for each IP or IP range) ksoftirqd takes 100 % CPU and the kernel starts massively dropping packets. It looks as if the iptables approach is doomed. Even if I optimize the rules so that heavy users are at the top I won't be able to stop the dropping. The internet became totally unusable during the test. – Christoph Wurm May 23 '10 at 23:12
  • @Legate: IPTables isn't designed to handle 9000+ rules in a single chain, so no, I wouldn't expect that setup to work. (Not your fault, I didn't imagine your workload was 2000 +users.) To streamline things, try creating a separate "user-defined chain" (google for docs) for each user, and use '--jump' rules to route packets to the per-user chains. The point is to cut down the number of rules the kernel has to check every time it routes a packet: Currently, it's 9000, but you can cut that down a lot. Post some rule samples, maybe I can help w/ the syntax. – Ryan B. Lynch May 27 '10 at 14:43
0

To complete Ryan B. Lynch's excellent response:

  1. The output format of tcpdump is documented in the man page (section "OUTPUT FORMAT", subsection "TCP Packets"). The example you mention does not look like tcpdump TCP formatting so I cannot explain what is 1260. The normal output is something like 192.168.2.1.38528 > 192.168.2.25.22: P 1649:2241(592) ack...
  2. The option -e may be useful because it prints the characteristics of the link-level (layer 2) header, including the length of the packet (at least on Ethernet). 19:12:32.238240 00:1c:23:00:6b:7f > 00:1e:8c:76:29:b6, ethertype IPv4 (0x0800), length 66: 192.168.2.25.22 > 192.168.2.1.58759: . ack 2369...
bortzmeyer
  • 3,941
  • 1
  • 21
  • 24
0

I think you should clarify what you mean by 'all packets with their size going through our gateway'. Do you just want a total ip traffic counters? Like 5678898 bytes passed through router during last hour. In this case you can just set up a couple of iptables rules and use its counters. Or do you need to see a size of every packet?

If you're into heavy traffic accounting, take a look at NeTraMet or pmacct projects.

thor
  • 658
  • 1
  • 7
  • 18
  • I need to account the traffic each of our users makes. I don't need the byte count to be 100% accurate, but the fault margin shouldn't be too high. – Christoph Wurm May 23 '10 at 23:15