-1

I'm writing a custom bandwidth monitoring system for my network. I want to be able to track bandwidth used by each IP on the internal (NAT) subnet. I already have all of the infrastructure coded to handle displaying logs, sending warnings, etc. and also do any needed statistical analyses.

Right now, I'm using a rather ugly method: I setup some rules in iptables that do nothing, and then periodically run iptables -vnxL from Python and parse out the byte counters. This is probably quite efficient comparatively (we're not parsing every single packet) but it leaves the possibility open for packets to be missed, e.g. if the system reboots, the counters will be reset (Especially if it reboots uncleanly), and if I have any cause to manually modify iptables rules, for some reason on my system the counters get reset from that act as well.

I've read about the ULOG target, which I could use to basically send all routed packets into userland space, where I could grab the packet sizes and add them to the counters stored in my database. I see two possible options:

  1. Somehow implement a ulogd equivalent in Python, that simply sees all packets going across the interfaces, parses the size and addresses out of each packet manually, and handles all the math. I feel this may end up being inefficient, since I'd be doing per-packet parsing in Python, so in the event of high-packet applications (VOIP and so on) we could end up with Python lagging behind or missing packets.
  2. Run ulogd itself, but somehow have it only log the IP addresses and the byte count for each packet into a file; then parse this file with python periodically. Each time python parses the file, I would want it cleared.

The reason I'd like to do this in Python rather than iptables is that I get more flexibility. Since I'm going to be parsing IP addresses and ports and all of that manually, I could statistically do things like "what percentage of traffic is HTTP". Right now, using the iptables method, all I can do is determine "how much is each host using."

Method 2 is where I'm leaning with this. Here's the methodology I'd like to see:

  1. The Python process starts. It somehow captures a snapshot of the current ulogd log file.
  2. ulogd is restarted with an empty log file. This is so that we don't miss any packets while Python is processing the snapshot of packets, and so Python doesn't have to keep track of where in the logfile it last reached.
  3. Python now has the freedom to parse the packets in the log snapshot at its own pace.

Problems I see:

  1. How does Python obtain this "Snapshot"? The only method I could think is: terminate ulogd, copy the file, delete or truncate the original file, then restart ulogd. This can take some time, and on modern connections those few seconds could result in megabytes of lost tracked data.
  2. If I work by reading the same file ulogd is writing to, I feel like some odd things could happen if I am reading from the file that ulogd has a write handle to. Things like write caching seem like we could still miss packets.

Could anyone give me, or direct me to somewhere to get, some advice on this project?

F

fdmillion
  • 4,823
  • 7
  • 45
  • 82
  • 1
    Why reinvent the wheel? Just use netflow... – EEAA May 03 '14 at 16:55
  • Isn't netflow part of cisco routers? I'm out for an all-free solution, and if I end up putting together something I was considering GPL'ing it anyway. – fdmillion May 03 '14 at 17:29
  • There are Linux implementations of netflow. – EEAA May 03 '14 at 17:32
  • It's worth looking into, but I'd still like advice on the critical point: how to access a log file from Python periodically while not interfering with the logging process and also not losing any data. This goes beyond packet logging, and the same skill would be useful in other ways. The bandwidth is just one specific (and major) way I need to understand it. – fdmillion May 03 '14 at 17:38
  • Nothing is preventing you from reading a log file. What's the issue here? Examine how tail and possible logstash works. – EEAA May 03 '14 at 17:42

1 Answers1

1

There are two methods that I can think of:

  1. Parse rotated files only. E.g. rotate ulogd files and parse them after they are rotated
  2. Parse the live file

At the end you need to develop for #1 and maybe extend for #2. I.e. you need to take into account log rotation, remember the latest rotated file that you parsed and in the next run parse from that point on. As such you'll definitely need to keep state between invocations.

For #2 you can follow the logtail approach. For example, keep the inode number of the live file and the latest offset you've parsed to. If the inode number changes (or the size becomes less than the offset) then you need to parse the whole file. Otherwise you can parse from that point ahead.

If you don't need real time data then working with rotated files should work quite well. IF however you want something more realtime then parsing the current log file a'la logtail should be quick.

Alternatively you can try a database backend for ulogd and solve the problem in there.

V13
  • 853
  • 8
  • 14