Python subprocess: stderr only saving the first line. Why?

Question

I am running tcpdump from within Python and I would like to know how many packets are dropped by the kernel.

When run on a command line, tcpdump looks like this:

me@mypc:$ sudo tcpdump -w myPackets.cap -i eth0 ip
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes
^C28 packets captured
28 packets received by filter
0 packets dropped by kernel

This is how I call tcpdump in my Python script:

f_out = open("tcpdumpSTDOUT", "w")
f_err = open("tcpdumpSTDERR", "w")
tcpdumpProcess = subprocess.Popen(['tcpdump',
                        '-w', 'myPackets.cap', '-i', 'eth0', '-n','ip'],
                        stdout=f_out,
                        stderr=f_err)
# a few seconds later:
tcpdumpProcess.kill()
f_in.close()
f_out.close()

Now, if I look at tcpdumpSTDERR, I only see the first of the usual output lines:

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

Where's all the rest?

EDIT I tried a different approach:

>>> myProcess = subprocess.Popen("tcpdump -w myPackets.cap -i eth2 ip",  shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> myProcess.communicate()

Then I killed tcpdump from a different shell, and the output of commnunicate() was displayed:

('', 'tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes\n')

... still the first line only!

EDIT 2 Interestingly:

>>> import shlex
>>> a = subprocess.Popen(shlex.split("tcpdump -w myPackets.cap -i eth2 ip"),  stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> a.terminate()
>>> a.communicate()
('', 'tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes\n221 packets captured\n221 packets received by filter\n0 packets dropped by kernel\n')

Have you tried output redirection within the `subprocess` call? i.e. `tcpdump ... > tcpdumpSTDOUT` — inspectorG4dget, Jul 16 '13 at 17:00
I just did. Think about how you would do I/O redirection on the shell and do that exactly in your `subprocess` call. So on the shell, when you call `tcpdump -w myPackets.cap -i eth0 ip`. Actually, re-reading your post now, I see that `sudo` is not part of your call in `subprocess`. Are you running the python script as root? — inspectorG4dget, Jul 16 '13 at 17:05
Yes, I am already root when I run the script. I didn't get what you are saying, though. I thought that redirection was happening already when I called `subprocess.Popen()` with the parameters `stdout=f_out` and `std_err=f_err`. What am I missing here? — Ricky Robinson, Jul 16 '13 at 17:15
Yes, that /should/ happen. However, it seems to not be happening. This is why I suggest `subprocess.check_call("tcpdump -w myPackets.cap -i eth0 ip &> path/to/file", shell=True)`. This way, you'll know for sure whether the problem is with tcpdump or some quirk in the way `subprocess.Popen` is being used — inspectorG4dget, Jul 16 '13 at 17:23
When I run the above line and then kill tcpdump with `ctrl+C`, the text output is written on screen and the file where redirection was supposed to happen remains empty. — Ricky Robinson, Jul 16 '13 at 17:39
Hmm.. you might need to spawn a thread, call `subprocess.Popen` in that thread and send it a `^C` from your parent process after some time. Then, open `sys.stdout`, seek to 0 and read. This a very dirty hack, so I'd recommend a cleaner approach if you have one (sadly, I'm not able to think of one at the moment) — inspectorG4dget, Jul 16 '13 at 17:48
In the end the only problem was that I was doing `.kill()` on the process instead of `.terminate()`. My fault! — Ricky Robinson, Jul 18 '13 at 12:18

score 1 · Answer 1 · answered Jul 17 '13 at 10:15

1

Use proc.terminate() instead of proc.kill():

import shlex
import subprocess
import time

with open("tcpdumpSTDERR", "wb") as f_err: # close the file automatically
    proc = subprocess.Popen(shlex.split("tcpdump -w myPackets.cap -i eth2 ip"),
                            stderr=f_err)
time.sleep(2)    # wait a few seconds
proc.terminate() # send SIGTERM instead of SIGKILL
proc.wait()      # avoid zombies

answered Jul 17 '13 at 10:15

jfs

399,953
195
994
1,670

You preceded me by 39 seconds :) One thing: why should SIGTERM generate zombies? I thought that only SIGKILL could. – Ricky Robinson Jul 17 '13 at 10:18
@RickyRobinson: the signal doesn't matter. You should call `proc.wait()` even if the process finishes by itself without any external signals. – jfs Jul 17 '13 at 10:22

score 1 · Accepted Answer · answered Jul 17 '13 at 10:15

The problem was that I called kill() on the process instead of terminate(). With the latter, all messages get stored in whatever I specified as stderr (tcpdump, for some reason, writes to stderr and not to stdout).

So, in case it might help others, I decided to redirect stderr to subprocess.PIPE and parse the string directly in Python:

>>> tcpdumpProcess = subprocess.Popen(['tcpdump',
                        '-w', 'myPackets.cap', '-i', 'eth0', '-n','ip'],
                        stdout=subprocess.PIPE,
                        stderr=subprocess.PIPE)
>>> tcpdumpProcess.terminate()
# stdout in [0], stderr in [1]
>>> tcpdump_stderr = tcpdumpProcess.communicate()[1]
>>> print tcpdump_stderr
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes
40 packets captured
40 packets received by filter
0 packets dropped by kernel

Python subprocess: stderr only saving the first line. Why?

2 Answers2