2

Short version:

Netstat works fine except when I redirect output to a file in NFS.

doesn't work: netstat -c > /nfs/mount/file

works: netstat > /nfs/mount/file

works: netstat -c

works: netstat -c > /tmp/file

Symptoms: empty file, no error messages.

This is using bash on Ubuntu 10.10

Longer version: I'm working on a project that requires we gather iostat and netstat statistics from all the nodes on a small cluster (~32 nodes) during test execution. We've written scripts to spin up iostat and netstat instances with the appropriate flags on each node, with each process writing to a file in a directory stored on an NFS share (each file name incorporates the host name it was written from).

The iostat scripts are working fine but we're seeing an issue with the netstat script.

For some reason, if I start a netstat process with the -c flag, indicating continuous output every one second, and pipe that to a file in the NFS directory, an empty file is created and no output is ever written to it.

If I do the same thing on the server that is providing the NFS storage, writing to the same directory (with the exception that it's local storage in this case) then every thing works out just fine. Also, if I write to a local file system on the host having issues (like /tmp/foo.txt) then everything is also fine.

Also worth noting, if I just run 'netstat', without the -c flag, and pipe that to a file stored on NFS, that works also.

So, there seems to be something a little hinky with netstat's continuous output flag that interacts with NFS in a manner that differs from other tools continuous outputs (like iostat's "-t 10" flag).

All hosts in this setup are running ubuntu 10.10.

Buck
  • 33
  • 6
  • What exactly is the command you're using? Unlike iostat and other programs, netstat's -c doesn't accept any kind of argument to change the number of seconds between runs, it always runs exactly every second. What shell are you doing the redirection in (it's the shell that is responsible for performing `>` redirection)? – DerfK Apr 05 '11 at 00:03
  • 1
    An `strace netstat -c` shows writes to standard out as would be expected. An `strace netstat -c > /nfs/mount/file` is significantly different (the `write` system call never appears). – Noah Watkins Apr 05 '11 at 00:15
  • For what it's worth, 'netstat -c > /nfs/mount/file' works fine for me on RHEL 5.5. – sciurus Apr 05 '11 at 04:24
  • Yes, us too. I verified on CentOS 4 as well. – Noah Watkins Apr 05 '11 at 15:40

3 Answers3

1

strace shows my netstat -c spending a large amount of time resolving hostnames before outputting its result - maybe try netstat -nc > /nfs/foo or wait longer before killing it. I don't think this has anything to do with NFS - I get the same long delay when redirecting to /tmp/foo

ggiroux
  • 244
  • 1
  • 2
  • Delays are to be expected with an NFS sink. The behavior I am seeing is that when NFS is the sink, `netstat` is suppressing standard out. – Noah Watkins Apr 05 '11 at 15:40
  • I tried `netstat -nc > /nfs/foo` and left the console alone for ~1 minute and here was still just an empty file – Buck Apr 05 '11 at 16:35
  • The lags were order of magnitude different between piping to `/tmp` and `/nfs/mount/file` but, in the end, netstat will eventually flush to nfs. – Buck Apr 05 '11 at 18:19
0

For particular configurations of NFS the delay to sync may be quite large, and it appears that netstat is not flushing its output after SIGKILL.

Killing the process after 15 seconds or so resulted in an empty file as you described. I waited for an extended period of time (I went to get coffee), and when I came back netstat had written out to the file.

Can you verify this behavior?

Noah Watkins
  • 133
  • 7
  • I started a netstat command from home, drove to work, and it had logged some data. That's ~25 minutes. So, it looks like there's some huge amount of lag, but it does eventually write. The issue now is if that lag is too large to render this solution useful. – Buck Apr 05 '11 at 18:18
0

What about this?

script -f -c "netstat -c" /nfs/mount/file > /dev/null

I think it may work, though I haven't got a NFS server handy to test it.

Eduardo Ivanec
  • 14,881
  • 1
  • 37
  • 43
  • Why I think this is a good option, from man 1 script: `-f Flush output after each write. This is nice for telecooperation: One person does "mkfifo foo; script -f foo" and another can supervise real-time what is being done using "cat foo".` – Eduardo Ivanec Apr 05 '11 at 17:02
  • That actually solves the problem. I'm not sure if this is adding much overhead, but it at least solves the immediate problem. Thank you ver much for your help. – Buck Apr 06 '11 at 04:05
  • Arrgghhh, but there's a catch. It looks like this works `script -f -c "netstat -c" /nfs/mount/file > /dev/null` but I want to use this in a script of my own, so I'd need to spin it up in the background and this `script -f -c "netstat -c" /nfs/mount/file > /dev/null &` returns instantly. So, won't work for me. This is a separate issue, which I'll look into, but if you have any thoughts, I'd love to hear them. – Buck Apr 06 '11 at 04:09
  • That's too bad, and I see what you mean. I couldn't find a way for it not to return immediately, but perhaps you could give this a try instead? `netstat -c | cat > /nfs/mount/file 2>&1 &`. The output of cat should always be unbuffered, so hopefully this will suffice. – Eduardo Ivanec Apr 06 '11 at 17:08