1

Here's an example:

( cp /tmp/one /nfs/one ) &
( cp /tmp/two /nfs/two ) &
( cp /tmp/three /nfs/three ) &
( cp /tmp/four /nfs/four ) &
( cp /tmp/five /nfs/five ) &
wait

Against a busy NFS server, does doing this gain anything vs

cp /tmp/one /nfs/one;
cp /tmp/two /nfs/two;
cp /tmp/three /nfs/three;
cp /tmp/four /nfs/four;
cp /tmp/five /nfs/five;

My thought is that I may be spending more time in the first example spinning up subshells (especially on a machine doing this many times at once, spinning up lots of user processes) where because of the bandwidth and NFS inode updating limitations, its not gaining me anything.

The files in question are between 20k and 2mb, never more than 6 at a time, and they're always getting copied to the same full path (just different filenames). There are a number of these happening on an array of servers (~50) all referencing the same small set of NFS mounts.

Justin
  • 141
  • 5

2 Answers2

2

parallelizing probably doesn't help. It's going to depend on a few things though: the speed of your local disk, the speed of your local network, and the same on the server.

If your local disk is pegged when doing a single copy, then no. It's not going to help.

If your local network is pegged when doing a single copy, then no. It's not going to help.

If your NFS server's network is pegged when doing a single copy, then no. It's not going to help.

If your NFS server's disk is pegged when doing a single copy then ... well, you get the picture.

The best way to determine if it's going to be helpful or not is to time both operations, repeatedly, for say, 10 runs, to get something statistically useful. Basically, if you find that you're not maxing out any of those four things above when doing a single copy, then you might see a performance benefit to doing it in parallel. But only testing will tell you this.

Travis Campbell
  • 1,466
  • 7
  • 15
1

Since some of the files are 20k you are probably spending some of your bandwidth on operations other than actual writing.

There is an excellent NFS -n option in the iostat tool that will show you what percentage of your IOPS are spent on actual writing.

# iostat -nmht 20

Filesystem:   ... rMB_svr/s  wMB_svr/s  ops/s  rops/s  wops/s
nfsserv:/share
              ... 0.00       17.5       9660   0.00    4508

Please wait a few periods (each period is 20 seconds here) for iostat to accumulate some statistics.

In this example only half of the IOPS are write operations. You can also see that the block size here is 4k. It is likely that you could improve the performance by overlapping WOPS and non-WOPS and running your processes in parallel.

Try to watch the CPU load on your NFS server as well as the network bandwidth utilization. On an NFS server with a disk array, running multiple processes should have a very good effect.

Dima Chubarov
  • 2,316
  • 1
  • 17
  • 28