2

So here is the setup: we've got temporary access to a very large TCP WAN connection and we want to use this pipe to do WAN filesystem testing.

We would like to generate massive amounts of data on the fly, writing it to the filesystem on the other side. We've got a large number of servers to work with, so generating enough data with the right simulation isn't a problem, but we would like to simulate actual HPC application data instead of something like just piping /dev/zero.

Like I mentioned, we're looking to actually write the data, so looking for something more than iperf or netperf. My question then is do any of you know of any HPC application data simulators? How would you test writing data to the other side of the link?

EDIT:

I am getting close to finding a tool that fits the bill.

The most promising is MADbench2 which is actual scientific simulation code adapted for parallel I/O testing purposes.

I'm going to investigate some more tools on this page listing Parallel I/O Benchamrks

It is unclear which of these actually writes data though, which is really what we're aiming for.

n0suchth1ng
  • 21
  • 2

2 Answers2

2

Most of the parallel I/O benchmarks on that page should actually write data (albeit often random noise). I've personally only used IOR and PIO-Bench from that list, but these are all relatively well-known and accepted benchmarks in the HPC community. "Relatively", because I/O benchmarks can produce holy wars in HPC (especially vs vendors)... but none of them are really bad.

IOR is the one I've had the most experience with, and it seems to give good results. It can be complex to set up though; make sure you have MPI installed and configured properly to run in parallel. For more information on IOR, see this paper; this page is also a useful quick reference.

To be honest, though, the best benchmark for any system is the real application. I trust running the real application against test data more than any benchmark, as I've been bit by overly-simplistic measurements not representing real performance.

ajdecon
  • 1,301
  • 4
  • 14
  • 21
0

How about something hand-rolled using dd and nc?

Perhaps you could do dd if=/dev/sda | nc otherside 9999 and on the other side, do nc -l 9999| dd of=/var/lib/somewhere

You could pipe it through pv to get some idea of the amount of data and speed..

Tom O'Connor
  • 27,480
  • 10
  • 73
  • 148