3

I want to test a data center routing algorithm using Mininet. The traffic needs to conform to certain parameters:

  1. It should consist of "files" of various sizes (note that these don't actually have to be files; traffic generated in, e.g., iperf is OK, as long as the size is controllable);
  2. The file sizes should be drawn from a particular distribution;
  3. The source/destination host pairs between which data is sent should be selected randomly for a given file;
  4. The interval between when a file is sent and its successor is sent should be random; and
  5. If a huge file gets sent between two hosts that takes a long time to transfer, it should still be possible to send data between other hosts in the network.

Points 1-4 are taken care of. I've been struggling with #5 for days and I can't get it working properly. My initial thought was to spawn subprocesses/threads to send iperf commands to the hosts: 

while count < 10:
    if (count % 2) == 0:
        host_pair = net.get("h1", "h2")
    else:
        host_pair = net.get("h3", "h4")

    p = multiprocessing.Process(target=test_custom_iperf, args=(net, host_pair, nbytes))
    p.daemon = True
    p.start()

    time.sleep(random.uniform(0, 1))

The command test_custom_iperf is modeled after the Python Mininet API's version of iperf to include the -n transfer size parameter: 

client, server = host_pair
print client, server

output( '*** Iperf: testing TCP bandwidth between',
        client, 'and', server, '\n' )

server.sendCmd( 'iperf -s' )

if not waitListening( client, server.IP(), 5001 ):
    raise Exception( 'Could not connect to iperf on port 5001' )

cliout = client.cmd( 'iperf -c ' + server.IP() + ' -n %d' % nbytes )
print cliout

server.sendInt()
servout = server.waitOutput()

debug( 'Server output: %s\n' % servout)
result = [ net._parseIperf( servout ), net._parseIperf( cliout ) ]
output( '*** Results: %s\n' % result )

Making this non-blocking has been extremely difficult. I need to be able to send the server.sendInt() command, for some reason, and to do this I need to wait for the client's command to finish.

I'd appreciate any advice on what I can try to make this work!

Toni
  • 131
  • 2
  • 11
  • Since this appears to be a homework assignment, I'll only hint at an answer. There is a reason that there exists a concept of a [Maximum Transmission Unit](https://en.wikipedia.org/wiki/Maximum_transmission_unit). at some point a "file" has to be sent out the line. If you allow an MTU of "absurdly large", then the line will be occupied for the entire duration of the monster file. It doesn't matter how much queuing or multithreding you put in front of the transmission line, once the monster starts it must complete. Perhaps there is something wrong with requirement #5. – msw Jul 27 '15 at 20:23
  • Hah! Sadly it's for my research, otherwise I would've asked a TA :-) This is intended to model data center traffic as described in [Benson][1], where there are lots of small files going through the network at any given time, but occasionally a huge one will come through that takes a long time to send. From the paper, 20% of flows are longer than 11 seconds. Requirement 5 seems necessary, then; if a source/destination are busy transferring a huge file, shouldn't the other hosts be (mostly) unaffected? [1]: http://research-srv.microsoft.com/pubs/138279/DC-Network-Characterization-imc2010.pdf – Toni Jul 27 '15 at 20:51
  • FWIW, the problem doesn't go away if I only send around small files. – Toni Jul 27 '15 at 20:51
  • I misread: if A→B is a monster, then you are right, C→D should be unimpeded (assuming a switched, not bus, network). I shall think on't some more. – msw Jul 27 '15 at 20:53

1 Answers1

0

I took a hint from here and used Mininet's host.popen() module to send the data around. Hopefully this helps someone else: 

def send_one_file(file_dir, host_pair, files): 

    src, dst = host_pair  # a tuple of Mininet node objects

    # choose a random file from files
    rand_fname = random.sample(files, 1)[0]
    rand_path = os.path.join(file_dir, rand_fname)

    port = random.randint(1024, 65535)

    # Start listening at the destination host
    dst_cmd = 'nc -l %d > /home/mininet/sent/%s.out' % (port, rand_fname)
    print os.path.getsize(rand_path)
    dst.popen( dst_cmd, shell=True )

    # Send file from the source host
    src_cmd = 'nc %s %s < %s' % (dst.IP(), port, rand_path)
    src.popen( src_cmd, shell=True )

Then the parent function calls send_one_file() at random intervals: 

def test_netcat_subprocess_async(net, duration):

    file_dir = "/home/mininet/sf_mininet_vm/data/MVI_0406_split"
    files = os.listdir(file_dir)

    start_time = time.time()
    end_time = start_time + duration

    # Transfer for the desired duration
    while time.time() < end_time:
        # Choose a pair of hosts
        host_pair = random.sample(net.hosts, 2)

        test_send_one_file_netcat(file_dir, host_pair, files)

        interval = random.uniform(0.01, 0.1)
        print "Initialized transfer; waiting %f seconds..." % interval
        time.sleep(interval)

This works without any of the problems I experienced with multiprocessing or threading (breaking the network after the session is over, blocking when it shouldn't, etc.).

Toni
  • 131
  • 2
  • 11