Mininet script sending traffic from virtual machine's IP instead of host machines'

Question

In a python3/mininet script I have a tested, valid dictionary of host machines and their IP addresses. For each of the keys - by iterating dictOfAllHostsAndIPs.keys() - I execute a script on each emulated host's terminal

for host in dictOfAllHostsAndIPs.keys():
    host.cmd(os.system( "python3 ./traffic_generator.py %s" % <my_args>))

This script that sends random packet to IP addresses randomly picked from the list of IPs on the emulated network -- it generates traffic quite randomly using echo to pipe some data to netcat that hands it out on a port that is normally not in use :

os.system("echo -n '%s'" % data_string + " | nc %s 1299" % str(random_ip)))

My issue is that when I launch packet sniffers (mostly tshark) on each and every switch on the network, in the resulting logs in the TCP frames data I see on the first line that they all come from my VM's IP (192.168.119.133) instead of the host machines (10.0.0.XX), and in the ethernet/source line I always get the MAC address of the VM itself instead of the virtual machines.

# frame# /time(epoch) /     source ip → dest ip    /   ports: src → dest
>       8 0.063202633 192.168.119.133 → 10.0.0.1     TCP 74 40370 → 1299

> Ethernet II, Src: VMware_34:8e:de (00:0c:29:34:8e:de), Dst:
> VMware_fa:1a:ac (00:50:56:fa:1a:ac)
>     Destination: VMware_fa:1a:ac (00:50:56:fa:1a:ac)
>         Address: VMware_fa:1a:ac (00:50:56:fa:1a:ac)
>         
>     Source: VMware_34:8e:de (00:0c:29:34:8e:de)
>         Address: VMware_34:8e:de (00:0c:29:34:8e:de)

The only thing that changes in the source section of the frames I see in my logs is the port from which the VM is sending the packets, which is always between 35000 and 49999 (or maybe 50k and something). (see first line of the log above, ports: src)

I initially assumed that each one of the ports being used was dedicated to sending traffic from the different emulated host machine, but they're way too many and they don't stay the same, so this is clearly not the case.

Not having realized this right away (everything else looked and still looks okay: packet destination and every other parameter I need to check for my research project), I have then implemented a modular, threaded refactoring of my code. So now I look at the logs of the different versions of my program and they all have the same issue, no matter how I run the traffic generation script (older versions) or the instantiation of trafficGenerator objects (newer versions), so this issue is not dependent on this aspect of my scripts' structure.

I checked on and fiddled with my calls many times, quadruple-checked the lists (then dicts, then lists again) that I iterate through in order to have the individual machines run the traffic generation code with for hosts in list(net.get(hosts)): host.cmd(<action>) The latest version goes like this:

    listOfTGs = []
### Create a list of threads for parallel execution
    listOfThreads = []
    for host in dictOfAllHostsAndIPs.keys():
###     Create a trafficGenerator object on the mininet machine, then call commands on host
        listOfTGs.append(tg.TrafficGenerator(durationFromArgument, host.name, callable_ips_list, net))
    for tg_instance in listOfTGs:
        listOfThreads.append(tg_instance.threadedExecution())
    print(listOfThreads)
    for th in listOfThreads:
        th.start()

I can confirm (after maaaany careful print-based debugging hours) that the action of generating traffic, as all other actions, is executed once for each emulated host machine, in parallel, with the correct identity for each host visible in the stdout. And still, the IP that I get when sniffing the packets is the one of the VM itself instead of the one of the host machine that should appear. And in papers from other researchers I see that they have it correctly in their sniffing analysis, so this is not just "something that mininet does this way" (I looked into it in order to be sure).

TL;DR: my problem is that the source of the packets being sent around is the VM itself on which I launch the mininet emulations, and not the host machines being emulated in whose name I call the traffic generation processes.

Answer 1

I think I see what is going on in the source, but i have not run the framework to confirm it.

It looks like mininet inatalls a NAT rule for every node:

    self.cmd( 'iptables -I FORWARD',
              '-i', self.localIntf, '-d', self.subnet, '-j DROP' )
    self.cmd( 'iptables -A FORWARD',
              '-i', self.localIntf, '-s', self.subnet, '-j ACCEPT' )
    self.cmd( 'iptables -A FORWARD',
              '-o', self.localIntf, '-d', self.subnet, '-j ACCEPT' )
    self.cmd( 'iptables -t nat -A POSTROUTING',
              '-s', self.subnet, "'!'", '-d', self.subnet,
              '-j MASQUERADE' )

     # Instruct the kernel to perform forwarding
    self.cmd( 'sysctl net.ipv4.ip_forward=1' )

So basically if traffic has a destination outside of the node's subnet a NAT (MASQUAERADE) rule is applied. Assuming this is also occurring in the root net namespace, then all outbound traffic is NAT'd (IE: takes the IP of the last hop).

How to prove this

1. Set up your mininet network.
2. Look at your root namespace NAT rules with iptables -t nat -L. If you see a MASQUERADE entry then that is the NAT rule that you are having issues with
3. You should be able to see the individual namespaces with ip netns list. I believe they should just be labeled as numbers.
4. For every namespace in that list enter ip netns exec $NETNSNAME iptables -t nat -L . Observe any more NAT rules.

How to test if mininet is executing commands in the root Net Namespace

Aside from a NAT rule, the other likely source of the issue would be that mininet is not executing the command from the node's net namespace but the root namespace. This should be easy to test as well.

1. Set up your mininet network.
2. You should be able to see the individual namespaces with ip netns list. I believe they should just be labeled as numbers.
3. For every namespace in that list enter ip netns exec $NETNSNAME ip addr show. Observe the IP addresses.
4. For the namespace that corresponds with the node you wish to send traffic from, run the following: ip netns exec $NETNSNAME bash -c "echo -n $DATASTRING | nc $RANDOMIP 1299" where you have the BASH variables for DATASTRING and RANDOMIP already set.
5. Observe traffic as you did prior.

If the traffic you observe still has the VM's src IP then it is likely a NAT rule causing the issue. If it has the node's IP then mininet is not executing the command in the correct namespace.