Why does pcap_sendpacket fail on Thunderbolt interface?

Question

In a multi platform project I am using pcap to get a list of all network interfaces, open each (user cannot select which interfaces to use) and send/receive packets (Ethernet type 0x88e1/HomePlugAV) on each. This works fine on Windows and on Mac OS X, but sometimes on Mac OS X pcap_sendpacket fails after some time on the interface that networksetup -listallhardwareports lists as "Hardware Port: Thunderbolt 1". The error is:

send: No buffer space available

When the program is run after the machine was booted, then it takes some time until the error occurs. When the error occurred once and I stop my program, the error occurs immediately when I restart my program without rebooting the machine.

ifconfig -v en9:

en9: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500 index 8
    eflags=80<TXSTART>
    options=60<TSO4,TSO6>
    ether b2:00:1e:94:9b:c1 
    media: autoselect <full-duplex>
    status: inactive
    type: Ethernet
    scheduler: QFQ 

networksetup -listallhardwareports (only the relevant parts):

Hardware Port: Thunderbolt 1
Device: en9
Ethernet Address: b2:00:1e:94:9b:c1

Tests show that on OS X 10.9 the interface is not up initially, but on OS X 10.9.2 and 10.9.3 the interface is up and running after booting.

On OS X 10.9 ifconfig initially says:

en5: flags=8822<BROADCAST,SMART,SIMPLEX,MULTICAST> mtu 1500 index 8

After ifconfig en5 up the problematic behavior is the same on OS X 10.9.

1. Why does pcap_sendpacket fail on the Thunderbolt adapter?
2. How can my program detect that this is a troubling interface before opening it? I know I could open the interface and try to send one packet, but I'ld prefer to do a clean detection beforehand.

Answer 1

As a workaround, you can ignore the "Thunderbolt 1" interface:

#include <stdio.h>

#include <pcap/pcap.h>

#include <CoreFoundation/CoreFoundation.h>
#include <SystemConfiguration/SCNetworkConfiguration.h>

const char thunderbolt[] = "Thunderbolt 1";

// Build with -framework CoreFoundation -framework SystemConfiguration
int main(int argc, char * argv[])
{
    // See: https://opensource.apple.com/source/configd/configd-596.13/SystemConfiguration.fproj/SCNetworkInterface.c
    // get Ethernet, Firewire, Thunderbolt, and AirPort interfaces
    CFArrayRef niArrayRef = SCNetworkInterfaceCopyAll();

    // Find out the thunderbolt iface
    char thunderboltInterface[4] = "";
    if(niArrayRef) {
        CFIndex cnt = CFArrayGetCount(niArrayRef);
        for(CFIndex idx = 0; idx < cnt; ++idx) {
            SCNetworkInterfaceRef tSCNetworkInterfaceRef = (SCNetworkInterfaceRef)CFArrayGetValueAtIndex(niArrayRef, idx);
            if(tSCNetworkInterfaceRef) {
                CFStringRef BSDName = SCNetworkInterfaceGetBSDName(tSCNetworkInterfaceRef);
                const char * interfaceName = (BSDName == NULL) ? "none" : CFStringGetCStringPtr(BSDName, kCFStringEncodingUTF8);

                CFStringRef localizedDisplayName = SCNetworkInterfaceGetLocalizedDisplayName(tSCNetworkInterfaceRef);
                const char * interfaceType = (localizedDisplayName == NULL) ? "none" : CFStringGetCStringPtr(localizedDisplayName, kCFStringEncodingUTF8);

                printf("%s : %s\n", interfaceName, interfaceType);

                if(strcmp(interfaceType, thunderbolt) == 0) {
                    // Make a copy this time
                    CFStringGetCString(BSDName, thunderboltInterface, sizeof(thunderboltInterface), kCFStringEncodingUTF8);
                }
            }
        }
    }
    printf("%s => %s\n", thunderbolt, thunderboltInterface);

    CFRelease(niArrayRef);
    return 0;
}

Answer 2

I'm guessing from

When the program is run after the machine was booted, then it takes some time until the error occurs. When the error occurred once and I stop my program, the error occurs immediately when I restart my program without rebooting the machine.

that what's probably happening here is that the interface isn't active, so packets given to it to send aren't transmitted (and the mbuf(s) for them freed), and aren't discarded, but are, instead, just left in the interface's queue to be transmitted. Eventually either the queue fills up or an attempt to allocate some resource for the packet fails, and the interface's driver returns an ENOBUFS error.

This is arguably an OS X bug.

From

In a multi platform project I am using pcap to get a list of all network interfaces, open each (user cannot select which interfaces to use) and send/receive packets (Ethernet type 0x88e1/HomePlugAV) on each.

I suspect you aren't sending on all interfaces; not all interfaces have a link-layer header type that has an Ethernet type field - for example, lo0 doesn't.

If you're constructing Ethernet packets, you would only want to send on interfaces with a link-layer header type (as returned by pcap_datalink()) of DLT_EN10MB ("10MB" is a historical artifact; it refers to all Ethernet types except for the old experimental 3MB Xerox Ethernet, which had a different link-layer header).

You probably also don't want to bother with interfaces that aren't "active" in some sense (some sense other than "is up"); unfortunately, there's no platform-independent API to determine that, so you're going to have to fall back on #ifdefs here. That would probably rule out interfaces where the packets would pile up unsent and eventually cause an ENOBUFS error.