8

I'm not our normal network guy... I've just been drafted to help with this issue, so please bear with me.

We have a fairly large (~4,000 devices?) network mostly comprised of HP Procurve gear. From time to time over the last several weeks, we've been getting some broadcast storms that pretty much keep all other traffic from being sent across the network. I set up Wireshark to do 5MB dumps, and I caught some of this in the act this morning.

You can download the packet capture. The fun began at packet #23968. A seemingly malformed NBNS packet gets repeated over and over again. However, it isn't just a straight loop. The source (143.226.8.185) and destination (143.226.44.79) IP addresses stay the same, but the source MAC address changes. The first packet seems to come from some insignificant device on the network, and is sent to a multicast address, 01:00:5e:7f:ff:fa. All of the packets after that come from the MAC addresses of our HP wireless access points and are sent to a different multicast address, 01:00:5e:62:2c:4f.

Here is the first packet:

No.     Time        Source                Destination           Protocol Info
  23968 122.229240  143.226.8.185         143.226.44.79         NBNS     Unknown operation (10) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding)[Malformed Packet]

Frame 23968 (1038 bytes on wire, 1038 bytes captured)
    Arrival Time: Sep 15, 2010 08:32:44.329966000
    [Time delta from previous captured frame: 0.004744000 seconds]
    [Time delta from previous displayed frame: 0.004744000 seconds]
    [Time since reference or first frame: 122.229240000 seconds]
    Frame Number: 23968
    Frame Length: 1038 bytes
    Capture Length: 1038 bytes
    [Frame is marked: True]
    [Protocols in frame: eth:ip:udp:nbns]
    [Coloring Rule Name: SMB]
    [Coloring Rule String: smb || nbss || nbns || nbipx || ipxsap || netbios]
Ethernet II, Src: IntelCor_d2:5e:6b (00:1f:3b:d2:5e:6b), Dst: IPv4mcast_7f:ff:fa (01:00:5e:7f:ff:fa)
    Destination: IPv4mcast_7f:ff:fa (01:00:5e:7f:ff:fa)
        Address: IPv4mcast_7f:ff:fa (01:00:5e:7f:ff:fa)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: IntelCor_d2:5e:6b (00:1f:3b:d2:5e:6b)
        Address: IntelCor_d2:5e:6b (00:1f:3b:d2:5e:6b)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IP (0x0800)
    Trailer: 7773643D22687474703A2F2F736368656D61732E786D6C73...
    Frame check sequence: 0x6f70653e [incorrect, should be 0x30019938]
Internet Protocol, Src: 143.226.8.185 (143.226.8.185), Dst: 143.226.44.79 (143.226.44.79)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 203
    Identification: 0x00d0 (208)
    Flags: 0x00
        0.. = Reserved bit: Not Set
        .0. = Don't fragment: Not Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 128
    Protocol: UDP (0x11)
    Header checksum: 0xe485 [correct]
        [Good: True]
        [Bad : False]
    Source: 143.226.8.185 (143.226.8.185)
    Destination: 143.226.44.79 (143.226.44.79)
User Datagram Protocol, Src Port: netbios-ns (137), Dst Port: netbios-ns (137)
    Source port: netbios-ns (137)
    Destination port: netbios-ns (137)
    Length: 183
    Checksum: 0x01db [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
NetBIOS Name Service
    Transaction ID: 0x4d2d
    Flags: 0x5345 (Unknown operation)
        0... .... .... .... = Response: Message is a query
        .101 0... .... .... = Opcode: Unknown (10)
        .... ..1. .... .... = Truncated: Message is truncated
        .... ...1 .... .... = Recursion desired: Do query recursively
        .... .... ...0 .... = Broadcast: Not a broadcast packet
    Questions: 16722
    Answer RRs: 17224
    Authority RRs: 8234
    Additional RRs: 8264
    Queries
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (12081)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (12081)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (11631)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (11631)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (25701)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (25701)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (25914)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (25914)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (25970)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (25970)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (18273)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (18273)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (24953)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (24953)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (26979)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (26979)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (3338)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (3338)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (14882)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (14882)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (28730)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (28730)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (25455)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (25455)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (8717)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (8717)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (28513)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (28513)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (29287)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (29287)
[Malformed Packet: NBNS]
    [Expert Info (Error/Malformed): Malformed Packet (Exception occurred)]
        [Message: Malformed Packet (Exception occurred)]
        [Severity level: Error]
        [Group: Malformed]

Here is the next packet:

No.     Time        Source                Destination           Protocol Info
  23969 122.229836  143.226.8.185         143.226.44.79         NBNS     Unknown operation (10) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding)[Malformed Packet]

Frame 23969 (217 bytes on wire, 217 bytes captured)
    Arrival Time: Sep 15, 2010 08:32:44.330562000
    [Time delta from previous captured frame: 0.000596000 seconds]
    [Time delta from previous displayed frame: 0.000596000 seconds]
    [Time since reference or first frame: 122.229836000 seconds]
    Frame Number: 23969
    Frame Length: 217 bytes
    Capture Length: 217 bytes
    [Frame is marked: True]
    [Protocols in frame: eth:ip:udp:nbns]
    [Coloring Rule Name: SMB]
    [Coloring Rule String: smb || nbss || nbns || nbipx || ipxsap || netbios]
Ethernet II, Src: HewlettP_05:de:da (00:17:a4:05:de:da), Dst: IPv4mcast_62:2c:4f (01:00:5e:62:2c:4f)
    Destination: IPv4mcast_62:2c:4f (01:00:5e:62:2c:4f)
        Address: IPv4mcast_62:2c:4f (01:00:5e:62:2c:4f)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: HewlettP_05:de:da (00:17:a4:05:de:da)
        Address: HewlettP_05:de:da (00:17:a4:05:de:da)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IP (0x0800)
Internet Protocol, Src: 143.226.8.185 (143.226.8.185), Dst: 143.226.44.79 (143.226.44.79)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 203
    Identification: 0x00d0 (208)
    Flags: 0x00
        0.. = Reserved bit: Not Set
        .0. = Don't fragment: Not Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 127
    Protocol: UDP (0x11)
    Header checksum: 0xe585 [correct]
        [Good: True]
        [Bad : False]
    Source: 143.226.8.185 (143.226.8.185)
    Destination: 143.226.44.79 (143.226.44.79)
User Datagram Protocol, Src Port: netbios-ns (137), Dst Port: netbios-ns (137)
    Source port: netbios-ns (137)
    Destination port: netbios-ns (137)
    Length: 183
    Checksum: 0x01db [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
NetBIOS Name Service
    Transaction ID: 0x4d2d
    Flags: 0x5345 (Unknown operation)
        0... .... .... .... = Response: Message is a query
        .101 0... .... .... = Opcode: Unknown (10)
        .... ..1. .... .... = Truncated: Message is truncated
        .... ...1 .... .... = Recursion desired: Do query recursively
        .... .... ...0 .... = Broadcast: Not a broadcast packet
    Questions: 16722
    Answer RRs: 17224
    Authority RRs: 8234
    Additional RRs: 8264
    Queries
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (12081)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (12081)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (11631)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (11631)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (25701)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (25701)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (25914)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (25914)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (25970)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (25970)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (18273)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (18273)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (24953)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (24953)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (26979)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (26979)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (3338)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (3338)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (14882)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (14882)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (28730)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (28730)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (25455)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (25455)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (8717)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (8717)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (28513)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (28513)
        Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (29287)
            Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
            Type: unknown
            Class: Unknown (29287)
[Malformed Packet: NBNS]
    [Expert Info (Error/Malformed): Malformed Packet (Exception occurred)]
        [Message: Malformed Packet (Exception occurred)]
        [Severity level: Error]
        [Group: Malformed]

Crazy, no? If you look through the packet capture, you will see much of this packet repeated after that point. It goes on and on after that, in to several more files.

If this were a loop, then why would only our APs be sending this packet around? These APs are scattered all over our campus.

A bit more info about our network... It is all flat. Straight Ethernet runs to everything, and we have a class B block of IPs. No subnets. There is a packet shaper, firewall, and router between our network and our WAN connection.

Finally, if you see this post and it seems familiar to you, that is because I have posted a similar issue in the past that we still have yet to resolve, but haven't seen recently. That can be found at HP switches sending multi-cast ping requests.

Thank you much for your time!

Edit: Packet 23968 is confirmed to be the trigger of this multicast storm. I've replayed that one packet into our network and kicked it off again.

Edit/Update: Doing some more experimentation. I've taken one of our HP access points and plugged it in directly to my PC. Nothing else attached to the segment. If I replay the initial packet that was causing the problems to the AP, the AP replies once. If I replay the AP's reply back to the AP, it replies again. Each time it does this, the TTL is lowered. What is happening here is that the APs on the network initially hear the broken multicast packet from the host and reply to it over multicast. Each AP hears these replies from all of the other APs and it replies to them. Each AP hears all of the replies to the replies and replies to them. Fortunately, it lowers the TTL each time so the storm goes away as soon as the TTL hits 0, and the packet is killed. Now all I need to do is figure out how to stop this behavior!

The AP I have in front of me is an HP Procruve 420 J8130B.

Edit (SOLVED!): After trying seemingly every configuration setting on the AP, I was still unable to keep it from retransmitting those multicast packets. I discovered we weren't on the latest firmware, so I tried upgrading, but the problem persisted. Then I tried downgrading to version 2.1.7 from November 29, 2006. No problems with this firmware! APs running 2.1.7 do not retransmit the packet!!! I am still waiting to figure out how the junk data got on the network in the first place, but the problem is solved for now. We are making a bug report with HP.

Community
  • 1
Brad
  • 1,419
  • 22
  • 43

3 Answers3

6

First and foremost, those are not NBNS packets, they're actually Universal Plug-n-Play packets attempting to search for "Internet Gateway Device" enabled devices. UPNP-IGD uses IPv4 multicast to locate such edge devices. The protocol, such as it is, says there should be only one. The give-away is in the packet payload:

M-SEARCH * HTTP/1.1
Host:239.255.255.250:1900
ST:urn:schemas-upnp-org:device:InternetGatewayDevice:1
Man:"ssdp:discover"
MX:3

.xmlsoap.org/ws/2004/08/addressing" xmlns:

IGD is used by some applications to tell consumer NAT gateways how to handle NAT traversal for certain protocols. IM applications and the like. You can make Wireshark show things better by telling it to decode UDP/137 as HTTP for that capture.

Now, why this is causing a multicast storm is the big question. You're getting the same kind of packet well before the storm hits, but they're correctly being sent to 239.255.255.250:1900. Packet 23955, in fact, comes from the same device that starts the storm in 23968. However, packet 23968 shows the same destination MAC address (one indicating IPv4 Multicast) but has a destination IP address that is in your /16 block and should NOT be multicast.

Packet 23604 is also very malformed. It has a valid Ethernet header, but the IP header is strangely truncated and ends in the same UPNP-IGD string I quoted above. The device that issued this strange, strange packet is the same device (well, comes from the same MAC address anyway) as packet 23968 that kicked off the multicast storm.

My best bet at this point is that the device at 00:1F:3B:D2:5E:6D is hosed in some way or is uniquely not handling these UPNP search requests correctly. Packet 24717 shows another M-SEARCH request going to 239.255.255.250:3702 that also comes from the same device. Right IP address, wrong port (should be 1900).

My guess is that the multicast storm is being kicked off by a packet with a Unicast IP address arriving with a multicast MAC address, and your network devices not handling that invalid case correctly. This is suggestive in the fact that the packets after the initial one all claim to source from the same IP (143.226.8.185), but the MAC address are all different. You have a bad device that managed to find a bug in multicast/unicast handling of your net-devices.

sysadmin1138
  • 133,124
  • 18
  • 176
  • 300
  • Thanks a ton for your insight! We've found the owner of 00:1F:3B:D2:5E:6D and are getting it in. I replayed packet #23968 and sure enough, got another storm. I am now going through the captures from all of the other times we've seen it and trying to see the common elements. I will update here soon. – Brad Sep 16 '10 at 13:45
  • Woot! See my last edit/update on the question itself for more information. I've figured out what is happening. Now I just have to figure out how to get it to stop. – Brad Sep 16 '10 at 15:34
  • Downgrading to an old firmware on the APs fixed the problem! I am marking your response as the answer, because it gave me the most help in tracking this down. Thanks again for your time. – Brad Sep 16 '10 at 19:40
  • @brad Thanks again for posting the packet capture, it's what lead me to the solution. – sysadmin1138 Sep 16 '10 at 21:29
0

@Brad: I just saw this and am wondering if it gives you any insight into the problem.

http://support.microsoft.com/kb/317843

joeqwerty
  • 109,901
  • 6
  • 81
  • 172
  • Thanks Joe. Doesn't look like this is it, as I found some more today with the same issue and data from other things such as DNS. I do appreciate the suggestion though! – Brad Sep 16 '10 at 15:59
0

My recommendation is to open the task manager in the host who is sending the broadcast and try to close one to one all the applicattions that could send something to the network and at the same time look to the packages in the network (Wireshark) to search for the app that is giving problems.

Gustavo
  • 101
  • 2
  • That would not work here. The problem intermittent, and seemingly caused by packets that were corrupt and not sent directly from any application. Even if they were, the APs should behave the way they did. – Brad Feb 14 '13 at 13:54