1

Periodically the internet is unreachable on my network, sometimes for 30+ minutes. After testing a direct connection to our modem, I realized this was not a problem with our ISP but the network itself.

What I've tried:

  • I can ping the firewall.
  • Restarting the firewall fixes the connection.
  • Disconnecting the firewall from the switch fixes the connection.
  • When I connect my computer directly to the firewall, I still cannot get out, but when I disconnect the firewall from the switch, I don't have any problems.

What should my next steps be for troubleshooting this? I know how to use Wireshark, but I'm a bit of a noob and don't know what to look for. I did notice while the internet was working that one of my switches was putting out a lot of ARP requests compared to the others, asking for the same IPs over and over. I'm not sure if this is normal or not, though. Also, the switch keeps sending Spanning Tree packets that say "Topology Change Notification" in Wireshark.

Reading a few similar questions on SO it sounds like I might have a loop somewhere in the network causing all the ARP requests. I'm not sure why it would be just the one switch sending them out so much, though, stead of all three on our network. I don't see any obvious looping in our setup, but I'm not sure how to rule this out, either.

Update Network diagram:

Modem --- Firewall --- Switch --- (multiple connections to other computers and switches on the network)

Gabriel Syme
  • 111
  • 3
  • 1
    Give us a diagram of your network. Why would the switch be sending ARP packets? ARP is used to resolve ip addresses to MAC addresses, which is not something a switch would do unless it is itself communicating with hosts (which they generally don't do) or it's a L3 switch performing routing. Do you mean that the switch is flooding ARP traffic? Do you have a physical loop between your switches and if so, are you running STP on the switches? When you disconnect the firewall from the switch what do you mean? Do you mean that you connected the switch to the modem, bypassing the firewall? – joeqwerty Jan 26 '15 at 18:32
  • It might be a 100% cpu usage on firewall. Check cpu usage logs on firewall. – Emir Jan 26 '15 at 19:42
  • Draw out all of your switches on paper and draw all of the connections between all of the switches. Then determine if you have more than one path from any one switch to any other switch. If you do then you have a switch loop and you need to have STP (Spanning Tree Protocol) enabled on all of the switches to eliminate the loop. If STP is not enabled on all of the switches then that will definitely cause problems. Can you answer my question about how you bypassed the firewall in your testing? Did you plug one of the switches directly into the modem? – joeqwerty Jan 26 '15 at 21:26
  • @joeqwerty I updated the post with a very basic diagram. When I look in wireshark, I see ARP requests asking about IPs saying to tell a specific IP address. The source is an HP mac address and the IP it's requesting to tell is the IP of the HP switch. (I know this because I can browse to that IP to access the web interface for the switch). I think I may mean it's flooding ARP traffic, but I'm not 100% certain what that means :). I've checked everything for a physical loop and haven't found one. – Gabriel Syme Jan 26 '15 at 21:27
  • When I said "disconnected the firewall from the switch" I meant I isolated it from the network, so the network had no physical way to get to the firewall. I had my laptop connected to the firewall, so I could then test if I could get out to the internet with the rest of the network disconnected. – Gabriel Syme Jan 26 '15 at 21:28
  • What type of host is the source MAC address that's sending ARP requests to the switch? Client machines don't ARP for the switch ip address unless the switch performs routing and is used by the clients as the Default Gateway. Otherwise, the ip address of the switch is for management purposes and is completely unknown and transparent to the clients. – joeqwerty Jan 26 '15 at 21:35
  • All that being said and based on your testing it sounds like the firewall is the problem. Can you look at CPU, Memory and link utilization statistics/metrics on the firewall? Also, you stated that you have multiple switches so you need to verify whether or not multiple paths exist between the switches and if so, that STP is running between the switches. – joeqwerty Jan 26 '15 at 21:36
  • ARP is part of any layer2 environment. A switch/subnet/vlan is layer2. In a VLAN machines communicating through a switch are 'routed' through switch ports that have mac address tables of the hosts that are advertised through that switch port. – Citizen Jan 26 '15 at 21:51
  • A switch doesn't ARP for connected clients unless the switch itself is communicating with those clients or it's a layer 3 switch and is routing traffic between VLAN's. The ARP table is the ip address to MAC address mappings. The MAC address table is the MAC address to switch port mappings. A layer 2 switch should have nothing in it's ARP table except for those hosts that it itself is communicating with, such as a management station or a monitoring system. A switch that is `switching` traffic from one port to another does not use ARP to perform that function. – joeqwerty Jan 26 '15 at 23:28
  • @joeqwerty Thanks for that explanation. That's helping my understanding of how all this works. Unfortunately I can't see the cpu usage on my Netgear firewall. I think I need to setup network monitoring software to get it's logs. As far as I could tell, the source MAC address for the ARP requests was the switch itself, but I may have been missing something. I think it's only operating as a layer2 switch, so it shouldn't be sending anything like you said. – Gabriel Syme Jan 27 '15 at 15:50
  • Oddly enough, the ARP traffic has stopped after I changed the topology (connected the firewall to a different switch). Thank you so much for helping me think through this and explaining how switches use ARP. I now know some more things I need to learn about. – Gabriel Syme Jan 27 '15 at 15:52

1 Answers1

0

I ended up changing the network topology by connecting the firewall to a different switch. The odd ARP traffic has stopped, and we haven't had any internet outages for the last 48 hours. I'm not sure why, but hopefully that fixed it!

Gabriel Syme
  • 111
  • 3