0

All of a sudden the network at my University has frozen and we managed to isolate the problem to massive amounts of UDP packets sent on the DHCP ports. On closer inspection, we found that some of the clients keep spamming the server with DHCP requests although the server seem to respond. I'm pasting a sample of the syslog file on the server ([[client IP]] is the same in all entries, the server and client IP are in the same subnet). The very odd thing is that it's not only one client that does this, there is even a wireless router and it just started happening today. isc-dhcp-server doesn't seen to have been updated. Any assistance would be greatly appreciated.

Feb 25 17:57:46 zeus dhcpd: DHCPRELEASE of [[client IP]] from 00:22:75:ea:e5:dc via eth1 (found)
Feb 25 17:57:48 zeus dhcpd: message repeated 3 times: [ DHCPRELEASE of [[client IP]] from 00:22:75:ea:e5:dc via eth1 (found)]
Feb 25 17:57:48 zeus dhcpd: DHCPDISCOVER from 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:49 zeus dhcpd: DHCPOFFER on [[client IP]] to 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:50 zeus dhcpd: DHCPDISCOVER from 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:50 zeus dhcpd: DHCPOFFER on [[client IP]] to 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:50 zeus dhcpd: DHCPREQUEST for [[client IP]] ([[server IP]]) from 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:50 zeus dhcpd: DHCPACK on [[client IP]] to 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:51 zeus dhcpd: DHCPDISCOVER from 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:51 zeus dhcpd: DHCPOFFER on [[client IP]] to 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:51 zeus dhcpd: DHCPREQUEST for [[client IP]] ([[server IP]]) from 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:51 zeus dhcpd: DHCPACK on [[client IP]] to 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:51 zeus dhcpd: DHCPDISCOVER from 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:51 zeus dhcpd: DHCPOFFER on [[client IP]] to 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:51 zeus dhcpd: DHCPREQUEST for [[client IP]] ([[server IP]]) from 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:51 zeus dhcpd: DHCPACK on [[client IP]] to 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:51 zeus dhcpd: DHCPDISCOVER from 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:51 zeus dhcpd: DHCPOFFER on [[client IP]] to 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:51 zeus dhcpd: DHCPREQUEST for [[client IP]] ([[server IP]]) from 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:51 zeus dhcpd: DHCPACK on [[client IP]] to 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:51 zeus dhcpd: DHCPDISCOVER from 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:51 zeus dhcpd: DHCPOFFER on [[client IP]] to 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:51 zeus dhcpd: DHCPREQUEST for [[client IP]] ([[server IP]]) from 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:51 zeus dhcpd: DHCPACK on [[client IP]] to 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:52 zeus dhcpd: DHCPDISCOVER from 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:52 zeus dhcpd: DHCPOFFER on [[client IP]] to 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:52 zeus dhcpd: DHCPREQUEST for [[client IP]] ([[server IP]]) from 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:52 zeus dhcpd: DHCPACK on [[client IP]] to 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:52 zeus dhcpd: DHCPDISCOVER from 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:52 zeus dhcpd: DHCPOFFER on [[client IP]] to 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:52 zeus dhcpd: DHCPREQUEST for [[client IP]] ([[server IP]]) from 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:52 zeus dhcpd: DHCPACK on [[client IP]] to 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:53 zeus dhcpd: DHCPDISCOVER from 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:53 zeus dhcpd: DHCPOFFER on [[client IP]] to 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:53 zeus dhcpd: DHCPREQUEST for [[client IP]] ([[server IP]]) from 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:53 zeus dhcpd: DHCPACK on [[client IP]] to 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:53 zeus dhcpd: DHCPDISCOVER from 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:53 zeus dhcpd: DHCPOFFER on [[client IP]] to 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:53 zeus dhcpd: DHCPREQUEST for [[client IP]] ([[server IP]]) from 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:53 zeus dhcpd: DHCPACK on [[client IP]] to 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:53 zeus dhcpd: DHCPDISCOVER from 00:22:75:ea:e5:dc via eth1
Feb 25 17:57:53 zeus dhcpd: DHCPOFFER on [[client IP]] to 00:22:75:ea:e5:dc via eth1

Contents of /var/lib/dhcp/dhcpd.leases:

lease [[client IP]] {
starts 4 2016/02/25 16:06:25;
ends 5 2016/02/26 16:06:25;
cltt 4 2016/02/25 16:06:25;
binding state active;
next binding state free;
rewind binding state free;
hardware ethernet 00:22:75:ea:e5:dc;
uid "\001\000\"u\352\345\334";
}
lease [[client IP]] {
starts 4 2016/02/25 16:06:25;
ends 5 2016/02/26 16:06:25;
cltt 4 2016/02/25 16:06:25;
binding state active;
next binding state free;
rewind binding state free;
hardware ethernet 00:22:75:ea:e5:dc;
uid "\001\000\"u\352\345\334";
}
lease [[client IP]] {
starts 4 2016/02/25 16:06:26;
ends 5 2016/02/26 16:06:26;
cltt 4 2016/02/25 16:06:26;
binding state active;
next binding state free;
rewind binding state free;
hardware ethernet 00:22:75:ea:e5:dc;
uid "\001\000\"u\352\345\334";
}
lease [[client IP]] {
starts 4 2016/02/25 16:06:27;
ends 4 2016/02/25 16:08:53;
tstp 4 2016/02/25 16:08:53;
cltt 4 2016/02/25 16:06:27;
binding state free;
hardware ethernet 00:22:75:ea:e5:dc;
uid "\001\000\"u\352\345\334";
}
lease [[client IP]] {
starts 4 2016/02/25 16:08:57;
ends 5 2016/02/26 16:08:57;
cltt 4 2016/02/25 16:08:57;
binding state active;
next binding state free;
rewind binding state free;
hardware ethernet 00:22:75:ea:e5:dc;
uid "\001\000\"u\352\345\334";
}
lease [[client IP]] {
starts 4 2016/02/25 16:08:57;
ends 4 2016/02/25 16:08:57;
tstp 4 2016/02/25 16:08:57;
cltt 4 2016/02/25 16:08:57;
binding state free;
hardware ethernet 00:22:75:ea:e5:dc;
uid "\001\000\"u\352\345\334";
}
lease [[client IP]] {
starts 4 2016/02/25 16:08:57;
ends 5 2016/02/26 16:08:57;
cltt 4 2016/02/25 16:08:57;
binding state active;
next binding state free;
rewind binding state free;
hardware ethernet 00:22:75:ea:e5:dc;
uid "\001\000\"u\352\345\334";
}
lease [[client IP]] {
starts 4 2016/02/25 16:08:57;
ends 4 2016/02/25 16:08:57;
tstp 4 2016/02/25 16:08:57;
cltt 4 2016/02/25 16:08:57;
binding state free;
hardware ethernet 00:22:75:ea:e5:dc;
uid "\001\000\"u\352\345\334";
}
rhobincu
  • 103
  • 1
  • 7
  • Two things come to mind: `1.` Potential malware on the client. `2.` The ip address the server is offering is already in use and the client is rejecting it. Both the server and the client have mechanisms to check that the ip address is available before offering/accepting it. If that mechanism is disabled or faulty on the server it may be offering an ip address that's already in use. Another possibility is that someone inadvertently manually assigned the offered ip address to a host on the network and the DHCP client is rejecting the offered ip address because it has detected that it's in use. – joeqwerty Feb 25 '16 at 16:28
  • One of the clients is a Linksys wireless router, can those be infected with malware? If the IP was taken, shouldn't I see a DHCPNAK? Also I ran an arping on the client IP and the MAC address that replied is the one in the logs. – rhobincu Feb 25 '16 at 16:32
  • Certainly the router could have been compromised in some way, yes. You're probably right about the DHCPDECLINE but I would verify it by scanning the network to verify whether or not the ip address is already in use. If it is in use it could be that the Linksys wireless router isn't behaving correctly in sending the DHCPDECLINE to the DHCP server. – joeqwerty Feb 25 '16 at 16:40
  • This would be solvable with classes and limiting the number of leases to one per one MAC, but so far isc-dhcpd isn't capable of it. At least I didn't manage to set it up this way (I just didn't find a way to create class properly for ethernet), and it seems like noone in the mailing list was interested to help me out. – drookie Feb 25 '16 at 17:04

2 Answers2

1

You might see this behavior if communication is only going one way between these clients and the DHCP server.

The requests make it through to the server, the server responds but the client never gets the response. So it keeps asking.

I have seen this happen due to damaged cabling. So the first thing I would try is giving one of the clients an appropriate static IP address and seeing if it can actually ping the DHCP server and get a response.

Grant
  • 17,859
  • 14
  • 72
  • 103
  • That way DHCP[DISCOVER->OFFER->REQUEST-ACK] chain wouldn't be completed. – drookie Feb 25 '16 at 17:01
  • That's a good idea, will test and come back with the results. Thanks! – rhobincu Feb 25 '16 at 17:19
  • OK, so, we eventually figured it out. The problems started with a faulty cable, indeed, but it wasn't the cable between that router and the gateway, it was another cable in a different section of the network. However, due to it spamming a lot of junk, it probably locked the router which required a hard factory reset (from the little reset button) to start working properly again. – rhobincu Feb 29 '16 at 17:41
1

As a temporary workaround to resolve network functioning I propose adding a static leases for these MACs with predefined IP addresses, in the form of

host probably-infected {
   hardware ethernet 00:22:75:ea:e5:dc;
   fixed-address <SOME FIXED IP>;
}

and keep investigating what is wrong with these MACs.

drookie
  • 8,625
  • 1
  • 19
  • 29