0

Problem

My debian server seems to lose network connection after a random amount of time (sometime after an hour, sometimes after a week, sometimes something in between). The server is hosted at a datacenter, so I do not have console access, except I order a KVM. When I restart the server remotely and boot in a rescue system, the server regains network connectivity.

At first I was not sure, whether the server froze or the network connection got lost. I ran a script which pinged 1.1.1.1 every minute and logged that to a file. I just checked that file and can now confirm, that the script was still running but could not reach 1.1.1.1 before I restarted the server.

Troubleshooting workflow

  1. Restart server via remote reboot (feature of the datacenter)
  2. Reboot into rescue system (virtual KVM)
  3. Check logs in /var/logs

Question

What can I do to determine the cause of this problem?

Marco Frost
  • 101
  • 2
  • 1
    Among the myriad of imaginable tests, you could keep running `ip -ts monitor` redirected to a file. Maybe it will give more information about what happens when the connectivity is lost. Anyway, having a remote console available to remotely do interactive actions while the server is still without network connectivity would be way better. – A.B Apr 21 '23 at 17:55
  • @A.B I'll give ip monitor a shot. Also I will try to get a remote console when the next outage strikes. What would be a good workflow to determine the cause then? – Marco Frost Apr 21 '23 at 18:02
  • Really, if I knew, I'd have written an answer. – A.B Apr 21 '23 at 18:14
  • Do you get an IP via DHCP or is it a static one? Recently I was experiencing a similar issue with this cause. Also with debian. – awado Apr 21 '23 at 18:48
  • @awado It's a static ip. – Marco Frost Apr 21 '23 at 21:05
  • Write a script that gathers all available information about the current network state (IP, link state, routes, iptables etc.) and write it in a file. Call that script via cron. Analyze the file after the next disconnect. – Gerald Schneider Apr 22 '23 at 05:26
  • @GeraldSchneider fwiw, that's the role of `ip -ts monitor` all of its own in realtime, except for iptables . (with iptables-nft) `nft monitor ruleset | ts` would also track rule changes in the same way. – A.B Apr 22 '23 at 10:08
  • `stdbuf -oL xtables-monitor -e | ts` is more useful (displays process name/pid too). There's also `tc monitor` `ethtool --monitor` and probably a few others to complement this. – A.B Apr 22 '23 at 10:25
  • @A.B: Thanks for this hint. BTW: `ts` must be installed manually and is part of `moreutils` in debian. – awado Apr 23 '23 at 21:32

0 Answers0