1

I have a server running Ubuntu 10.04LTS (I know it is too old), which over the last couple of weeks has on occasion been unresponsive to network traffic, and needed a hard reset first thing in the morning (9am when the office opens).

I have looked through kern.log and noticed a pattern.

After this happens there is usually a UFW block from an incoming external ip address, CIFS VFS error, another UFW block from a different ip address and then nothing logged until the restart. I'm not sure if this means the server has crashed and logging stopped or if the log has been turned off or sanitized.

The external IP addresses vary each time.

Example log:

Feb 19 01:46:43 Server1 kernel: [139893.285676] [UFW BLOCK] IN=eth0 OUT= MAC=00:26:b9:2e:79:5c:3c:81:d8:44:41:00:08:00 SRC=50.30.32.186 DST=10.0.0.1 LEN=52 TOS=0x00 PREC=0x00 TTL=107 ID=2976 DF PROTO=TCP SPT=57588 DPT=80 WINDOW=8192 RES=0x00 CWR ECE SYN URGP=0 
Feb 19 02:57:20 Server1 kernel: [144130.370015]  CIFS VFS: No response for cmd 50 mid 52893
Feb 19 02:57:21 Server1 kernel: [144130.760010]  CIFS VFS: No response to cmd 4 mid 52894
Feb 19 02:57:21 Server1 kernel: [144130.760015]  CIFS VFS: Send error in Close = -11
Feb 19 03:36:47 Server1 kernel: [146497.272912] [UFW BLOCK] IN=eth0 OUT= MAC=00:26:b9:2e:79:5c:3c:81:d8:44:41:00:08:00 SRC=86.108.49.79 DST=10.0.0.1 LEN=52 TOS=0x00 PREC=0x00 TTL=109 ID=29914 DF PROTO=TCP SPT=65452 DPT=80 WINDOW=8192 RES=0x00 SYN URGP=0 
Feb 19 09:00:56 Server1 kernel: [165946.155435] [UFW BLOCK] IN=eth0 OUT= MAC=00:26:b9:2e:79:5c:00:21:9b:29:91:aa:08:00 SRC=169.254.164.54 DST=10.0.0.1 LEN=89 TOS=0x00 PREC=0x00 TTL=255 ID=24 PROTO=UDP SPT=64676 DPT=53 LEN=69

Does this look like an attempted / successful attack, or a problem with the server itself?

UPDATE

Looks like the general consensus is that this is not an attack but a bug. As I was fixated on a security breach I neglected to mention that this morning, restarting the server didn't fix all the problems and I to troubleshoot a little more. During this I plugged the server into a different port on the switch and things worked shortly after. Just tested the port again and it seems dead. Would this cause and error that would stop logging to kern.log?

Gavin
  • 173
  • 1
  • 10

3 Answers3

5

The firewall entries and the CIFS errors have no obvious relation to each other; they occur way too many minutes apart. And besides, that traffic was blocked.

Michael Hampton
  • 244,070
  • 43
  • 506
  • 972
  • Yeah, that was my thoughts initially. Its always directly after a UFW Blocked log entry where the logging seems to stop (or disappear) so I thought is was worth an ask. – Gavin Feb 19 '14 at 13:00
1

As Michael says, the firewall blocks seem irrelevant. A failure in a filesystem (CIFS VFS being Common Internet File System Virtual File System (yes, I accept there's redundancy in there)), however, can cause a machine to become very unhappy indeed. If that's what's killing you, it's much more likely to be a bug than anything malicious - but it will be taking you down, just the same.

I'd spend my debugging time looking into those errors, and ignore the firewall stuff.

MadHatter
  • 79,770
  • 20
  • 184
  • 232
  • There is a backup script that runs after midnight via cron. Nothing has changed in the script for probably 12 months but I suppose thats a starting point. – Gavin Feb 19 '14 at 12:55
  • And has anything changed in the file system you're backing up? If so, that could tickle a bug in the FS code (and if not, why are you still backing it up?). – MadHatter Feb 19 '14 at 12:56
  • The backup is a of critical files (database, intranet source code and a large amount of image files) so they change daily but the hardware and config are the same as they have been for a long time. – Gavin Feb 19 '14 at 13:06
1

It is probably not an attack, altough it can't be closed out (it never can't be). They are coming from totally different sources, targeting totally different ports, different times and some of them have an internal and same of them an external source.

I see the most probably cause, that not only your software is old, but also your hardware. This send errors looks as some hardware problem, probably there is some DMA communication failure between the memory and the ethernet controller.

In your place I

  • tried to use an external network card

  • if it doesn't work, I replaced the motherboard of the server (maybe upgraded motherboard/cpu/ram).

peterh
  • 4,953
  • 13
  • 30
  • 44
  • I see your point but the server is used daily serving an intranet and shared file storage to 20 people and rarely goes wrong during this time. It seems to only fail whilst not in use (apart from a cron job) during the night. – Gavin Feb 19 '14 at 12:58