0

We noticed this morning that we are unable to login with SSH into our server. Our server providers then installed a minimal Linux on the ram. After mounting the disks and chrooting into it, I stopped denyhosts temporarily and when I went in hosts.deny file to clear our IP addresses, we see the following. What exactly is this? 

# DenyHosts: Sat Jan 28 05:01:43 2017 | sshd: ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@$
sshd: ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@$
# DenyHosts: Sat Jan 28 05:01:43 2017 | sshd: ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@$
sshd: ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@$
# DenyHosts: Sat Jan 28 05:23:14 2017 | sshd: ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@$
sshd: ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@$
# DenyHosts: Sat Jan 28 05:23:14 2017 | sshd: ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@$
sshd: ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@$

Also, when I give the command, iptables -L, I get the following. I want to clear iptables rules :

iptables -L
libkmod: ERROR ../libkmod/libkmod.c:554 kmod_search_moddep: could not open moddep file '/lib/modules/4.8.15/modules.dep.bin'
iptables v1.4.14: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.

This is how I chrooted into the environment :

 mount --bind /proc /mnt/proc
 mount --bind /dev /mnt/dev
 mount --bind /sys /mnt/sys
chroot /mnt

Thank you.

Update

 modprobe filter
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.8.15/modules.dep.bin'
modprobe: FATAL: Module filter not found in directory /lib/modules/4.8.15

Update

Log

Jan 30 02:39:34 HOSTNAME sshd[6483]: refused connect from 112.85.42.18 (112.85.42.18)
Jan 30 02:40:18 HOSTNAME sshd[6495]: warning: /etc/hosts.deny, line 64: missing newline or line too long
Jan 30 02:40:18 HOSTNAME sshd[6495]: warning: /etc/hosts.deny, line 64: all the subsequent rules will be ignored
Jan 30 02:40:18 HOSTNAME sshd[6495]: refused connect from 112.85.42.18 (112.85.42.18)
Jan 30 02:40:57 HOSTNAME sshd[6504]: warning: /etc/hosts.deny, line 64: missing newline or line too long
Jan 30 02:40:57 HOSTNAME sshd[6504]: warning: /etc/hosts.deny, line 64: all the subsequent rules will be ignored
Jan 30 02:40:57 HOSTNAME sshd[6504]: refused connect from 112.85.42.18 (112.85.42.18)
Jan 30 02:41:37 HOSTNAME sshd[6519]: warning: /etc/hosts.deny, line 64: missing newline or line too long
Jan 30 02:41:37 HOSTNAME sshd[6519]: warning: /etc/hosts.deny, line 64: all the subsequent rules will be ignored
Jan 30 02:41:37 HOSTNAME sshd[6519]: refused connect from 112.85.42.18 (112.85.42.18)
We are Borg
  • 177
  • 1
  • 18

1 Answers1

1

Looks like the kernel on the minimal installation provided by your ISP is different than the kernel in the actual system. Then it tries to load the filter module for iptables, and fails to load because your chroot environment does not have the module for your kernel version.

Try loading the filter module before entering chroot with modprobe filter. Hope the minimal system has the filter module available.

What comes to the DenyHosts entries, ^@ is null byte. Maybe your log files contain null bytes for some reason instead of IP addresses, and that is why denyhosts adds those to the rules.

Another alternative is that there is some bug in either sshd or denyhosts that causes null bytes to show up.

You can check your log files if null bytes are there. If they are, then sshd outputs them there and denyhosts simply copies them, which means sshd has an issue. If log files contain normal entries, then the bug is in sshd.

Tero Kilkanen
  • 36,796
  • 3
  • 41
  • 63
  • Thank you. I gave modprobe filter and it gives an error. I have added the log to bottom of main post. Secondly, I purged openssh-server and reinstalled it. I had to setup the config file again. We use public key authentication, but have also setup password based for now, so we dont have much problems. Here is the config file : http://pastebin.com/838J0pee . Does sshd not work in chroot environment, as I get sshd fail when I ask service sshd status. Thanks. – We are Borg Jan 30 '17 at 11:51
  • Due to the long sshd_config file, I have added it in pastebin. I hope that's not a problem. Thanks. – We are Borg Jan 30 '17 at 11:51
  • Did you exit the chroot before trying to load the module? And did you check the log files how IP addresses appear there? – Tero Kilkanen Jan 30 '17 at 11:52
  • Before I didn't exit chroot, but now I did, and I get fatal : module filter not found. I checked auth.log in /var/log, and discovered that some user tried some weird way to get in the server since friday, and that has resulted into this problem. I have added the log in main post. Can you check it out. Is it safe to exit chroot and boot into our disk now, so I can check if ssh normally is possible. – We are Borg Jan 30 '17 at 12:01
  • The minimal rescue system from your provider doesn't contain the filter module. You can clear the `hosts.deny` file and reboot into your system. Update all system software after that. – Tero Kilkanen Jan 30 '17 at 12:03
  • This worked. Thanks. Any idea how I can prevent such an user from attacking. Thank you. – We are Borg Jan 30 '17 at 12:16
  • Make sure your software is up-to-date, so that bugs like this are fixed. That is all you need to do. – Tero Kilkanen Jan 30 '17 at 13:04
  • Thank you. I will keep that in mind. Is this a known bug? – We are Borg Jan 31 '17 at 09:24
  • I don't know about that, but if it is, one can find it in DenyHosts changelogs. – Tero Kilkanen Jan 31 '17 at 11:38