Today I faced a what seemed like a DDOS attack. My server provider warned me about excessive CPU usage (400% for over 6 hours) and I couldn't access any website, could not login via SSH either. Lish console reported an error that went something like 'php-fpm out of memory'.
Only thing I could do was a hard reboot. After server was up again, I looked at fail2ban's status and it shows 'active(exited)'. I restarted it, then looked through logs and here's what I have:
fail2ban.log.2: File ends with
2021-07-25 09:10:11,793 fail2ban.server [26723]: INFO Shutdown in progress...
2021-07-25 09:10:11,794 fail2ban.server [26723]: INFO Stopping all jails
2021-07-25 09:10:11,795 fail2ban.filter [26723]: INFO Removed logfile: '/var/log/auth.log'
2021-07-25 09:10:11,817 fail2ban.filter [26723]: INFO Removed logfile: '/var/log/apache2/error.log'
2021-07-25 09:10:12,062 fail2ban.actions [26723]: NOTICE [sshd] Flush ticket(s) with iptables-multiport
2021-07-25 09:10:12,070 fail2ban.actions [26723]: NOTICE [sshd] Unban [IP]
2021-07-25 09:10:12,070 fail2ban.actions [26723]: NOTICE [sshd] Unban [IP]
2021-07-25 09:10:12,070 fail2ban.actions [26723]: NOTICE [sshd] Unban [IP]
2021-07-25 09:10:12,070 fail2ban.actions [26723]: NOTICE [sshd] Unban [IP]
2021-07-25 09:10:12,070 fail2ban.actions [26723]: NOTICE [sshd] Unban [IP]
2021-07-25 09:10:12,082 fail2ban.jail [26723]: INFO Jail 'sshd' stopped
2021-07-25 09:10:12,189 fail2ban.actions [26723]: NOTICE [apache-noscript] Flush ticket(s) with iptables-multiport
2021-07-25 09:10:12,286 fail2ban.jail [26723]: INFO Jail 'apache-noscript' stopped
2021-07-25 09:10:12,289 fail2ban.database [26723]: INFO Connection to database closed.
2021-07-25 09:10:12,289 fail2ban.server [26723]: INFO Exiting Fail2ban
Notice the dates.
fail2ban.log.1 is an empty file
fail2ban.log has the startup logs caused by me restarting the service above.
I need to find out more information about this. Was fail2ban offline since July 25? Why did it exit?
I also checked php-fpm logs and they're filled with lines like
[05-Aug-2021 05:31:40] WARNING: [pool 154995045616202] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are are 0 idle, and 2897 total children
Yeah that would go out of memory fast. If I could find out which server/domain started these servers, that would be a good start to the investigation.
Ubuntu: 18.04 LTS
Fail2ban: 0.10.2
Update: I think what's happening here is that the server was rebooted on July 25 but fail2ban didn't start automatically. I've made necessary changes now and it should come back up. Will update here what I find.
