0

Im running a Centos 7 VPS server for the past several years. Originally when I set it up I didnt install any firewall (iptables specifically) because I was under the impression the host (linode) needed them open to admin the box from their side. Yes I know, really dumb mistake. Other things like ssh, apache etc I locked down and hardened well along with using fail2ban and I keep things patched on a daily basis where required.

At this point I believe Im likely hacked after closing things up via iptables and locking things down. It's a LAMP box. I have no proof at this point just suspicion. Im the only user and things are working fine.

When I run nmap against the server now I have everything locked down the way I want it except I see the following open even though iptables should be blocking 8080 via a DROP:

sudo nmap -sS -O -p8080 x.x.x.x

Starting Nmap 7.94 ( https://nmap.org ) at 2023-06-09 19:52 MST
Nmap scan report for mydomain.com (x.x.x.x)
Host is up (0.018s latency).

PORT     STATE SERVICE
8080/tcp open  http-proxy
Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port
Aggressive OS guesses: Linux 2.6.32 (94%), Linux 2.6.32 or 3.10 (94%), Linux 2.6.35 (94%), Linux 3.5 (94%), Linux 4.2 (94%), Tandberg VCS video conferencing system (94%), Synology DiskStation Manager 5.1 (94%), Linux 2.6.39 (92%), Linux 3.10 (92%), Linux 3.10 - 3.12 (92%)
No exact OS matches for host (test conditions non-ideal).

Running lsof and doesnt show me anything on port 8080 however I do see some rpcbind listed and am unsure if any of these could be related?

COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
rpcbind   451    rpc    6u  IPv4  25873      0t0  UDP *:sunrpc 
rpcbind   451    rpc    7u  IPv4  25922      0t0  UDP *:compaq-evm 
rpcbind   451    rpc    8u  IPv4  25923      0t0  TCP *:sunrpc (LISTEN)
rpcbind   451    rpc    9u  IPv6  25924      0t0  UDP *:sunrpc 
rpcbind   451    rpc   10u  IPv6  25925      0t0  UDP *:compaq-evm 
rpcbind   451    rpc   11u  IPv6  25926      0t0  TCP *:sunrpc (LISTEN)

From the outside I can telnet x.x.x.x 8080 and it responds, however trying to use a common GET HTTP/1.1 command doesnt reply at all and it times out after a minute or so.

When I telnet in on port 8080, I run lsof | grep http and I see the following:

lsof | grep http
NetworkMa   514          root  mem       REG                8,0    162144       5038 /usr/lib64/libnghttp2.so.14.17.0
gmain       514   517    root  mem       REG                8,0    162144       5038 /usr/lib64/libnghttp2.so.14.17.0
gdbus       514   519    root  mem       REG                8,0    162144       5038 /usr/lib64/libnghttp2.so.14.17.0

I dont run nghttp2 so Im guessing that it part of Linode admin package they preinstalled?

Any ideas on how to find this process and kill it? A bonus would be understanding what it is as well but Id be happy just to get rid of it.

Thank you for any advice or pointers.

Tim
  • 203
  • 1
  • 9
  • Does this answer your question? [How do I deal with a compromised server?](https://serverfault.com/questions/218005/how-do-i-deal-with-a-compromised-server) – Ginnungagap Jun 10 '23 at 06:14
  • This is a little helpful, though much of it I already know. I'll have to re-install for sure but would like to learn forensically from this mistake and understand it better and hopefully plug this particular hole until I can find time to re-install. – Tim Jun 10 '23 at 14:00

1 Answers1

1

looks like lsof doesn't show sockets owned by user who created that 8080 binding.
From this output it says NetworkMa[nager?] with pid=514 (if I got it correctly without header)

lsof | grep http
NetworkMa   514 ...

Does this service restart automatically? pstree can show a chain with that process (probably it can clarify a bit)

p.s. cannot put it in comments, so just added in answers

yvs2014
  • 111
  • 2
  • Yes, NetworkManager starts automatically. I didnt set it up, it came installed on the machine. I looked at pstree and didnt see anything that stuck out as being the process thats causing this. I know Ill have to re-install the machine at some point I just dont have several days to do so at this point, so plugging the hole is my best choice ATM (realizing that there might be others I dont know about). – Tim Jun 10 '23 at 13:57
  • yes, maybe it's the right choice to be sure – yvs2014 Jun 10 '23 at 15:01
  • another one suggestion to get some extra info if that NetworkManager is problematic: compare checksums of the local /usr/sbin/NetworkManager and its original binary (extracting this file from .deb package of the same version) – yvs2014 Jun 10 '23 at 15:16