-1

Good'day

Just out of curiosity I did a tcpdump port 80 on one of my remote virtual machines. I see a huge amount of requests to this http-port, 95% going to some AWS EC2 instance. Strange thing: The Web-Server was not on high load at all. The packets from tcpdump look like this:

20:57:16.860028 IP user123.example.net.http > ec2-55-155-123-123.ap-northeast-2.compute.amazonaws.com.12345: Flags [S.], seq 1234567, ack 12345567, win 12300, options [mss 1460], length 0

(user123.example.net.http is the address of my remote machine)

All of them looks basically the same, only thing that changes, is the actual EC2-adress (aka IP-adress) and the destination port. Sometimes it's an incoming packag (to port 80, too), but most of them are outgoing.

I run this tcpdump on port 80 today, from the office now. Now the traffic is between the remote virtual machine and another IP-adress, not the AWS-instance anymore:

08:34:46.369161 IP user123.example.net.http  > 123.12.123.132.37176: Flags [R.], seq 0, ack 2821777444, win 0, length 0
08:34:46.388933 IP 123.12.123.132.55539 > user123.example.net.http : Flags [S], seq 2934790784, win 29200, length 0

In this case, the IP-Adress does not change.

Only thing that changed: I remotely logged in from my working place now, using another IP-Adress (not the one you can see in tcpdump) and another SSH-client.

Investigation

What I tried so far, without success:

  • I checked the logfile of nginx, they don't show any traffic (I am not running huges sites, just some little private projects)
  • I stopped nginx, the only apparent port80-daemon in this setup
  • I stopped all other non-system services: no changes

Current setup

I am running nginx on this server on port 80 and 443, but traffic is generally routed to HTTPS. Also I am running PHP 7.3 with FPM, but on sockets, not via TCP.

Besides that, I am running some e-mail services (dovecot, postfix, ...), Unbound, a *MySQL-*Server and an *ELK-*stack. This is what netstat -tulpn confirms: (I x'ed my public IP-Adress and my SSHD-port, which is not the default one)

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      858/unbound
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      2806/master
tcp        0      0 x.x.x.x:25              0.0.0.0:*               LISTEN      2806/master
tcp        0      0 127.0.0.1:8953          0.0.0.0:*               LISTEN      858/unbound
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      7942/nginx -g daemo
tcp        0      0 0.0.0.0:4190            0.0.0.0:*               LISTEN      628/dovecot
tcp        0      0 0.0.0.0:993             0.0.0.0:*               LISTEN      628/dovecot
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      491/mysqld
tcp        0      0 127.0.0.1:587           0.0.0.0:*               LISTEN      2806/master
tcp        0      0 x.x.x.x:587             0.0.0.0:*               LISTEN      2806/master
tcp        0      0 0.0.0.0:143             0.0.0.0:*               LISTEN      628/dovecot
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      7942/nginx -g daemo
tcp        0      0 0.0.0.0:???             0.0.0.0:*               LISTEN      27952/sshd
tcp6       0      0 ::1:53                  :::*                    LISTEN      858/unbound
tcp6       0      0 ::1:25                  :::*                    LISTEN      2806/master
tcp6       0      0 ::1:8953                :::*                    LISTEN      858/unbound
tcp6       0      0 :::443                  :::*                    LISTEN      7942/nginx -g daemo
tcp6       0      0 :::4190                 :::*                    LISTEN      628/dovecot
tcp6       0      0 :::993                  :::*                    LISTEN      628/dovecot
tcp6       0      0 ::1:587                 :::*                    LISTEN      2806/master
tcp6       0      0 :::143                  :::*                    LISTEN      628/dovecot
tcp6       0      0 :::80                   :::*                    LISTEN      7942/nginx -g daemo
tcp6       0      0 :::???                  :::*                    LISTEN      27952/sshd
udp        0      0 127.0.0.1:53            0.0.0.0:*                           858/unbound
udp6       0      0 ::1:53                  :::*                                858/unbound

After stopping all of the above mentioned servies, this is the output - Packets on port 80 still remains: 

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:???             0.0.0.0:*               LISTEN      27952/sshd
tcp6       0      0 :::???                  :::*                    LISTEN      27952/sshd

That's the output of service status-all|grep + (marked some with an * - those didn't stop when stopping the parent process, still trying to figure that out)

 [ + ]  amavis-mc *
 [ + ]  amavisd-snmp-subagent *
 [ + ]  apache-htcacheclean *
 [ + ]  apparmor
 [ + ]  cron
 [ + ]  fail2ban
 [ + ]  gdomap
 [ + ]  ip6tables
 [ + ]  iptables
 [ + ]  lm-sensors
 [ ? ]  modules_dep.sh
 [ + ]  nscd
 [ ? ]  php-fpm-chroot-setup.sh (make PHP available after reboot)
 [ + ]  procps
 [ + ]  quota
 [ + ]  rc.local
 [ + ]  resolvconf
 [ + ]  rsyslog
 [ + ]  ssh
 [ + ]  sysstat
 [ + ]  udev
 [ + ]  unattended-upgrades
 [ + ]  urandom
 [ + ]  uwsgi

Question is..

Why is there traffic on port 80, even if nginx is not running? Where does this traffic comes from (what process)?

n.r.
  • 289
  • 1
  • 2
  • 10
  • Are you sure the packet source is `user123.example.net.http`? The port http points to a webserver. – Eduardo Trápani Oct 29 '19 at 20:28
  • Can you please edit your question to be a bit more precise. Is this your web server making outgoing requests to a variety of EC2 servers, or your home PC? What other software is running on the server? If you shut down the services one by one which one stops the requests being sent? – Tim Oct 29 '19 at 21:01
  • @EduardoTrápani Gotta admit, I don't understand your question. Or I am confused as you are. As far as I understand tcpdump, the left side is the packet source, in this case port 80 of my local machine. So: Yes, I am sure. – n.r. Oct 29 '19 at 21:01
  • @Tim Sure, added some more details. – n.r. Oct 29 '19 at 21:03
  • @Tim Didn't try stopping services one by one, thought it has to do something with nginx, but will try this definetely tomorrow. – n.r. Oct 29 '19 at 21:11
  • Your question is still vague and imprecise. When you edit it, don't just tack info on, you can rewrite the whole question to make sense. If you're confident it's Nginx then shut it down, show it's not happening, then edit your question to include the full Nginx configuration. Then we can help. – Tim Oct 29 '19 at 23:29
  • What your packet capture shows is not the behaviour you are describing. – womble Oct 30 '19 at 01:04
  • @Tim sorry, your totally right, I rewrote it, hopefully it's better to read now. – n.r. Oct 30 '19 at 07:40

1 Answers1

1

If your webserver initiates the outgoing request on port 80, stopping your nginx webserver doesn't necessarily help because many services can initiate an outgoing connection on port 80.

You won't get any more detail with tcpdump because it's on the ethernet level. You need to find the source of the problem. Most of the time, when you see such a high requests of outgoing traffic, it's originating from the outside, executing for example a PHP script on one of your sites. But since you disabled Nginx, this is not obvious. But double check if really nothing is listening on port 80 or 443 anymore when nginx is disabled, since such a PHP file can be called from 443 or any other port your webserver is listening on. Check if something still listens on a specific port with this command:

netstat -tulpn

If really nothing is running on port 80 or 443 anymore and you still see the initiated outgoing port 80, I would block incoming requests, see if it's originating from your webserver itself for sure.

If that's the case, check all running services, because only a running service can then be the initiator. You can also use:

netstat -tulpn

or

service --status-all

Check for suspicious services and try to disable them, see if that helps.

Erjen Rijnders
  • 201
  • 1
  • 5