Netstat and ss doesn't see connection

Question

I'm trying to find which application running on Kubernetes cluster is using a specific port to communicate with RabbitMQ cluster. On RabbitMQ I see that connection is coming from 192.168.1.10:34226. 192.168.1.10 is one of Kubernetes nodes. After logging in to 192.168.1.10 I checked if connection is established from that node by tcpdump command tcpdump -i eth0 port 34226. I see traffic with both directions. Now I'm trying to find which process is making this connection. First I tried netstat:

netstat -tapn | grep 34226

and it found nothing. Then I tried ss but still nothing. Connection is established, with tcpdump I can see heartbeat packets sent to RabbitMQ and response from RabbitMQ. But netstat and ss don't report established connection.

UPDATE 1

I found that is something related to network namespaces. My question is similar to this question but I need to find PID of process which make connection from 34226 port.

Try `kubectl get svc`, and look for the port. That will lead you to which `POD` is the `Service` exposing. `kubectl get po` will show you list of pods. Keep in mind that if they are not on a default namespace you might have to add `--all-namespaces=true` — Crou, Feb 22 '19 at 14:26
It's not matter of exposed ports but ports used to initiate connection from container. — QkiZ, Feb 22 '19 at 14:55
I think you need to be more specific or describe it in a different way. If you are looking a way to login to a `POD` inside a cluster you can do `kubectl exec -it --/bin/bash` — Crou, Feb 27 '19 at 11:11
I think that I described problem understandable. I know how to run command in container. I'm trying to find which container is making connection on outgoing port. — QkiZ, Mar 01 '19 at 16:38

score 5 · Accepted Answer · answered Apr 02 '19 at 14:01

5

After couple of weeks I found answer. I needed to run netstat -tapn in all namespaces with following command:

sudo ip -all netns exec netstat -tapn | grep 34226

Now I have PID number of process which start tcp connection from 34226 port.

answered Apr 02 '19 at 14:01

QkiZ

634
2
9
22

Not working for me on ubuntu 18.04 – spuder May 11 '20 at 18:15
Maybe you dont have any namespaces – QkiZ May 12 '20 at 20:28
Doesn't work for me on Ubuntu 18 either. I'm running a docker container listening on port 3003 (via docker run -p 3003:3003) and it doesn't show up. It *does show* when I find the container PID and invokes netstat like this: `nsenter -t 13401 -n netstat -tapn` – Juraj Martinka Jul 07 '20 at 10:01
More information can be found here: https://unix.stackexchange.com/a/522435/63528 (the custom script works well) – Juraj Martinka Jul 07 '20 at 10:14
Docker doesn't use network namespaces. The question is related to Kubernetes which deals with networks differently. – QkiZ Feb 25 '21 at 12:49
worked fine on GKE nodes – sivann May 18 '23 at 07:49

Netstat and ss doesn't see connection

1 Answers1