1

I'm trying to identify the network traffic between multiple backend microservices running on the same server. (they make some rest http calls to each other)

Unfortunately, when i use tcpdump, the lines i see for each call, only allow me to identify the 'destination service' for the call. 

tcpdump -nn -i lo 
14:03:52.612985 IP6 ::1.31822 > ::1.9093: Flags [P.], seq 2474698995:2474699366, ack 4107952262, win 697, options [nop,nop,TS val 3238273 ecr 3231488], length 371
14:03:52.616946 IP6 ::1.9093 > ::1.31822: Flags [P.], seq 1:875, ack 371, win 568, options [nop,nop,TS val 3238274 ecr 3238273], length 874

The destination port (here 9093) is stable and is the one on which one of my microservices run. But the source port (31822) is one of those 'ephemeral / dynamic port' allocated randomly by the OS so I don't know what application made the call.

Is there an easy way to trace what microservice is the source of that call. A way to link the source dynamic port with a running app PID?

(without modifying the running application themselves, i don't have control of them all)

things i tried like netstat,lsof led me nowhere. Thanks

MikaelW
  • 111
  • 1
  • This could be done by writing a wrapper function to log the appropriate system calls, then using `LD_PRELOAD` to inject it into the applications you want to inspect. I would have thought someone would have already developed a solution for this problem, but I don't know of one off the top of my head. – Robbie Mckennie Oct 11 '17 at 00:42

1 Answers1

0

iptables -I OUTPUT -m owner --pid-owner 1234 -m state --state NEW -j LOG --log-prefix MSX

This will log every connection initiated by the microservice with pid 1234 using a prefix MSX so you can distinguish its connections from other microservices.

You could integrate management of these rules into the init script of the microservice. E.g., for systemd add this as an ExecStartPost command using $MAINPID and have a corresponding ExecStopPost command that removes this rule.

Mark Wagner
  • 18,019
  • 2
  • 32
  • 47