0

I'm trying to find a problem server hiding behind a virtual IP (using LVS/ipvs). I've got a test program that sends requests to the virtual IP until it gets the bad response, but how can I tell to which real IP a request to the virtual IP got routed?

On the box doing the virtual IP magic, here's the virtual IP configuration (for the service I care about):

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
...
TCP  10.1.0.254:5025 nq
  -> 10.1.0.5:5025                Route   1      0          1         
  -> 10.1.0.6:5025                Route   1      0          5         
  -> 10.1.0.7:5025                Route   1      0          2         
  -> 10.1.0.9:5025                Local   1      0          3         
  -> 10.1.0.11:5025               Route   1      0          3         
...

My client program is sending TCP requests to 10.1.0.254:5025, usually getting a good response but sometimes a bad response. With this few servers, I could send my request to each server in turn until I discover the culprit, but I wonder if that technique will scale as we add servers. What means exist for me to find out where requests got routed?

  • Kernel: Linux 2.6.32
  • OS: Debian testing (whatever that's called these days).
  • ipvsadm is version 1.25, compiled with ipvs v1.2.1
Wayne Conrad
  • 675
  • 1
  • 7
  • 20

3 Answers3

1

This is quite a typical problem with IPVS/ldirectord, you have several options to try to resolve this

  • Check logging on the destination servers looking for the request (not very accurate but most of the time it'll cover the bill)
  • If your destination server replies with headers (like apache or ftp) just add a new custom header like X-Served-by or some hint in the welcome header
  • Activate logging on ldirectord, the logging is a bit obscure but I'm pretty sure it'll be useful, just add to your ldirectord.cf logfile="/var/log/ldirectord.log" in the global options
lynxman
  • 9,397
  • 3
  • 25
  • 28
  • We don't appear to have an ldirectord. keepalived, we got. In any case, although our servers don't currently reply with headers, the protocol is under our control so we can add any headers we like. I don't know why we didn't think of that before. Thanks! – Wayne Conrad Feb 02 '11 at 05:38
1

ipvsadm -Lcn on the LB (aka "director") should list your connections. Grep for the IP you are searching.

Nils
  • 7,695
  • 3
  • 34
  • 73
0

You best bet would be to rn tcpdump on the real servers to see where you request is going.

Gotcha. LVS-TUN listen on the on the virtual interface for the client address because the packets are encapsulated you wouldn't be able to identify the packet based on users IP

$ tcpdump -i tunl0 host USER_IP

Sameer
  • 4,118
  • 2
  • 17
  • 11
  • Thanks. The IPVS box has a tunl0 device, but it's not up. `ifup tunl0` reports "Ignoring unknown interface tunl0=tunl0". – Wayne Conrad Dec 21 '10 at 21:43