8

We're seeing latencies of ~100-300ms for services accessing a remote resource within the same subnet, yet when we use ping (ping size was of a similar size as of packets sent normally), we see responses of 5ms or less. Anything else we should check?

An excerpt from resource monitor

Mindaugas Mozūras
  • 181
  • 1
  • 1
  • 4
  • 12
    Just as an aside, the 10.0.0.0/8 network is reserved for private networks and is not publicly routable. Obscuring part of the IP address is therefore not necessary. More info here - http://en.wikipedia.org/wiki/Private_network#Private_IPv4_address_spaces. – Buggabill Dec 21 '11 at 20:47

4 Answers4

21

What does "latency" mean ?
Are you comparing ICMP echo response times with application processing round-trip times ?

If you are, that would explain it - and also tell you not to compare apples and oranges :)

adaptr
  • 16,576
  • 23
  • 34
  • 4
    Agreed - do a packet capture on the server and watch how long it takes for the server to issue a reply to application requests. – EEAA Dec 21 '11 at 17:01
  • 3
    …but they're both *round* (and that works for RTT and the specified fruits!) – voretaq7 Dec 21 '11 at 17:02
  • 4
    ..and ICMP packets are 64 bytes. Their overall and measured latency is pretty much guaranteed to be better than anything containing actual data payload. – adaptr Dec 21 '11 at 17:03
  • 2
    On top the ping takes nearly no time on the server o be processed AND you may have QOS prioritizing pings (I do on´ some links). – TomTom Dec 21 '11 at 17:16
  • I probably expressed myself poorly - I'm not exactly comparing the two (I understand that ping uses ICMP), just stating one of the symptoms and looking for more answers/questions that would help me go into correct direction. – Mindaugas Mozūras Dec 21 '11 at 17:24
  • 1
    I would say: check network, interfaces and stuff. Maybe your apps show such latency because of reset messages due to problem on interfaces – Alfabravo Dec 21 '11 at 17:25
  • ...no, mostly, network applications have latency because they... DO THINGS ;) – adaptr Dec 22 '11 at 09:41
8

can it be you have packet losses [that cause reteransmissions] for instance for larger packets?

maybe try ping -s 1400 address under linux or ping -l 1400 address under windows.

if that does not help try looking at the traffic with wireshark - maybe there are some strange re-transmissions, corrupted packets?

pQd
  • 29,981
  • 6
  • 66
  • 109
3

A discrepancy like that usually shows an issue in the application / server. While it is possible that your ping is different due to QoS / network prioritization, that's not the first place I'd look if you're talking about systems on the same local network.

What's your latency time accessing the service from the same host it is served by? Does that host provide other services faster? Is it under a heavy IO load?

Jeff Ferland
  • 20,547
  • 2
  • 62
  • 85
1

As you're suspecting here, ping latency != a TCP round trip time, particularly if there's data being transferred. It's possible that all of this this is due to delayed ACKs (which can impose ~200ms, the common default setting) or the bad interaction between Nagle/and DACK which can push it up to ~500ms. A packet capture is in order here.

Also note that the delayed ack timer is independent of a TCP flow, and is instantiated when the system boots. So it can fire any time between 1-200ms inside of a flow, and commonly causes a 'range' style delay as you describe.

mcauth
  • 430
  • 2
  • 5