1

I have 2 servers sitting in a rack running Ubuntu 16.04 with a 1 meter Ethernet cable between them, both having standard Intel Ethernet adapters.

The ping between the two is about 300 us (microseconds).

This is a standard latency I've seen in most Gigabit-Ethernet setups.

But this latency still seems quite high compared to theoretical limits; why is it? I have read that 1 GbE can achieve 40 us latency.

Is this the minimum latency I can expect, or is there software tuning I can perform to reduce this latency? What is the bottleneck? Is it Linux? On this gamer web site for Windows the tool in the screenshot seems to suggest 40 us latency in most cases, but that doesn't help me much for my Linux servers.

(How) can I make my ping 40 us?

EDIT: Looking at the screenshot again, it might be that the 40 us shown are not actually roundtrip time, but it is actually a specific delay within the Windows kernel and thus the 40 us may only be part of the total roundtrip time, which may be higher and is not listed. This would also be in line with the answers in here.

(I originally asked this question at superuser; at this time it wasn't clear to me that ServerFault would be a more appropriate community to ask network performance questions, and I don't have enough reputation there to move the question, so I reposed here. I have also switched the hardware to server hardware.)

nh2
  • 818
  • 3
  • 11
  • 21
  • @ceejayoz According to [this](https://meta.stackexchange.com/questions/7334/any-guidelines-to-find-the-correct-place) and the [FAQ for ServerFault](https://serverfault.com/tour) and the [FAQ for SuperUser](https://superuser.com/tour), SuperUser is for *"computer software or hardware"* and ServerFault is for *"every question about server, networking, or related infrastructure"*. This is a server/networking/infrastructure question, and one that experienced network engineers are more likely able to answer than PC power users. – nh2 Sep 18 '17 at 00:44
  • 1
    @RonMaupin I took that quote to which you anwered "no" directly from the [ServerFault FAQ](https://serverfault.com/tour). I have this problem in my business environment, in particular I'm trying to reduce network latency to make my [GlusterFS](https://www.gluster.org/) network file system faster. – nh2 Sep 18 '17 at 01:26
  • @ceejayoz I did not change my question in order to make it on-topic here. I have this problem with real GlusterFS servers right now; figuring this out is [part](https://bugzilla.redhat.com/show_bug.cgi?id=1478411) of my job. And there is an actual problem to be solved: The latency between my servers is 10x as high as it can reportedly be. – nh2 Sep 18 '17 at 01:37

2 Answers2

11

A ping does not measure latency nor does it measure round-trip time. It measures ICMP echo request response time. ICMP messages run with low priority and take way longer than serious traffic.

As @barbequesauce has pointed out, twisted pair requires some complex encoding, so 1000BASE-T NICs will require at least 10 µs per side for the codecs (plus system bus and IRQ latency). Twisted-pair cat.6 has a velocity factor of 65% (~200,000 km/s), so it adds .05 µs or 50 ns per 10 m. Fiber isn't really faster (VF of 67%) but uses simpler encoding with maybe 5 µs per side instead of 10 µs.

The IP stack will add some more µs to your latency budget. On a fast system you could maybe get away with ~20 µs.

Of course, the exact values for the NICs, your system bus and the IP stack depend on your hardware and software. The figures above are about as good as you can get. And don't use ping to measure.

In a nutshell, you most certainly have no real problem at all, the problem is with interpreting the ping output.

Zac67
  • 10,320
  • 2
  • 12
  • 32
  • Thanks for your answer, some clarifying questions: (1) What would you use to measure instead? I've used `netperf` (in `TCP_RR` and `UDP_RR` modes) and it reports numbers in line with `ping`. Is it correct to expect ping to be equally fast given that there's a direct cable between both machines and they are otherwise idle, so there is no congestion ("serious traffic")? (2) Regarding IP stack latency, are the ~20 ms for each way through the stack, i.e. should I expect 4x 20 ms IP stack latency in a full roundtrip, or a different factor? Thanks! – nh2 Sep 18 '17 at 20:32
  • 2
    Ping _may_ report values close to real round trip times but there's no guarantee. Especially with network hardware, ping is usually handled by the cpu and may be orders of magnitude slower than 'real' traffic. Note that I was counting µs, not ms. All in all, IP stack, encoding, transfer, decoding, IP stack and back you can maybe get away with 120 µs (minimum frame, maximum adds 12 µs per hop). Lowest latency mandates that you deactivate all power management (CPU, PCIe). – Zac67 Sep 18 '17 at 20:56
  • The `ms` was a typo, well spotted, and thanks for the extra information! – nh2 Sep 18 '17 at 21:27
  • Accepting this answer because it contains the most detail. – nh2 Sep 25 '17 at 20:36
4

So two things to consider -

Copper adds latency, compared to fiber - 15us per side of an exchange, if I remember correctly from my HFT days. A 1m copper cable will intro 60us of delay all by itself. Fiber, on the other hand, is ~5us.

The other (arguably more important) consideration lies in IP stack tuning and serialization delay. Out of the box, the stack might be tuned for smaller packets; in that case ping would be chunked up for sending - and receiving - so the stack would slow down response time... make sure both sides are tuned for the same sizes to be sure.

barbequesauce
  • 41
  • 4
  • 1
    The actual propagation delay is pretty negligible for copper UTP, compared to other latencies, because the max distance supported is only ~100 metres at ~2/3rd the speed of light. e.g. https://en.wikipedia.org/wiki/Transmission_time#Propagation_delay calculates that 100m of UTP adds 0.5 us propagation delay to the total latency from send to receive. So the difference between copper and fiber is all in the encoding overhead, not the actual physical propagation over the medium itself. – Peter Cordes Jun 28 '19 at 23:33
  • Absolutely right - if it's a single piece of copper. It's the exchanges that add the latency I'm referring to - RJ45 head of a patch cord into a jack on a motherboard or into a patch panel, the other side of the patch panel to a patch cord, etc. The physical transitions add up. – barbequesauce Aug 01 '19 at 16:33