7

I'm wondering about various techniques/tips/tricks/silver bullets fellow SO'ers utilize when investigating network latency issues and the various solutions and optimizations that are Incorporated.

I'm looking for solutions that can be applied to both software or hardware.

For example:

  1. Increase socket buffer sizes
  2. Use OS's best option for reactor pattern (select is not always the best option)
  3. Use of extreme end NICs such as bigfoot Killer 2100
  4. Keep network cables as short as possible, do not tie network cables around each other
  5. When copying large amount of data between 2 points - take into account round trip latencies and use multiple socket connections to fully saturate link

What are some others suggestions?

BalusC
  • 1,082,665
  • 372
  • 3,610
  • 3,555
  • There is **no such thing** as a silver bullet - except, of course, for those things that I keep loaded in the anti-werewolf gun under my pillow. – Matt Ball Dec 08 '10 at 02:17
  • @Matt: I was trying to be ironically humorous :D –  Dec 08 '10 at 02:18
  • 1
    Should this be on ServerFault? – El Yobo Dec 08 '10 at 02:53
  • lol at Killer NIC, how about a Type R sticker? – Steve-o Dec 08 '10 at 03:33
  • 2
    Low latency and high bandwidth are competing goals. For example, increasing socket buffers and fully saturing connections are good for bandwidth, but terrible for latency. You need to decide which goal is more important to you. – caf Dec 08 '10 at 04:03
  • @El Yobo: The question straddles software/hardware solutions. SO seems more appropriate - Though I may reword the question specifically for h/w options and post it on severfault. –  Dec 08 '10 at 05:04
  • @Steve-o: Didn't get the reference, please explain. –  Dec 08 '10 at 05:19
  • Avoid the killer NIC in my opinion. I've had to reinstall windows machines a few times due to their shitty drivers. – JimR Dec 08 '10 at 21:01
  • @JimR: Been using killer2100 on ubuntu 10.4 for sometime now, never had a problem with it or its drivers, it just works out of the box. –  Dec 09 '10 at 04:35
  • @Oxsnarder: I've never used it in Linux which is why I said Windows. Maybe the Linux driver authors were competent. :) – JimR Dec 11 '10 at 21:35
  • @Oxsnarder: KillerNIC is snake oil, it cannot from their own documentation accelerate TCP which almost any cheap gigabit NIC can, or excelled at with ToE from Intel Server NICs. – Steve-o Dec 29 '10 at 04:29

2 Answers2

2

Here are a couple things that have helped me speed things up or increase bandwidth over the last 10 years or so...

1) Turn off the Nagle algorithm when your nodes are physically close together on a fast link and the data in the packet is not smaller than the TCP packet overhead. MS did a paper about this. A counter example to this is when you have a satellite connection in your link and a packet takes 800 milliseconds one way due to the distances involved. In that case, Nagle is a good thing.

2) Make sure you have the MTU size set correctly for your link and your machine to avoid fragmentation issues.

3) If you're on Windows and outgoing packets are small and the latency is high, you should also set the socket option SO_SNDBUF to 0 to avoid transmission delays.

JimR
  • 15,513
  • 2
  • 20
  • 26
  • 1
    How does one go about empiracally determining the MTU between so the NIC and the switch it is connected to? –  Dec 08 '10 at 05:14
  • 1
    @Oxsnarder: I do it with ping on windows. ping -l -f yahoo.com . Ping will complain that the packet needs to be fragmented until the size of the packet is small enough to fit in the MTU. Size includes the TCP and IP header. Take a look here: http://en.wikipedia.org/wiki/Path_MTU_Discovery – JimR Dec 08 '10 at 21:00
0

It's not always possible but if you can insert locally caching servers nearer to the destination that can help.

Pace
  • 41,875
  • 13
  • 113
  • 156