4

We make a lot of HTTP requests. Recently, we've started thinking about optimizations at the OS level to make more requests from a single machine.

In order to check the performance of our OS, I've created a small benchmark to compare things on different machines. The benchmark uses curl -w like that:

#!/bin/bash
for (( ; ; ))
do
  curl $URL -o /dev/null -s -w "SIZE:   %{size_download}    SPEED: %{speed_download}    LOOKUP: %{time_namelookup}  CONNECT:    %{time_connect} START:  %{time_starttransfer}   TOTAL:  %{time_total}\n"
done

Now I ran it for for 1 single URL. The results are as follows: From my local dev machine (connected with Fiber):

SPEED (b/sec)   LOOKUP  CONNECT START   TOTAL
13,331.2481     0.0022  0.0228  0.2163  0.2175

On one of our production servers (with XEN virtualization), the results are slightly different:

SPEED (b/sec)   LOOKUP  CONNECT START   TOTAL
22,764.7700     0.0093  0.0455  0.1318  0.1334

And on another unrelated server, without XEN virtualization (different data center, different continent, further from the resource)

SPEED (b/sec)   LOOKUP  CONNECT START   TOTAL
32,821.3569     0.0004  0.0080  0.1608  0.1672

As you can see the performance on our production server is far from satisfying. The data transfer rate is faster than on my local laptop, but latency is killing us. Since we fetch HTTP resources whose size is rather small, we need to optimize this latency.

Any idea where to start?

UPDATE: this is not about scaling a web server. It's about scaling web requests.

Julien Genestoux
  • 609
  • 8
  • 19
  • Interesting topic, sometimes we have the same issue, we thought to be this problem is related to DNS however from this point of view I will rething about the problem. We were just investing in our DNS services to solve this problem. – Harun Baris Bulut Dec 14 '13 at 13:32
  • By the way Xen has %5 of CPU overhead and if you have iSCSI storage you may need to check the storage performance. Plus if your DNS server is in that Xen environment also, try to take that out of the environment. – Harun Baris Bulut Dec 14 '13 at 13:35
  • Do you have a local dnsmasq or similar service running? A lack of that might explain your slow lookup times on your production server. – Michael Dec 14 '13 at 14:19
  • Michael, we do and we'll probably need to fine tune it. However, based on what we see the CONNECT is even worse than the DNS lookup. – Julien Genestoux Dec 14 '13 at 14:26

2 Answers2

1

This is a well studied problem ("high performance web crawling"), with loads of available research: http://scholar.google.com/scholar?q=web+crawling+performance ... yes, I'm cheating, but honestly, you should go through the literature first.

Based on my own experience with building this type of system in the past: you can't beat speed of light, so one way or another you will incur it. What you can do is optimize how and when you schedule resource fetches. For example, you can have optimized subsystems for dealing with parts of the problem - e.g. DNS resolution. You can pre-resolve names and connect directly to the IP address (just add the right Host header). After that you will have to incur the TCP connect cost, no way around it. That said, if you have multiple requests to same host, then you can leverage that to serialize multiple requests over an existing connection: this helps amortize the TCP/TLS handshake costs and gives you better end-to-end performance. From there, you have to move up the protocol ladder: sometimes you can track redirect chains and remember the last location to skip the extra redirects in the future (just have a fallback). In fact, same applies for DNS.. You can implement an optimistic strategy and connect directly to the IP and then use a fallback if that fails. For TLS you can store session tickets and other metadata to get a faster reconnect (that is, assuming you reconnect frequently enough).

tl;dr: I'm not adding anything new here, all of the above tips (and more) are covered in available research. Grab a coffee, spend some time reading through the existing papers!

igrigorik
  • 346
  • 2
  • 3
0

I don't know where your http requests are going but you could see whether the webserver(s) in question support SPDY.

Developed at Google, SPDY is an attempt to pipeline multiple https requests for greater throughput and lower latency.

I'd also second any recommendations above around DNS optimization. You really want to setup a caching forwarding DNS to make things faster. If you have any control over the TTL of the webserver(s), it pays to increase them as far as you are comfortable.

dmourati
  • 25,540
  • 2
  • 42
  • 72