High request time

Question

I've got high execution time of HTTP request (ex. 42KB in 13s).

I'm using haproxy to load balance traffic. Static files are server by lighttpd behind varnish accelerator. So... it should be served below the speed of light :P

http://img580.imageshack.us/g/34411388.png/

Long DNS Lookup: pic1

Long receive: pic2

Simulationous download: pic3

I use GoogleDNS. The servers are in the USA. Firstly I added IPs to /etc/hosts to ommit DNS query. Secondly I migrated static to separate machine omiting haproxy. In that's way it is connecting direct to varnish. The traffic is not big, because it's only development, but it happens also in production.

http://img580.imageshack.us/g/34411388.png/

After changes: pic4

It's a bit better, but it's still sloooooow...

Answer 1

First of all, you must check from where Varnish is serving the image. Is it caching the image or using the backend at each time?

You can check that in many ways:

1. Use the command line switch to trace the VCL and see exactly what request is doing ( -p vcl_trace=true if my memory works well.
2. Use varnishlog to watch some request to the image and see what is happening (if it's serving from cache, etc). You must use this if you set the vcl_trace.
3. Add a header to the response delivery and examine it on the browser to see if it was a hit or miss.

Like this:

sub vcl_deliver {
   if (obj.hits > 0) {
                set resp.http.X-Cache = "HIT";
                unset resp.http.cookie;
        } else {
                set resp.http.X-Cache = "MISS";
        }
}

It will add a X-cache header on each request Varnish serves. After checking if varnish is indeed caching correctly, we can search for the reason of the long time to answer, but I suppose that watching the requests on trace will give you some hints.

Answer 2

You could be hit by Buffer Bloat.