5

Is there a way to get recorder real network traffic to web server, e.g. from web server logs (Apache), and replay this traffic to either profile web application (in Perl) under real load, or benchmark and compare speed of different implementations before choosing one or the other?

If it matters, webapp is written in Perl, and runs under plain CGI, FastCGI, mod_perl (via ModPerl::Registry), PSGI (via Plack::App::WrapCGI).

Crossposted to Pro Webmasters

Similar questions on Server Fault:

Community
  • 1
  • 1
Jakub Narębski
  • 309,089
  • 65
  • 217
  • 230

5 Answers5

3

A quick scan on Google for this yielded an interesting blog entry with subsequent, useful comments are at http://www.igvita.com/2008/09/30/load-testing-with-log-replay/. A commenter also mentioned Tsung by Process-One that allows for recording sessions real-time, with the obvious note that you should be able to replay it back. That doesn't help so much with existing Apache access logs though.

Emond
  • 50,210
  • 11
  • 84
  • 115
zerolagtime
  • 181
  • 3
  • First link gives me "Page not found", unfortunately – Jakub Narębski Nov 08 '10 at 20:33
  • 1
    Try the Google cache at http://www.google.com/url?sa=t&source=web&cd=1&ved=0CCcQIDAA&url=http%3A%2F%2Fwebcache.googleusercontent.com%2Fsearch%3Fq%3Dcache%3ATGz8YxnCvcgJ%3Awww.igvita.com%2F2008%2F09%2F30%2Fload-testing-with-log-replay%2F%2Bload-testing-witlh-og-replay%26cd%3D1%26hl%3Den%26ct%3Dclnk%26gl%3Dus&ei=P4fZTOv8BcLflgfp5YGDCQ&usg=AFQjCNG1yZBmtfxBJO19whcRu4nEjz_7mA&sig2=fzkGw6lyW_g5eczMxXFEQA – zerolagtime Nov 09 '10 at 17:39
2

Been here lately. I figured that if I dumped tcp traffic with tcpdump I could rewrite the destination of the packages and then replay it to the new app servers. So I started out with something like this:

tcpdump -i eth1 dst -s 0 -w - port 80 | \
tcprewrite --mtu-trunc --infile=- --outfile=- \
--dstipmap=<source_ip>:<destination_ip> | \
tcpslice -w - - | tcpreplay --intf1=eth1 -

It did not work for various reasons, so I started digging some more and found Gor: a small Go project by Leonid Bugaev from Granify, written for exactly what we wanted to accomplish here.

This is how we ended up using Gor: http://devblog.springest.com/testing-big-infrastructure-changes-at-springest/

We have a Chef cookbook for it as well: https://github.com/Springest/gor-chef

Hope this helps.

wrdevos
  • 2,029
  • 16
  • 19
1

Here's a simple perl way to record real http traffic and play it back:

http://patrick.net/sprocket/rwt.html

Patrick
  • 11
  • 1
1

Short answer was given on the otherside.

Longer answer is that you can't: you will be missing request headers and POST bodies.

Leolo
  • 1,327
  • 9
  • 14
  • For the specific web application I want to benchmark / profile it doesn't matter: it uses only GET requests, and there is virtually no difference wrt. request headers (besides serving as 'text/html' vs 'application/xml+xhtml'). – Jakub Narębski Nov 08 '10 at 10:53
0

If only GET requests are needed and there is no session-tracking implemented via query parameters, then this is possible.

One question: do you want to do it this way because (1) you want to emulate real-world distribution of traffic among your pages or (2) there are too many pages to even consider building any sort of test scripts?

CMerrill
  • 1,857
  • 1
  • 14
  • 16
  • The problem is not only optimizing for a real world traffic, but the fact that real traffic generates I/O pressure that is felt by web app (which is I/O rather than CPU or memory, or network bound). There is main page, 800+ subpages, then those subpages have many subsubpages... http://git.kernel.org – Jakub Narębski Nov 08 '10 at 15:47