How to test test environment with real data from production

Question

i've the following infrastructure:

1x load balancer (nginx) 4x apache upstream 1x mysql server

The website is pretty busy and i bought other 6 servers that are meant to replace the previous infrastructure.

Since the setup is different i'd like to trasparently test my new server infrastructure with live data. I found https://github.com/wangbin579/tcpcopy but for some reason it's not working for me (it does not send 99% of the requests).

I must say I don't have physical access to the server nor i can manipulate the switch. The two servers are in 2 different networks ( but, of course, they see each other ).

Thanks for your time :)

I am the author of tcpcopy. For what reason tcpcopy is not working for you? what does "it does not send 99% of the requests" exactly mean ? — , Jun 18 '13 at 13:56
hello @wangbin579 thanks for your reply. I contacted you by mail and you solved the problem. For those having the same issue with tcpcopy, i recommend setting up a vpn between the real and test machine. — Andrea Baccega, Jun 21 '13 at 20:30

score 2 · Answer 1 · answered Apr 06 '13 at 12:13

You can try duplicating traffic at the load balancer level using iptables TEE target, from the iptables(8) manpage:

TEE The TEE target will clone a packet and redirect this clone to another machine on the local network segment. In other words, the nexthop must be the target, or you will have to configure the nexthop to forward it further if so desired.

   --gateway ipaddr
          Send the cloned packet to the host reachable at the given IP address.  Use of 0.0.0.0 (for IPv4 packets) or :: (IPv6) is invalid.

   To forward all incoming traffic on eth0 to an Network Layer logging box:

   -t mangle -A PREROUTING -i eth0 -j TEE --gateway 2001:db8::1

unfortunately this could not work in my situation. but the answer could be accepted since it could work for others. — Andrea Baccega, Apr 07 '13 at 21:44

score 1 · Answer 2 · answered Apr 06 '13 at 12:22

1

Fix your logging.

The best way to do this is to replay traffic (maybe at a different speed) using actual data from your logs. Standard apache logs are not good enough as they don't include all the headers (assuming your application uses these headers to e.g. serve simplified pages to web crawlers) and don't contain POST data.

So make your applications log properly and replay the logs on your new setup. And make 200% sure that you don't send any data back and don't change live data :)

answered Apr 06 '13 at 12:22

Dennis Kaarsemaker

19,277
2
44
70

Thanks for your time and suggestion. Do you also know some tools to do some proper logging? (Including post data etc ? ) – Andrea Baccega Apr 07 '13 at 21:45
This answer would really be a lot better if you supplied some tips on logging software. I have never even heard of software that is able to export logs to a format that is re-playable. – oligofren Oct 07 '14 at 10:33
This article goes into how one can do this with simple GETs: https://www.igvita.com/2008/09/30/load-testing-with-log-replay/ – oligofren Oct 07 '14 at 10:39

How to test test environment with real data from production

2 Answers2