4

I've only got a small amount of experience with A/B testing; but from what I've seen it seems like the standard approach to do an A/B test is to introduce some conditional logic in an application's code. This can be tricky to implement properly (depending on the complexity of the test) and requires extra work both for setup and cleanup.

It got me wondering: are there any frameworks or approaches to A/B testing that simplify matters using, e.g., Git branches? I'm envisioning something at the load balancer level, which directs half of traffic to a server where "master" or "default" has been deployed, and the other half to a server with "experiment" deployed. This way the code itself could always be completely agnostic of any A/B tests going on; and presumably the act of choosing either A or B for full deployment would be a simple flip of a switch.

I'm sure this would not be trivial to set up properly. But still I wonder if it's possible, and if in fact it's already been done.

Dan Tao
  • 125,917
  • 54
  • 300
  • 447
  • +1 for "the code itself could always be completely agnostic of any A/B tests going on". That's exactly what I'd like to achieve, but googling for a solution just led me to this unanswered question... I don't suppose you ever found anything like this? – Ciaran Phillips Dec 08 '13 at 15:15
  • @CiaranPhillips: Sadly, no, I haven't found anything like it. That doesn't mean it doesn't exist, of course. I may just not have looked hard enough. But I'm at least confident that there isn't anything like this that's very widely used. – Dan Tao Dec 09 '13 at 15:16
  • I guess I'll just have to keep an eye out (wouldn't exactly feel confident about trying to build it myself right now!) Thanks for the response! :) – Ciaran Phillips Dec 16 '13 at 12:37

1 Answers1

0

it's relatively easy to build and definitely doable. you need to implement a deployment system where you deploy all branches starting with "ab_* for an example into different folders on your servers. then at some point in your code you can decide which folder should be included in the actual user session based on your actual test. it's not really a 'framework', it's a simple architecture design pattern you have to add to your own system, i was doing the same in production before.

Tamas Kalman
  • 1,925
  • 1
  • 19
  • 24