1

We started using Optimizely and we're blown by the results so we started creating aggressive campaigns. But then we realized something was wrong and I would appreciate any help we can get.

We tested (for example) on the homepage instead of a blue CTA, we created a variation with a green CTA and we saw a 35% increase in conversions with the variation. So then we switched things and set the green CTA as the control and the blue CTA as the variation. If it was really true that the green CTA is better we should now be seeing a 35% decrease in conversions with the variation of the blue CTA. Instead, the variation came back with a higher conversion rate.

I can't pinpoint what we're doing wrong, but in general all the variations come in with a higher success rate than the control, but if we switch things around and set the variation as the control, the results don't switch.

Haim
  • 1,107
  • 2
  • 10
  • 19
  • Sounds like something is up. That shouldn't happen if everything else is controlled for. How many visitors / clicks are you looking at (maybe too small sample)? Did your results reach significance? Did any traffic patterns change (maybe different incoming traffic prefers different treatments - e.g., FB likes Blue for CTA but Google likes Green)? – TomFuertes Dec 08 '16 at 19:17
  • @TomFuertes this is exactly my problem, traffic sources are the same and the sample sizes are the same, but for some reason whatever we set as the variable wins – Haim Dec 08 '16 at 21:44

3 Answers3

0

How do you measure conversions? It's usually a good practice to measure in more ways than once - for example, for a CTA experiment I would measure clicks, landing on the page where the link redirects, valid form submissions etc. Expanding your view on the data and see how a change will affect other parts of the user experience is always a good idea.

Zsolt
  • 1
0

Try: On Results page apply different segments to identify improvement in each variation I recommend setting a 90% of statistical significance If you tested in 2 separate test, maybe you didnt let it the right time i usually prefer to have them by a week to avoid seasonality,

Julio Farfan
  • 1
0

Are you making these adjustments to an experiment that has already been running? You should keep in mind that users are bucketed once you've started an experiment so making adjustments like switching who belongs to a control or not, after or whilst an experiment has run will not be starting you fresh so to say. If you're making the adjustments you describe then you'll need to create a new experiment.

user6810289
  • 31
  • 3