0

I have a search results page with different types of sorting available for user to choose via drop down control. I need to test variation of one of the sorting options, let's say Relevance (variation B has different algorithm of relevance definition behind it).

Trick is that sorting option selected by the user is remembered and next time user visits the search results page - his default sorting option is equal to his last chosen option. E.g. lets say a user some time ago selected as sorting option Price (from lowest to highest), meaning he will never get results sorted by relevance anymore until he will explicitly choose 'Relevance' sorting option.

Question is - if it is still OK to test variations of the page in total as it is, or I need to setup/limit A/B test only for the case where users have selected 'Relevance' as a sorting option comparing in this way only user conversion via variations of this particular option?

Thank you for any hints.

O.C.
  • 1
  • 3

1 Answers1

0

I realize that this question is old but just in case someone comes along with a similar question here's an answer

You can use conditional/manual activation in order to ensure the users who are bucketed actually saw the experiment. Conditional activation is only available to enterprise accounts but if you have that using something like $('#my-dropdown').val() === 'relevance'. If not you could use manual activation and add javascript to the page that checks what the currently selected option is and fires this: 

window.optimzely = window.optimizely || [];
window.optimizely.push(['activate', experimentID]);

Here's the Optimizely article on activation modes: https://help.optimizely.com/hc/en-us/articles/200040225

from a statistical perspective though you should still have valid results if you bucket all visitors to the page in the experiment, but you would have to run more traffic through the experiment to get significance on the same actual effect. How much more traffic would depend on the % of total traffic that sees the chosen version of the page.

Nate
  • 319
  • 2
  • 8