3

I want to make an algorithm which will enable the conduct of A/B testing over a variable number of subjects with a variable number of properties per subject.

For example I have 1000 people with the following properties: they come from two departments, some are managers, some are women etc. these properties may increase/decrease according to the situation.

I want to make an algorithm which will split the population in two with the best representation possible in both A and B of all the properties. So i want two groups of 500 people with equal number of both departments in both, equal number of managers and equal number of women. More specifically, I would like to maintain the ratio of each property in both A and B. So if we have 10% managers I want 10% of sample A and Sample B to be managers.

Any pointers on where to begin? I am pretty sure that such an algorithm exists. I have a gut feeling that this may be unsolvable in some cases as there may be an odd number of managers AND women AND Dept. 1.

Almostamaster
  • 29
  • 2

2 Answers2

0

Make a list of permutations of all a/b variables. 

 Dept1,Manager,Male
 Dept1,Manager,Female
 Dept1,Junior,Male
 ...
 Dept2,Junior,Female

Go through all the people and assign them to their respective permutation. Maybe randomise the order of the people first just to be sure there is no bias in the order they are added to each permutation.

 Dept1,Manager,Male-> Person1, Person16, Person143...
 Dept1,Manager,Female-> Person7, Person10, Person83...

Have a second process that goes through each permutation and assigns half the people to one test group and half to the other. You will need to account for odd numbers of people in the group, but that should be fairly easy to factor in, obviously a larger sample size will reduce the impact of this odd number on the final results.

Steve E.
  • 9,003
  • 6
  • 39
  • 57
0

The algorithm for splitting the groups is simple - take each group of people who have all dimensions in common and assign half to the treatment and half to the control. You don't need to worry about odd numbers of people, whatever statistical test you are using will account for that. If some dimension is so skewed (i.e., there are only 2 females in your entire sample), it may be wise throw the dimension out.

Simple A/B tests usually use a t-test or g-test, but in your case, you'd be better of using an ANOVA to determine the significance of the treatment on each of the individual dimensions.

dfb
  • 13,133
  • 2
  • 31
  • 52