4

I can not find it anywhere, so I hope someone already stumbled upon this and can give me an answer.

I have been playing for a long time with Firebase remote config. In some occasions I have set parameters to be applied with different values to certain % of my user base.

Recently I started being interested in proper A/B testing and saw that Firebase has a feature for this (in beta right now). In the description of the A/B testing feature they state that one of the use cases is by setting parameters through remote config to alter the app's behaviour (makes sense, this is what I did until now).

My question though is whether A/B testing feature is doing anything different (or in addition to) to what remote config is doing. In particular, I am very interested in knowing whether remote config ensures me that when users open the app multiple times they will always be getting the same remote config values (maybe through tracking their device/user ID?) or is this only achieves if I use A/B config?

My experiments on this are not conclusive. It seems though that remote config does not ensure the same values over time.

maxmitz
  • 258
  • 1
  • 4
  • 17
XAnguera
  • 1,157
  • 1
  • 11
  • 25

1 Answers1

2

Firebase A/B Testing builds on top of Firebase Remote Config, and Google Analytics for Firebase (and some other Firebase products) to offer full A/B testing capabilities.

Once a user is part of a certain group in an A/B Testing experiment, they will remain part of that group for the duration of the experiment.

When you use Firebase Remote Config without A/B Testing, you are in control of the groups completely yourself. So in that case you determine what value(s) the user gets.

Frank van Puffelen
  • 565,676
  • 79
  • 828
  • 807
  • Thank you @Frank, just to confirm: by your response I understand that if I define a "user in random percentile" condition in remote config Firebase does not ensure that a given user will always get the same value every time he opens the app (and it fetches content from Firebase remote config) but instead the random condition will be played every time. On the other hand, A/B testing feature does add some layer on top of remote config that keeps giving the same group of users the same values. Is this right? – XAnguera Sep 17 '18 at 09:44
  • I don't know about Remote Config standalone, but I'd expect it to work the same as in combination with A/B Testing. – Frank van Puffelen Sep 17 '18 at 14:28
  • is there any documentation on this? I would like to confirm whether remote config ensures that my users will consistently get the same values, as A/B testing seems to do. Thanks. – XAnguera Sep 17 '18 at 14:35
  • I honestly don't know if there is documentation for this, which is why I started by describing the relationship between Remote Config and A/B Testing. Since they're the same, I'd expect them to behave the same. – Frank van Puffelen Sep 17 '18 at 15:08
  • 1
    Thanks, I was wondering whether there was any extra layer implemented in A/B testing feature to ensure users stay in the same test group. My tests using remote config seem to indicate that every fetch of data to Firebase returns a new set of parameters, which is not what I was expecting. I will do some more experiments and will report on my findings – XAnguera Sep 17 '18 at 17:07