0

Does anyone know why are users with older app version which doesn't have new experiment in config counted as part of experiment? This is scenario:

I have app version 1 with experiment A with activation event for new users new_install

Then I create app version 2 and new experiment B with same activation event for new users new_install . I start experiment on firebase console.

Now I am waiting for app to be approved by apple appstore or starting rollout on android. Even before any user gets app version 2 I already see many users showing in experiment B on firebase console.

I would expect that users with app version 1 will be not part of experiment because remote config doesn't have default value for experiment B in app version 1 but it seems that because users are still installing app version 1 and new_install event is fired, they are counted as part of experiment B event when it's not in app version 1.

Is it feature? Bug? I am supposed to create custom activation event for every experiment?

Martin Vandzura
  • 3,047
  • 1
  • 31
  • 63

1 Answers1

0

That's actually easy to prevent - and good practice for A/B testing in general - just target users with at least given app version. Usually it will be version since you have remote config properly set up.

I would not rely on Firebase including or excluding users if they don't have remote config parameter. Today it might work differently than how it's going to work in a month. If there is no guarantee, just make sure you are targeting users who are properly set up.

michalbrz
  • 3,354
  • 1
  • 30
  • 41