0

WHAT ARE WE TRYING TO ACHIEVE

We are using Google Analytics Cohort Analysis to AB test a feature. We generate a parameter on the client side and set 50% of users to "Yes" and 50% to "No". This parameter is then sent to Google Analytics, and used to create two custom segments:

  • Feature Enabled: Feature "Yes"
  • Feature Disabled: Feature "No"

We then wait for a week and generate two retention cohort analysis, one per segment, to determine which version of the feature (A or B) is better.

THE ISSUE

We noticed that Google Analytics returns significantly different version of data, apparently without reason. More specifically, it seems to have TWO different version of data, different enough to lead us to completely opposite conclusions, and to show one or the other depending on other random factors, such as whether I add a third segment to the analysis.

"Version A is clearly better. <\page refresh> Oh no wait, now version B is better!?"

Anybody else experiencing the same issue? Is this a bug on Google Analytics?

Francesco Bovoli
  • 326
  • 2
  • 9
  • I'm voting to close this question as off-topic because its not programming related may be better suited for https://webapps.stackexchange.com/ – Linda Lawton - DaImTo Jul 18 '18 at 11:53
  • Thanks for the feedback. I opened the question here following the link found on the official Google Analytics help page, if this is not the right place perhaps they should update it: https://developers.google.com/analytics/help/ – Francesco Bovoli Jul 18 '18 at 13:16
  • I understand that but Google does not own or control stack overflow as per stack overflow your question is off topic because its not programming related. I have contacted Google about having the link changed its in their backlog and i am sure they will update it someday. In the mean time i am hear to direct you to the proper place for your question so that you can get the assistance you need. – Linda Lawton - DaImTo Jul 18 '18 at 13:33
  • 1
    I appreciate that and will open the question there. I just reported why I opened it here so you could fix the problem at its root. – Francesco Bovoli Jul 18 '18 at 14:39
  • I sent another mail off to the technical writers linking your question. Feel free to submit feedback on that page. – Linda Lawton - DaImTo Jul 18 '18 at 14:40
  • https://issuetracker.google.com/issues/111565525 – Linda Lawton - DaImTo Jul 18 '18 at 14:51

0 Answers0