13

I was torn between posting this on Cross Validated, User Experience, or here, but here goes:

Many have heard about how Google, not wanting to rely on the hunches of designers, tested 41 shades of blue for their links and found that the best performing shade led to $US200M in revenue a year.

Google's commitment to data-driven decisions is well reported, and the company has been ridiculed for the "50 shades of blue" episode, when then Google executive Marissa Meyer led a project testing the impact of using different coloured links in ads.

But a new insight proves that the company significantly benefitted from the experiment, to the tune of $200m.

I'm not skeptical so much about whether the tests actually happened, or even if Google projected that the change would generate that much extra revenue, but the counter-intuitiveness of the result leads me to wonder if something was wrong with their analysis. The experiment seems ripe for this kind of statistical error.

Are the data and analytical processes used in this study available anywhere online?

Community
  • 1
panofsteel
  • 231
  • 1
  • 3
  • 2
    So what is your question, exactly? Did Google do 41 tests? Did they achieve a sufficient degree of statistical accuracy with those tests? Did those tests lead to $200m increased revenue? – Flimzy Dec 23 '15 at 13:45
  • Neither "is this analysis wrong?" nor "Is this study available online?" are on topic at this site. – DJClayworth Dec 23 '15 at 16:46
  • 2
    @DJClayworth Some questions on this site are skeptical that a study took place. Isn't it on topic to be skeptical of the study itself? – panofsteel Dec 23 '15 at 23:17
  • 1
    @Flimzy "Did those tests lead to $200m increased revenue" gets to the heart of my question quite well. – panofsteel Dec 23 '15 at 23:23
  • 2
    As far as I can tell this isn't a testable claim, since it's based in Google's actual CPM as reported by one of their UK directors, which is anecdotal evidence which we can't easily repeat. – Avery Dec 24 '15 at 00:32
  • @Avery What does CPM stand for? – panofsteel Dec 24 '15 at 20:07
  • Cost per thousand impressions -- the stat that would get a bump from increased sales. – Avery Dec 24 '15 at 23:34
  • Two obvious questions: 1) how do they *know* that the shade of blue increased the number of clicks after their experiment was over? They don't. They just have some figures on clicks and they extrapolated from that. 2) how much of a percentage is $200 million? It's probably only one per cent of their ad revenue. That's part of the point of his story. When you have huge revenues, a 1% increase is a lot of money. – AmbroseChapel Dec 29 '15 at 23:17

0 Answers0