8

Blind listening tests are strongly critiqued by Robert Harley in this editorial, published in The Absolute Sound. Harley seems to think the tests prove nothing.

Every few years, the results of some blind listening test are announced that purportedly “prove” an absurd conclusion. These tests, ironically, say more about the flaws inherent in blind listening tests than about the phenomena in question.

The article goes on to describe research done by the Swedish Radio. Apparently an artifact inherent to a codec that hadn't been noticed in 20,000 blind listening evaluations, was noticed almost instantly in non-blind circumstances. This gives Harley grounds to claim the following:

The answer is that blind listening tests fundamentally distort the listening process and are worthless in determining the audibility of a certain phenomenon.

Is this true? Do blind listening tests significantly inhibit the listeners' ability to notice certain audible phenomena?

This article was mentioned in a comment thread by Christopher Galpin. Oddthinking suggested making a question about it. Since nothing has happened in almost a year and the question is interesting, I'm bringing it up.

Community
  • 1
StackExchange saddens dancek
  • 3,829
  • 1
  • 25
  • 37
  • Is this a fun question? Blind tests are made to exclude bias. How would you compare a blind test to a non blind test, to prove an advantage of the non blind test, to show it is superior, if you can't prove it? Of course you could test audio cables, by measuring the output. Such an instrument for measuring the output of an audio equipment exists: It is called `microphone`. If you can't measure a difference between cheap cable and expensive cables, I guess there is another excuse: Only humans can detect the difference - but only if not tested blind. Poor marketing. – user unknown Mar 04 '12 at 03:17
  • 20,000 tester can't find an artifact without being told it is there and that "proves" that something is wrong with *not telling them*? Really? Should we quit doing blinded drug effectiveness studies too, because people won't know they're taking the "good stuff" if we don't tell them? *Come on!* – dmckee --- ex-moderator kitten Mar 04 '12 at 04:14
  • 1
    The claim feels ridiculous to me, too. If it were true, similar questions about double blind methodology in general might also be relevant. I can't think of a plausible explanation for Harley's claim (but I don't know that anyone has debunked it either). Still, I tried to frame the question in an "objective" way as showing bias in the question doesn't help anyone. – StackExchange saddens dancek Mar 04 '12 at 09:12
  • 2
    I'm having trouble seeing how this question could possibly be answered, but I am keen to see someone smarter than me try. After all, I *believe* blinded-tests are better than non-blinded, and as a skeptic I should have evidence for my beliefs... – Oddthinking Mar 04 '12 at 14:34
  • 1
    @Oddthinking I wonder whether a sophisticated range of tests that involve a mix including deliberate *misleading* in some non-blind experiments would tease out the issues. Maybe we need a statistician. – matt_black Mar 04 '12 at 20:28
  • 2
    @matt_black: deliberately misleading? You mean like a placebo? Isn't that what a good blind test does? (Also, not sure that a statistician is the right specialist here.) – Oddthinking Mar 05 '12 at 00:19
  • If I understand the claim right it's that, an expert who first hears the unblinded versions can find features in it that allows him afterwards to distinguish the versions in the blinded test. In this case the expert is also supposedly able to teach others to distinguish the two versions in the blinded test. | If you look at drug tests, no drug gets developed in a completely blinded way. Before you do a blind test with a new drug you first try to understand what the drug might do. You tell the people in your drug trial about the possible side effects of the drug. – Christian Aug 15 '12 at 13:09

0 Answers0