16

We obtain a great deal of data about trends in sensitive and personal topics through anonymous surveys.

For example, in a science writing class I'm taking, the professor showed us results from an anonymous survey taken by undergraduate students. The survey showed the prevalence of various forms of academic dishonesty. (I didn't copy down the reference, but I can get it if it's important to be specific here.)

The assumption here is that anonymity makes participants more likely to respond honestly. This sounds plausible - it's the old "strangers on a train sharing secrets" story. But is it actually a valid assumption?

  • How honest are people's responses to anonymous surveys?
  • What sorts of studies are done to examine the honesty of these surveys?
Oddthinking
  • 140,378
  • 46
  • 548
  • 638
Mark Eichenlaub
  • 482
  • 5
  • 14
  • It really depends on the survey: in some cases it can be more accurate than a non-anonymous, and vice-versa. Can you be more specific about the survey? – Sklivvz May 06 '11 at 10:49
  • I updated the question to change "accuracy" to "honesty" since I think that might have caused some confusion about my question. Specifically, I wasn't trying to compare anonymous and non-anonymous surveys. – Mark Eichenlaub May 06 '11 at 10:56
  • 2
    How would you find that out? With a (anonymous) survey? ... [Survey Honesty](http://www.marketresearchworld.net/index.php?option=com_content&task=view&id=161&Itemid=74) – Oliver_C May 06 '11 at 11:18
  • @Oliver_C One example: In some cases, it might be possible to infer the percentage of people who ought to respond "yes" to a given question, then compare that to the percentage who actually respond "yes". For example, a survey could have a question asking if the responder has ever had a certain STD. Then, by examining medical records, it might be inferred that a certain percentage of people (within a range) have actually had the STD. Then the percentage inferred from medical records and the percentage from the survey could be compared. – Mark Eichenlaub May 06 '11 at 11:29
  • My question is about whether that or other techniques to assess honesty have been implemented, and what the conclusion are. – Mark Eichenlaub May 06 '11 at 11:30
  • The answers to this question may be very misleading: people *aren’t* honest on anonymous surveys. But this doesn’t matter since there are statistical methods that reliably remove the bias incurred by dishonesty. – Konrad Rudolph May 06 '11 at 11:36
  • @Konrad I would like to know more about your comment. Do you know some references or do you have the time to write up a full answer? – Mark Eichenlaub May 06 '11 at 11:48
  • @Mark Unfortunately I don’t, at the moment. That’s why I wrote a cautionary comment rather than a proper answer. – Konrad Rudolph May 06 '11 at 11:52
  • What's the claim being questioned here? Right now, this question looks closer to something that could be asked on stats.SE; if you have a specific claim regarding surveys or statistical methods, ask about that; this isn't an open-ended "ask anything I'm curious about" site. – Shog9 May 07 '11 at 17:31
  • @Shog9 How about the claim, "people tell the truth about themselves, their preferences, and their actions when they take anonymous polls." That is a specific claim. It is also a "practical, answerable questions based on actual problems that you face." (from the FAQ) – Mark Eichenlaub May 08 '11 at 21:29
  • @Mark: I've edited your question to reflect this - please review. – Shog9 May 09 '11 at 23:04
  • @Shog9 Looks good. Thanks for your edits. – Mark Eichenlaub May 10 '11 at 00:17
  • If it's reasonable to assume that there would be negative consequences if a student is caught cheating, it is reasonable to conclude that if you did a non-anonymous version of that same survey, no students would admit to cheating. So in that sense, you know that the students are more honest on the anonymous survey. – Brian Schroth May 10 '11 at 14:42
  • However, to take the opposing view, it's also possible that people will lie on a survey in a real-world equivalent to trolling- "For the lulz". They may get a kick out of screwing up the study's data. The effect of anonymous results on this form of dishonesty would be to *increase* it. Citation: John Gabriel's Greater Internet Fuckwad Theory. http://www.penny-arcade.com/comic/2004/3/19/ – Brian Schroth May 10 '11 at 14:46
  • @Mark:What would be a non-anonymous survey? Would confidential surveys count as non-anonymous? – apoorv020 May 12 '11 at 11:07
  • @apoorv020 Let's say that a survey is anonymous when it doesn't have your name or other identifying information on it and the survey taker believes their answers cannot be associated with them. – Mark Eichenlaub May 12 '11 at 11:14
  • I'm not sure about the answer but, I do know that people will be more likely to stick to thier answer if it is not anonymous. My first thoughts are, how can you know what the answerer was thinking when they answered the question? – Jodrell May 12 '11 at 12:00

1 Answers1

7

A majority of the literature on the net seems to support that non-anonymous and anonymous-surveys give very similar results.

The paper "A comparison of confidential versus anonymous survey procedures: Effects on reporting of drug use and related attitudes and beliefs in a national study of students." studies anonymous vs. confidential surveys in a specific case (drug usage reported in 8th and 10th graders), and concluded that there was a very modest or possibly no effect on the data.

From the abstract:

This study presents a comparison of reporting of drug use and related attitudes and beliefs by national samples of 8th and 10th grade students under two different modes of administration conditions: confidential and anonymous. Data come from the Monitoring the Future project for the year 1998. The results show that there were clearly no differences between the conditions in 10th graders' reports of drug use and related attitudes and beliefs. With 8th graders, the results show, at most, only a very modest mode of administration effect and quite possibly no effect at all.

The same conclusion is drawn in more countries in the paper "Adolescent Substance Abuse in Mexico, Puerto Rico and the United States: Effect of Anonymous versus Confidential Survey Formats".

Similarly another study "The Effect of Anonymous Vs. Nonanonymous Rating Conditions on Patient Satisfaction and Motivation Ratings in a Population of Substance Abuse Patients" concluded that

Anonymity had either no effect on ratings or accounted for <1% of the variance.

In a study about collecting data regarding bullying, "The Efficacy of Non-Anonymous Measures of Bullying" concluded that

The findings supported the hypotheses that the respondents did not differ in their report of the incidence of either bullying or victimization, regardless of whether they were required to identify themselves by writing down their names on the questionnaire forms.

Another study, "Differences between ‘talking about' and ‘admitting' sensitive behaviour in anonymous and non-anonymous web-based interviews" which (among other things) compared difference between two questionnaire modes- anonymous web-based forms or dektop-based video interviews concluded that:

Nevertheless, the expected differences between the interview modes were not observed.

On the other hand:

A study called "High risk behaviour and fertility desires among heterosexual HIV-positive patients with a serodiscordant partner – two challenging issues"(PDF) says that

Non-anonymous data collection on condom use may underestimate high risk behaviour

However, there are several caveats to these papers, some issues which may need exploring:

  1. The surveys sometimes seems to be carried out through a trusted authority.
  2. The surveys seem to target some specific portion of the population. The validity for general polls like opinion polls etc. may be different.
  3. Non-anonymous may mean several things, perhaps each with it's own effect on results.
apoorv020
  • 3,266
  • 2
  • 25
  • 37