24

enter image description here

Comedian Stephen Wright once joked that 42.7% of all statistics are made up on the spot.

Recently, Arizona Senator and notably stubborn Jon Kyl provided us with the most famous example of just such behavior.

He can be seen here, stating that well over 90% of what planned parenthood does is abortions, despite the fact that the real amount is 3%. Later, he attempted to claim that his words were "not intended to be a factual statement" once it became clear he had just made up the 90% figure.

enter image description here

In a perfect world, all claims would be appropriately and accurately referenced, but we all know this just simply isn't the case, especially when it comes to politicians, the media, and those pushing an agenda dishonestly.

Has there ever been any scientific examination into how often quoted statistics are just made up? Do we honestly have any way to figure this out?

I assume any study of this would have to be limited to verifiable instances, such as recorded speeches or publications, as there would be no way to evaluate the amount of times it's done in personal conversations.

Sklivvz
  • 78,578
  • 29
  • 321
  • 428
Monkey Tuesday
  • 24,456
  • 19
  • 109
  • 136
  • While as a resident of Arizona I strongly agree with your characterization of Senator Kyl as an "ass-hat", [it's inappropriate to call him out as such on Skeptics](http://meta.skeptics.stackexchange.com/questions/13/when-is-it-appropriate-to-be-a-dick-on-skeptics-stackexchange). – Patches May 22 '11 at 22:26
  • 10
    Not intended to be a factual statement – Monkey Tuesday May 22 '11 at 22:31
  • 12
    `Not intended to be a factual statement` is right up there with the Texas Board of Education textbooks! The Chinese, Indians, and every other country in the world must be chortling and just saying, "Excellent!" as they watch the US slide into Idiocracy. – Larian LeQuella May 23 '11 at 01:33
  • 2
    I agree with @Patches. You are adequately discrediting Jon Kyl with a simple reference to his words and behaviour. There's no need to include emotive ad hominem terms on top of that. (This is not a defence of the man; I am not a US voter and have not formed an opinion of him.) – Oddthinking May 23 '11 at 04:16
  • 3
    @oddthinking An ad hominem attack implies that it is either Jon Kyl's claims or credibility in question. I assure you, they are not, he is referenced only because his behavior is a famous and recent example which illustrates the larger question about the prevalency of fabricted statistics. Was it an insulting thing to say? Absolutely. Was it necessary? Probably not. But ad hominem attack? Nope. – Monkey Tuesday May 23 '11 at 04:38
  • 3
    I had a disagreement with a gentleman in the comments section of my hometown newspaper. He, a staunch Catholic, included emergency contraception pills (Plan B) in his figures for 'abortions performed by PP' - because he considered each instance of their use an abortion. However, he did not include the figures of women who utilize Mirena or the 'regular' pill. I called him out for intellectual dishonesty; because if he was going to include Plan B, he needed to also claim the Mirena IUD and 'pill' performed abortions as well, since they were the same chemical compound for the same purpose. – Darwy May 23 '11 at 07:30
  • 3
    But in response to your question, many statistics have a verifiable source: the ass of the person that came out with them. – Darwy May 23 '11 at 07:31
  • @Monkey Would... "Senator Kyl's statement was purposeful and done with malice aforethought*." ...be considered an ad hominem attack ? *A general evil and depraved state of mind in which the person is unconcerned for the lives of others. – Rusty May 23 '11 at 07:52
  • 2
    @Rusty is it intended as a factual statement? – Monkey Tuesday May 23 '11 at 08:01
  • @Darwy Intellectual dishonesty. Is that the same as lying ? – Rusty May 23 '11 at 08:01
  • @Monkey How could it not be a factual statement ? – Rusty May 23 '11 at 08:03
  • 1
    @Rusty It's possible. It would depend on what was able to be proven regarding his motives for the initial fabrication, but a statement of that sort could also be considered libelous in print, given the legal nature of the term "malice aforethought". Thankfully however, ass-hat is not an actionable term. – Monkey Tuesday May 23 '11 at 08:19
  • 2
    @Monkey Tuesday, I accept your characterisation: an unnecessary insult, but not ad hominem. I retract my statement that it was ad hominem, sorry. Now that we've agreed on that, I still think it should be removed! I don't think the fact that it is unactionable makes it appropriate. (I would call for the term to be removed because it hasn't a reference, but I fear you will provide one!) – Oddthinking May 23 '11 at 08:29
  • 5
    @Rusty intellectual dishonesty is essentially the advocation of a position known to be false. The other gentleman claimed that use of Plan B causes an abortion. This is false. Plan B is not an abortificant. But because he is Catholic the use of Plan B is, in his opinion, the same as having an abortion. However, the use of the ordinary birth control pill and the mirena iud are only 'problematic' for him, and not considered abortion to him despite the fact it's the same chemical compound. – Darwy May 23 '11 at 08:34
  • @oddthinking I can understand it's the offensive language that bothers you, as you made no mention of the fact that he is also called one of gaddafi's sexy female ninja guards. However, Jon Kyl is widely known mainly _because of_ the ridicule he's received (deservedly)for this particular incident, and is thus more widely known as an ass-hat than as a United States Senator. I agree the term is offensive, moreso to some than others, but I feel it is indeed an adequate description in this case. – Monkey Tuesday May 23 '11 at 08:47
  • @oddthinking I know you were being sarcastic about the references,but.... [here's one](http://allthatnatters.com/2009/05/24/asshat-of-the-day-sen-jon-kyl-r-az/) [and another](http://theleftneckchick.com/index.php?option=com_content&view=article&id=149:fith-the-ass-hat-sessions&catid=35:articles&Itemid=54) [here's one with video](http://lies.videosift.com/video/Blatant-Lie-on-the-floor-of-the-Senate-Asshat) – Monkey Tuesday May 23 '11 at 09:06
  • 2
    It isn't that I am offended by the language. Fuck, no. It is that it is that it is mocking behaviour, which we have a low tolerance for on this site. (Go ahead and put it on your blog with my blessing.) The issue with the mocking is that it may scare off the exact audience of people we want to read this stuff - e.g. those who support such politicians, or those who believe it is acceptable behaviour to make up statistics without references. (I got the ninja guard reference, but you didn't use it in the question, and I didn't quite follow what its absence proved.) – Oddthinking May 23 '11 at 09:07
  • Through your references, you have now proved he is indeed noted as an asshat. Doesn't change my position. (If you are trying to be Judd Nelson from From The Hip, you have succeeded. We are now more focussed on the word "asshat" than the original question.) – Oddthinking May 23 '11 at 09:13
  • @Monkey Kyl has "privilege". He would be required to prove the statement was made with reckless disregard for the truth. That dog won't hunt. Personally I just use profanity so insane that the entire concept of decorum is briefly excluded from the universe. Thanks for your help. – Rusty May 23 '11 at 09:18
  • @odd I agree,and that does distress me.I really didn't expect what I figured was kind of a throw-away joke to require this much defense.Still, I feel the term adequately conveys the fool he's known to have made of himself in front of the whole world.Also, the mere mention of his name (and he is the picture-perfect example of fabricated statistical evidence) would probably have caused someone to comment calling him something or other and we'd probably be right here anyway, only in that case it might actually be worse due to his association with the abortion debate. – Monkey Tuesday May 23 '11 at 09:37
  • Please keep ass hat in here, it's adding to my lexicon. – Hairy May 23 '11 at 09:57
  • 1
    Removed the offensive term. It was *completely out of place*. Please keep further comments on topic. – Sklivvz May 23 '11 at 12:27
  • 5
    12 out of 8 people don't understand statistics ;) – Oliver_C May 23 '11 at 13:04
  • I dont know, but to help you out, you certainly have to include politicians in your equation –  May 23 '11 at 20:32
  • Yes. But it could be a lot of work. First you would have to limit the scope. I presume what's said in pubs or taxi's doesn't interest you (or maybe it does). Once scope is decided use a suitable statistical model and random sampling to determine with 9x% certainty that y% of statistics are made up! Your scope can only contain cases for which you can prove the statistic was made up, or at least have a high probability of certainty. – John McNamara Jun 07 '11 at 14:19

1 Answers1

12

Noted skeptic, Ben Goldacre, took part in a published study recently. It doesn't directly answer your stated question, but it does address some of the concern behind it.

Benjamin E.J. Cooper, William E. Lee, Ben M. Goldacre, Thomas A.B. Sanders, "The quality of the evidence for dietary advice given in UK national newspapers", Public Understanding of Science, April 2001, doi: 10.1177/0963662511401782

Goldacre described the research in his Guardian column.

They didn't look at all statistics quoted. They restricted their scope to health claims (note: not just statistics) made by newspapers. They diligently tried to find evidence to support them.

(Can someone please invite those researchers to come to this site next, and use those same skills?)

Goldacre summarised in his column:

Here's what we found: 111 health claims were made in UK newspapers over one week. The vast majority of these claims were only supported by evidence categorised as "insufficient" (62% under the WCRF system). After that, 10% were "possible", 12% were "probable", and in only 15% was the evidence "convincing". Fewer low quality claims ("insufficient" or "possible") were made in broadsheet newspapers, but there wasn't much in it.
Oddthinking
  • 140,378
  • 46
  • 548
  • 638