77

In this article in the Guardian, Rachel Roberts claims that:

"Five major systematic reviews have also been carried out to analyse the balance of evidence from RCTs [randomized controlled trials] of homeopathy – four were positive (Kleijnen, J, et al; Linde, K, et al; Linde, K, et al; Cucherat, M, et al) and one was negative (Shang, A et al)."

Is this claim true? And if so, is it representative?

I am aware of the risk of cherry-picking trials to support one's hypothesis, but I would expect that systematic reviews would remove this selection bias.

To be clear, I'm not interested in whether there is a scientific basis for homeopathy (since the author of the article does not argue that there is one, apart from anything else).

Glorfindel
  • 1,452
  • 1
  • 17
  • 28
mdarwin
  • 887
  • 1
  • 6
  • 7
  • 43
    Reading the cited studies they basically say that he trials were crap and don't prove anything beyond that there was some improvement, which may be due to placebo or even just having a doctor listen to the patients. So they don't really support the claim that homeopathy is effective. – dont_shog_me_bro Nov 30 '17 at 11:26
  • 42
    This is a very common tactic: linking to a reputable journal article which happens to be about that topic, claiming that it supports one's views, even if it doesn't, and hoping most readers don't actually read those papers and are satisfied instead with a vague "scientists proved it!" feeling. One of the greatest examples was a Nature article proving that vitamin C slightly alleviated one of the many harmful side-effects of chemotherapy used in cancer treatment, and this article was picked up by the media as "What they don't want you to know: vitamin C instantly cures all forms of cancer!" – vsz Nov 30 '17 at 13:47
  • @Philipp definitely a duplicate. Moreover my answer to the other question has more recent results than the ones referred to in the question. – matt_black Nov 30 '17 at 16:51
  • 37
    Only four in five million reviews were favourable but after dilution it became four our of five. – TheMathemagician Dec 01 '17 at 13:03
  • 4
    Homeopathy is IMHO one of the best proof that to some extent people who see plots everywhere may have a point: It has absolutely no logic to supports itself on (unless you believe in "water memory") but still manages to be in medicines shop and have doctors specialized in it... – Caterpillaraoz Dec 01 '17 at 15:03
  • @TheMathemagician you are a wonderful human being. – Tobia Tesan Dec 05 '17 at 12:44

4 Answers4

135

This statement is only true if you heavily distort the meaning of the word "positive". The following are quotes from the abstract of all four meta-analyses the article claimed are positive:

At the moment the evidence of clinical trials is positive but not sufficient to draw definitive conclusions because most trials are of low methodological quality and because of the unknown role of publication bias. This indicates that there is a legitimate case for further evaluation of homoeopathy, but only by means of well performed trials.

 

We conclude that in the study set investigated, there was clear evidence that studies with better methodological quality tended to yield less positive results.

 

There is some evidence that homeopathic treatments are more effective than placebo; however, the strength of this evidence is low because of the low methodological quality of the trials. Studies of high methodological quality were more likely to be negative than the lower quality studies. Further high quality studies are needed to confirm these results.

 

the results of our meta-analysis are not compatible with the hypothesis that the clinical effects of homeopathy are completely due to placebo. However, we found insufficient evidence from these studies that homeopathy is clearly efficacious for any single clinical condition. Further research on homeopathy is warranted provided it is rigorous and systematic.

One of these didn't actually address the question whether homeopathy works or not, they examined the influence of study design and came to the conclusion that better studies tend to show more negative results on homeopathy. All other three essentially said that their results are not conclusive and that more research is necessary.

They all state that the quality of the studies they're based on is often low, and that this affects their conclusion. It's very misleading at least to claim that these meta-analyses support homeopathy.

Mad Scientist
  • 43,643
  • 20
  • 173
  • 192
  • 74
    _"At the moment the evidence of clinical trials is positive"_ I read that as _"there is evidence that trials have taken place"_ :D – Baldrickk Nov 30 '17 at 15:26
  • 28
    Indeed... "evidence *of* clinical trials" and "evidence *from* clinical trials" are two very different statements. – T.J.L. Nov 30 '17 at 15:44
  • 16
    @Baldrickk No, it's clearly intended to say "the clinical trials have evidence which is positive". But as we'd all expect, it goes on to say (very politely!) that all the trials showing a positive outcome were garbage, and properly-run trials tend towards only placebo effect. And as good scientists, the authors correctly say that they don't have enough sigma to firmly establish it as only placebo effect. – Graham Nov 30 '17 at 17:36
  • 1
    @Baldrickk They're not really sure, though, as they reserve to change their opinion on whether clinical trials have taken place. It just currently looks like it, according to them. – UTF-8 Nov 30 '17 at 18:39
  • 17
    In medical jargon, as seen in medical journals, a "positive outcome" is one where the patient didn't get _worse_. That's it. I learned this to my dismay when reading articles before my wife underwent a certain eye surgery. – davidbak Nov 30 '17 at 19:33
  • 22
    It's very hard to get a mediocre study that shows that homeopathy doesn't work published as high-quality journals don't publish mediocre studies and low-quality journals like positive results. There's not a lot of demand for high-quality studies of homeopathy because it's, bluntly, silly. As a result, the majority of published studies will show that homeopathy works. – David Schwartz Nov 30 '17 at 23:25
  • 3
    It would make this answer more complete were it to include a quote from the abstract of the *negative* meta-analysis as well. Strictly speaking, you only analyzed the truth of four out of the five claims made in the cited article (though I'm certainly not doubting the fifth claim). – Wildcard Dec 01 '17 at 00:32
  • 3
    The only conclusion I could read from this is that; As the quality of the trial goes up, the evidence for homeopathy goes down. – Oscar Bravo Dec 01 '17 at 07:16
  • @Graham Oh I know what was written and the intention behind it. It was just one of those times your brain fills in the gaps when scanning text, generating a phrase that is almost, but not quite what was written. – Baldrickk Dec 01 '17 at 08:56
  • 1
    @Baldrickk Ah, one of those. I misread "editable text" as "edible text" yesterday. The cure was a very-much-non-homeopathic solution of natural plant extract containing caffeine... – Graham Dec 01 '17 at 09:19
  • What they are saying is that there is a positive outcome being detected but it's so small, it's hard to tell if it's real or just a statistical error due to the way the study is designed conducted. The fact that the positive outcome gets lower for better designed studies suggests that it's not real but just a noise in the data. – ventsyv Dec 01 '17 at 14:24
  • @ventsyv Error; you added statistical error. An error due to poor study design is not always a statistical error. For example, imagine if we did a study on if kids who eat chocolate get taller. We then offer kids of tall parents chocolate, and not kids of short parents. Then we discover that chocolate correlates with kids growing taller. There is an error here, but it isn't statistical. – Yakk Dec 01 '17 at 18:34
26

The claim is loosely worded, but a reasonable interpretation is that there are only five systematic reviews (otherwise, the whole point of avoiding cherry-picking a study is negated). This meta-meta review found eleven independent reviews, and found "Collectively they implied that the overall positive result of this meta-analysis is not supported by a critical analysis of the data. " https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1874503/

And another article looked at 57, and found "Though that body was mixed in size and quality, no clear signal of effectiveness emerged from the higher quality studies." http://blogs.bmj.com/bmj/2016/02/16/paul-glasziou-still-no-evidence-for-homeopathy/

Also, the JREF prize can be considered a review, and no one has claimed it.

Finally, there is an idea that we should approach claims with an "open mind", and entertain all hypotheses, but there is a point at which a hypothesis is so absurd that it cannot be meaningfully evaluated. Even if there were studies that showed that homeopathy works, that would do nothing to prove homeopathy; homeopathy basically requires that there be some supernatural being keeping track of what water has been in contact with what substance, and altering people's biology accordingly. And once you entertain that hypothesis, you might as well entertain the hypothesis that the studies don't actually support homeopathy, but aliens are mind-controlling scientists into thinking they do.

Acccumulation
  • 2,873
  • 1
  • 14
  • 19
  • 10
    @Accumulation "homeopathy basically requires that there be some supernatural being keeping track of what water has been in contact with what substance" - and to quote Tim Minchin, "it somehow forgets all the poo it's had in it" – Herr Pink Dec 01 '17 at 09:56
  • 6
    +1 I forgot who said it but the suggestion is that we should stop evaluating studies for the effectiveness of homeopathy altogether: doing so fundamentally misunderstands how empirical evidence works in science (which is, simplified, through a Bayesian process); since we *know* that homeopathy cannot work (it contradicts other, vastly better established scientific principles), no amount of limited studies will ever shift the overall evidence unless it *also* explains seemingly-contradictory principles of science. – Konrad Rudolph Dec 01 '17 at 15:39
  • @HerrPink - I had to join this stack just to upvote your comment & point you to my profile picture... which is that precise moment from the Tim Minchin video ;) – Tetsujin Dec 04 '17 at 17:56
  • @KonradRudolph Which is why some now-established theories got sidelined for so long, because continents can't drift and rocks can't fall from the sky to the Earth. Homeopathy is not the best case, but saying that something can't happen doesn't it contradicts current theories is a good way to avoid seeing evidence that would establish new theories. – prosfilaes Apr 20 '22 at 18:02
  • @prosfilaes Uh, no, those are *fundamentally* different. You seem to be equating lack of observation or evidence with existence of positive, compelling evidence. But these are not the same. We have evidence that directly contradicts the core claims of the mechanics behind homoeopathy. Before Wegener nobody had any actual evidence that continents couldn’t drift. They had just never observed it. Nothing in my previous comment precludes being open to new evidence and new hypotheses. In fact, it (briefly) describes the very process of incorporating new evidence into existing theories. – Konrad Rudolph Apr 21 '22 at 08:32
18

Long comment - Wrote this before I had commenting privileges.

Homeopathy most certainly does not work, except as a placebo.

You linked to an article in The Guardian, written by Rachel Roberts. Click her bio on The Guardian, and what'd you know, she's a 'professional homeopath':

Rachel Roberts is a professional homeopath, who qualified in 1997, and a registered member of the Society of Homeopaths. Holding a degree in biological sciences, Rachel lectures in both homeopathy and medical sciences at various colleges in the UK and overseas. She currently holds the posts of research consultant for the society of homeopaths and executive co-ordinator of the Homeopathy Research Institute

Her article is from 2010, and that's the only thing she wrote for The Guardian. Purely a fluff piece used to prop up her quack pseudoscience.

Nearly eight years later, nothing has changed. Homeopathy still doesn't work.

That's on top of the criticisms of the actual studies put forth by user Mad Scientist.

Rest assured, Homeopathy has not been vindicated in the slightest by this.

Edit: I see some criticism that I didn't address the truth or falsehood of the studies. That's because someone else already did that, and I referred to that person - Mad Scientist - for the answers to that. I addressed the potential conflict of interest that further casts shadow on her credibility in reporting on these studies.

Christian
  • 369
  • 1
  • 4
  • 27
    The question asked about the truth of a certain cited statistic; I don't think this answer really addresses that at all. – HDE 226868 Nov 30 '17 at 17:09
  • @HDE226868: If 4 of 5 peer reviewed journals actually published it, it would mean the peer review process failed. – Joshua Nov 30 '17 at 17:33
  • 13
    @Joshua That assertion requires you to already know the "correct" answer regarding homeopathy's effectiveness. Since this site is about evaluating evidence, dismissing evidence because you "know" it is wrong would be a bad idea. Similarly, this answer rejects the claim based on conflict of interest, but doesn't address the actual claim made; again, it is not based on actual evidence. That is not to say that the claims are true, just that these are poor methodologies for dismissing it. – IMSoP Nov 30 '17 at 18:20
  • @IMSoP: Homeopathy is the model bunk pseudo-scientific claim. It is uniquely fit for this purpose as it has no religion that requires it to work and no large-scale political movement that requires it to work either and its explanation of how it works is self-defeating. – Joshua Nov 30 '17 at 19:40
  • 11
    @Joshua You're preaching to the choir. I am in no way defending homeopathy. What I am defending is the very purpose of this site, which is to address claims from a skeptical and evidence-based approach. Your comment, and this answer, do not do that. You may be right, but you are not providing any useful evidence that you are right. – IMSoP Nov 30 '17 at 20:41
  • 3
    @Joshua Put it another way, should we discount the claims of a doctor because they practice modern, western medicine? No! Even if they have the same arguable conflict of interest: they are a professional in the field in question. That's all that's being pointed out. – Nate Diamond Nov 30 '17 at 20:55
  • 14
    Unfortunately, your argument merely relies on emotive words, e.g. "most certainly", "fluff piece", "quack", "rest assured", and "not... in the slightest". This site is about getting beyond that, and basing arguments on scientific investigation, quality journalism, etc. – Craig McQueen Nov 30 '17 at 23:25
  • The comments on this question should be pointed out to any user of this site who objects to their pet (unproven) theory being given short shrift. They are very good examples. – Wildcard Dec 01 '17 at 00:29
  • @IMSoP One can easily come to the conclusion that homeopathy is false based on the evidence, and then conclude that this claim is false. – Acccumulation Dec 01 '17 at 01:22
  • 2
    The fact that the author is a professional homeopath, and that's the only article she wrote for the paper, would make a good comment somewhere, maybe on the question. I have to agree this is not a good skeptics.SE answer. – Peter Cordes Dec 01 '17 at 06:34
  • 1
    I had not yet made an account on this site, which meant that when I did, I didn't have comment privileges. If I did, I would have commented instead. So I took care to mention in my post that @Mad Scientist posted a great answer to the actual question, and that my post was merely additional reasons to be wary of her reporting of these studies, since it's just a commercial for her own profession. I know it's a breach of etiquette to post a non-answer as an answer, but I felt it pertained to the answer strongly enough to justify it. – Christian Dec 01 '17 at 06:44
  • @Christian Just for the record, I didn't downvote this answer; I don't think it's all that bad. It was Joshua's comment (and Accumulation's comment above is similar) which seemed to be completely missing the point of this site. – IMSoP Dec 01 '17 at 09:54
  • 3
    Without disclosing my opinion on homeopathy: To increase the quality of the answer, I would suggest to at least remove the sentence `Homeopathy most certainly does not work, except as a placebo.` . That is simply a logical fallacy. We do know that placebos objectively work (I have no studies but would suggest to check the references in the Placebo wikipedia page for plenty). So if homeopathy works as a placebo, then it *does* work. Also, if you assume that "talking medicine" works, then this part of it "works" as well. This question is not about homepathy, but about studies about homeopathy. – AnoE Dec 01 '17 at 14:54
  • @AnoE It works only as a placebo, and not as what it pretends to be. If you take a glass of water and give it to someone who doesn't know any better, and says it's a homeopathic wellness tonic, they might feel better. If you give someone a bottle of homeopathic heart medicine or something, it's not going to cause them any harm at all. Because the water and the heart medicine are exactly the same. It's all psychological. Now try that experiment again with for example a sugar pill, and a cyanide pill, they'd soon find out which is which. – Christian Dec 01 '17 at 17:07
  • 1
    @Christian A tip, if you ever need to "answer" on an SE because you lack commenting privileges: preface the post with "this is a long comment" so people understand it's not intended as a true answer. – J.G. Dec 03 '17 at 20:07
  • @J.G. Fair enough, I'll keep that in mind. Thanks! – Christian Dec 04 '17 at 08:57
  • 1
    This answer seems to be more of an *ad hominem* than a meaningful answer. – Robert Columbia Dec 04 '17 at 15:35
  • @Robert Columbia As I've stated previously, it was meant as additional information to supplement a prior explanation posted by another user. I simply hadn't unlocked commenting privileges yet, and didn't know about the etiquette regarding prefacing post with "this is a long comment". That said, I did address the person making the claim rather than the claim itself, because the person making the claim has very good reason to cherry pick data, and perhaps even outright lie to you, since she's essentially just advertising her own product. – Christian Dec 05 '17 at 06:30
  • @Christian It's still an *ad hominem* argument. The things that you mention have no bearing on the veracity of the assertion made, any more than you being, say, a founding member of the International Association of People Determined to Stamp Out the Quackery of Homeopathy would have any bearing on the veracity of the assertion you make. – BobRodes Dec 10 '17 at 00:49
  • @BobRodes Absolutely. As I said it was meant to be a long comment to supplement a better answer posted previously, but I didn't have comment privileges on this board at the time. The reason I deemed it worth mentioning, is because she has a vested interest in promoting this quackery, so even if those studies were entirely correct, one should be cautious about thinking that's the whole story, and look beyond just her cited studies. We know what she's saying, but what isn't she saying? Well, of those five studies, four might show what she wants, but what about others? – Christian Dec 11 '17 at 07:04
  • @Christian One might just as well argue that the reason that you deemed it necessary to mention it is that you have no real evidence to support your position that it is quackery, coupled with an emotional need to prove that position and a certain laziness that prevents you from going to the trouble of finding real evidence to do so. If your need to prove your position is strong enough to speak up, then it ought to be strong enough to find and present evidence that actually supports your position. What about other studies, indeed? Suppose you present some that show what *you* want. – BobRodes Dec 11 '17 at 07:46
  • @BobRodes Glad to see you're not above the occasional ad-hominem or getting emotional either. You want me to post sources debunking that water somehow has memory, and that a liquid so diluted that it's in fact just water has the power to cure anything except dehydration? I don't think it's my job to do that until the water wizards have given something substantial to show why it might really work, which they have not. – Christian Dec 12 '17 at 06:12
  • @Christian If you're going to call into question some studies that purport to support the efficacy of homeopathy, then it's on you to support your point of view. You don't do that by saying that the person who mentions the studies is biased, and you don't do that by putting forth a characterization of homeopathy that "everyone knows" is ridiculous (an "ad populum" argument). Neither supports your point of view. Now, just so you know, I'm skeptical of homeopathy myself. But that means I have an open mind; I'm equally skeptical of your position. You haven't said anything to change that. – BobRodes Dec 15 '17 at 01:21
4

The article is formally correct in its statements (four of these five reviews indeed say that), but it is either deliberately misleading or abysmally researched, and written by someone lacking professional knowledge (or, both).

It took me 20 seconds on Medline to find another four systematic reviews (although in the Guardian author's defense, three of them published after the Guardian article), and a reference to a publication of around the same time as the Guardian article which did a more profound research on the Cochrane database of SRs, coming to a much different conclusion:

The findings of currently available Cochrane reviews of studies of homeopathy do not show that homeopathic medicines have effects beyond placebo.

The Guardian article's author is very obviously in favor of homeopathics and writes in an heavily argumentative, anecdotical-emotional, and if I'm allowed to say almost polemic way (some emphasis added by me):

(An anecdote fo how I ruined a dinner party)
[...]
I know homeopathy works
[...]
The facts, it seems, are being ignored. By the end of 2009, 142 randomised control trials (the gold standard in medical research) comparing homeopathy with placebo or conventional treatment had been published in peer-reviewed journals.
[...]
The "sceptics" campaign had a breakthrough

The facts. The facts are that the quoted reviews state that the overwhelming majority of trials, if they had any interpretable results (apparently there were others too, no information), was of sheer embarrassing quality. Gold standard is a nice buzzword, but the facts are, almost all of the studies named in e.g. Kleijnen's review do not meet the gold standard (not even remotely), as most of them were (contrary to the above claim) not even placebo controlled (or properly randomized, or in some cases both). In addition to that, the larger number of studies had population sizes under 25 which make a meaningful interpretation of data... let's say, a challenge.
Oh heck, I guess I'm with the "sceptics" campaign.

I am personally quite impressed how the reviewers in those 5 systematic reviews still decided to be so forgiving as to allege a possible effect, or some effect with a weak level of evidence when they more or less uniformly stated that significant results were only to be found in the poorest of all studies. Cucherat even calculated (in my opinion doubtful) combined significance levels, followed by explicitly stating that P rapidly goes towards 0.08 if you eliminate the trials with the lowest quality. Then again, however, followed by the puzzling statement "there is some evidence (... of low strength)" when he basically just demonstrated the opposite.

Be that as it may, the claim made by the Guardian article is, from a certain point of view, perfectly correct. Four of these exact five reviews indeed attribute some effect.
Only just, the article, uh... omits some (probably unimportant!) information that doesn't fit so well into the story.

Damon
  • 432
  • 2
  • 7