33

This popular tweet (over 10000 retweets) claims that:

This study shows that after three months the vaccine effectiveness of Pfizer & Moderna against Omicron is actually negative. Pfizer customers are 76.5% more likely and Moderna customers are 39.3% more likely to be infected than unvaxxed people.

It offers the following image as proof: Table

It also links this study: Vaccine effectiveness against SARS-CoV-2 infection with the Omicron or Delta variants following a two-dose or booster BNT162b2 or mRNA-1273 vaccination series: A Danish cohort study

I want to verify two things:

  1. Is this assessment of the study accurate?
  2. Is the study itself trustworthy and accurate?
Laurel
  • 30,040
  • 9
  • 132
  • 118
Ruslan Oblov
  • 3,513
  • 6
  • 19
  • 38
  • Old question, but note that vaccine effectiveness is a technical term and not the same as vaccine efficacy (which is more what we'd intuitively think it means). More about this at a recent, similar question https://skeptics.stackexchange.com/a/54220/8865 – cbeleites unhappy with SX Jan 02 '23 at 22:59

3 Answers3

68

No it doesn't. That is directly mentioned in the study:

The negative estimates in the final period arguably suggest different behaviour and/or exposure patterns in the vaccinated and unvaccinated cohorts causing underestimation of the VE. This was likely the result of Omicron spreading rapidly initially through single (super-spreading) events causing many infections among young, vaccinated individuals.

This study essentially compared vaccinated and unvaccinated people directly, it didn't try to adjust for many confounding factors. This is a really simple study, it's only 6 pages in total. It's more like looking at raw data, there are a ton of potential confounders here that simply aren't part of this study. You can see from the quoted paragraph that the authors clearly think that the negative values are an artifact of the study design, and not any real effect.

Mad Scientist
  • 43,643
  • 20
  • 173
  • 192
  • 27
    Pardon my ignorance but I don't get this answer. What you say seems to be just one interpretation. Yes, the data *absolutely can* be interpreted to mean that the effectivenes of said vaccines vs omicron is negative after said amount of time. In fact, that is literally what the table says. Where is the comparison with unvaccinated people? What do they have to do with it? The certainty with which you deny what the table evidently indicates seems scientifically questionable to my humble little brain. Even if slight underestimation of VE was true, a whole **-76%** is just an artifact? *Really?* – csstudent1418 Dec 25 '21 at 10:25
  • 59
    @csstudent1418 They compared vaccinated with unvaccinated people to estimate vaccine effectiveness. The problem is that you have to make really sure you're comparing similar populations, or the results will not actually measure vaccine effectiveness. For example, if only vaccinated people are allowed to attend restaurants or parties, this will likely lead to many more vaccinated people being infected by a variant that can circumvent the vaccine protection, because the vaccinated people have more exposure than the unvaccinated that can't go to restaurants. – Mad Scientist Dec 25 '21 at 11:20
  • 54
    @csstudent1418 It's the typical "correlation != causation" problem. If lots of companies require their workers to be vaccinated, unvaccinated people won't be at the worklace, so they'll have fewer workplace accidents. But that doesn't mean that the vaccine causes workplace accidents. – Barmar Dec 25 '21 at 13:12
  • 10
    I'm with @csstudent1418, if the line with by far the largest number of cases is suspected to be an artifact, then for sure the other lines with down to 0,5% the number of cases have to be much more suspicious. Just denying one of four lines in the same table, and just the line that suggests vaccines being not as effective as promised, seems like blatant pro-industry-bias to me. – Haukinger Dec 25 '21 at 13:33
  • 27
    @Haukinger I wouldn't trust any of the values in that paper on their own. The interesting part is the decline over time, and that is likely to be equally affected by these kinds of biases, so the trend is more reliable than the individual values. I wouldn't really trust this particular paper very far, but the conclusion that the protection wanes over time is also supported by a lot of other observations. – Mad Scientist Dec 25 '21 at 13:41
  • 1
    Given the high vaccine uptake rate in Denmark and stratification of unvaccinated populations, it'll be pretty hard to do a case control analysis. The only reliable part of this study is relative within population (as noted). We can get a better absolute vaccine efficiency estimates from uncontrollable spread in countries with poor vaccine uptake. – CJR Dec 25 '21 at 14:54
  • 8
    @Haukinger This answer does nothing more than quote from the study itself and elaborate on the issues with it. Specifically, nowhere does it "just deny one of the four lines on the same table". Claiming pro-industry bias is a leap too far. Your comment would be more helpful if it were pointing out something specific in this answer which you feel is incorrect. – JBentley Dec 25 '21 at 16:19
  • 11
    I don't really accept this as an answer since the comment in the study is just a hypothesis on what could, in theory, produce this result. The result could have also been produced, or contributed to, by a vaccine that causes one to be more susceptible to this variant. It's fair to say that it's unknown at this point. There is no clear data proving either. – Ruslan Oblov Dec 25 '21 at 21:00
  • 3
    @RuslanOblov For the previous variants we have very solid data indicating that no antibody dependent enhancement happens. And there are very obvious biases in this dataset that can explain the numbers, so it would be quite a huge leap to interpret this result as ADE. – Mad Scientist Dec 26 '21 at 10:15
  • 2
    @RuslanOblov Science works 'just on hypotheses' as you correctly point out. You are welcome to present alternative hypotheses, as long as you can provide evidence to support them. You claim proof, it seems, to accept any explanation but evidence turns into proof only in legal courts, not in the scientific realm, where we must live with the nagging uncertainty of 'yet unfalsified'. – Niels Holst Dec 27 '21 at 10:09
  • 2
    This is a prime example of why things need to be tested exhaustively, instead of relying on a single source. One study or paper doesn't provide a full understanding of the situation and more testing needs to be done with many different parameters to either verify these results or verify the doubts/concerns of the paper's very own authors. – computercarguy Dec 27 '21 at 18:16
  • 1
    Another point to mention: the paper seems to measure *infection* rates, whereas for most individuals, more important is the risk of getting sick, hospitalised, dying, or getting long covid. The current vaccines protect poorly against getting infected with new variants, but protect quite well against getting seriously sick with them. Therefore, even if more people get infected with omicron after being vaccinated, that does **not** show that the effectiveness againt getting sick is negative. – gerrit Dec 28 '21 at 09:57
  • This answer would be greatly improved if you showed how the figure could be an artefact, rather than real. Maybe show, as an example, that assumptions in the beginning drastically change the numbers, or whatever it is that you mean when you basically say that simple changes in the study could drastically change this figure. –  Dec 28 '21 at 14:45
  • 1
    @csstudent1418 I don't think anyone (who knows how to read the most basic of statistics) would describe .76 as a plausible random fluke, but an artifact can be as arbitrarily large as a genuine effect size. Whether it's collection bias (eg people who distrust vaccines are also less compliant with testing and government data collection) confounders (eg people who are vaccinated are protected by not leaving the house as much) or just a mistake in choosing the baseline (eg categorising people who have immunity from a prior infection as baseline) these could all have been the IV in another study. – Josiah Dec 29 '21 at 00:08
  • I would think that Omicron first infects mostly the unvaccinated, and after a few months there are no unvaccinated left to catch it, and the new victims would be vaccinated. – gnasher729 Jan 05 '22 at 20:57
17

This is a very good illustration of one of the most fundamental lessons of statistics: statistics can show you that an effect exists but not what the cause or nature of the effect is.

Suppose that a certain country introduces a new road rule that cyclists cannot use public roads unless they wear a helmet. Six months later, someone does a study and shows, statistically, that cyclists wearing helmets are more likely to have a fatal crash than those who aren't. Should we then conclude that the helmets make cycle crashes somehow more dangerous? Should we suggest that helmets may physically work but the psychological effect is to make the cyclists take more risks? Or should we note instead that cyclists travelling in the most dangerous setting amidst traffic are selected to be the helmet wearers, whereas cyclists without helmets are restricted to the park? Actually the answer is necessarily unclear and the statistics alone cannot clarify it.

The situation is the same here. Yes, it is theoretically possible that the vaccines are biochemically opening the door to omicron, but that is not proven and would be similarly surprising to helmets accentuating head injury. There is prior research into the economic theory of moral hazard, suggesting that vaccinated people may take more risks because they think that they are safer than they are. Denmark uses the coronapas system to filter access to public transport, service and hospitality industries, and other large or close packed groups, which means vaccinated people are more frequently selected for the dangerous settings a bit like our cyclists on the highway.

As such, it is from the statistics alone intrinsically unclear whether the vaccine is unfit for purpose, or whether the rule that vaccinated people are assumed safe enough to be allowed into dangerous places is unfit. Denmark believes that other evidence favours the latter explanation, and as such the covidpas validity will change to require a vaccine not more than 7 months ago.

Josiah
  • 286
  • 1
  • 5
  • 1
    The keyword you might mention here is [risk compensation](https://en.wikipedia.org/wiki/Risk_compensation) or risk homeostasis. – gerrit Dec 28 '21 at 09:55
  • Indeed, short of some natural experiment, it will be hard to convince people. I would think however, that the unvaccinated camp skew younger, and tend to be less/not worried about the virus, and thus take more exposure risks. With this in mind, I would propose we could get a reasonably confident read on efficacy vs omicron *if* we also had an age control. Age is the big factor to absolutely control for however. We know the vaccines don't work particularly well in general, and that age is the dominating protective factor. If unvaccinated are sufficiently younger, that can drive this effect. – JPErwin Dec 29 '21 at 16:10
  • 1
    @JPErwin I don't think we can know the relative importance of confounders ahead of time. One that I'm concerned about, because it is particularly hard to untangle, is variation in testing /reporting compliance. If unvaccinated people test less and report their tests less than vaccinated people (and we have no data on how much less; they tend to distrust researchers) it could just be that many unvaccinated people got infected but their infections aren't in the database. I don't think an age control or any other statistical method could even theoretically fill in systematically missing data. – Josiah Dec 30 '21 at 00:24
  • On further investigation, it seems that even the test asymmetry can be addressed. I will try to update the answer later. – Josiah Dec 30 '21 at 11:22
1

We should be careful not to confound two things:

  • Protection against getting infected, and
  • Protection against getting sick with the subsets of hospitalised, dead, and long covid.

The study measures protection against getting infected:

Vaccine effectiveness (VE) was estimated in a time-to-event analysis of Danish residents ≥12 years comparing the rate of infections in unvaccinated and vaccinated individuals with a two-dose BNT162b2 or mRNA-1273 vaccination series.

It doesn't seem to consider whether those infections are symptomatic or not. This is not the authors' fault, but in science communication, people may see the headline "negative effectiveness", and think they're safer without a vaccine. This conclusion is not supported by the available evidence in the paper. In fact, they themselves quote another study:

A recent study from England (in preprint) found higher effectiveness against symptomatic Omicron initially after BNT162b2 vaccination followed by a rapid decline in protection, and that VE increased to 75.5% (56.1 to 86.3%) two weeks after booster vaccination using unvaccinated individuals as comparison.

Unfortunately, they don't actually cite the preprint, so we cannot tell the magnitude of the declining efficiency against symptomatic infection for people who have received two doses of the vaccines under consideration.

All quotes from the original paper.

(On another note: the manuscript is very short and has not yet been peer-reviewed. It is possible that even the conclusion as presented in the manuscript does not follow from the available data.)

gerrit
  • 17,636
  • 17
  • 84
  • 137