Is there a correlation between vaccination levels and new COVID-19 cases?

Question

In the study Increases in COVID-19 are unrelated to levels of vaccination across 68 countries and 2947 counties in the United States, it is claimed that there is no relation between vaccination and increase in COVID-19 cases. Already in the title it says:

Increases in COVID‐19 are unrelated to levels of vaccination across 68 countries and 2947 counties in the United States

Their main finding is:

At the country-level, there appears to be no discernable relationship between percentage of population fully vaccinated and new COVID-19 cases in the last 7 days (Fig. 1).

Here's a reproduction of Figure 1:

I'm particularly interested in this topic because I made exactly the same analysis (I think) for my own home country Germany, just yesterday. My results are show in the following diagram, exhibiting a very clear relation, with 10% lower vaccination leading to an increase of (roughly) 350 cases per 100.000 averaged over the last 7 days.

---

EDIT: on request by a commenter:

The data is provided by the Robert-Koch-Institut. Unfortunately, the relevant information is easily accessible only in the German version. In the English version I had difficulties to find the vaccination rate per federal state.

In addition, I used the data that was available at that day but the data is updated frequently. I haven’t checked for historical data.

Answer 1

I think there are several sources of confusion here;

1.

Increases in COVID‐19 are unrelated to levels of vaccination across 68 countries and 2947 counties in the United States

This statement is basically impossible, but is a very common misstatement of statistical results. You can see that the slope in figure 1 is not exactly zero (and it would never be for any real data set!); speaking very precisely, "... are unrelated ..." is false. (In fact, the observed relationship is positive). However, it is extremely common for scientists to frame relationships that are not statistically significant at some predetermined level (usually p=0.05) as "no relationship".

The authors go on to say

At the country-level, there appears to be no discernable relationship between percentage of population fully vaccinated and new COVID-19 cases in the last 7 days (Fig. 1).

This is a little better, if we equate "no discernable relationship" with "no statistically significant relationship"; the authors use other phrases like "lack of a meaningful association" and "marginally positive association".

In fact, a quick re-analysis of the data given by the authors suggests that the relationship has a p-value of 0.04 ("significant" by the usual rules), with a slope of 14 (cases per million per week)/(% vaccinated), with a 95% confidence interval from 0.65 to 27.

2. The much more fundamental problem is that it is almost impossible to draw reliable conclusions from this kind of observational data: as pointed out in the comments, there are way too many confounding variables, time lags, and sources of error to conclude either that the true underlying (causal) effect of vaccination on COVID-19 incidence is small or large, positive or negative. (It's very unlikely that vaccination would increase COVID-19 incidence, although conceivably this could be caused by risk compensation or by increased transmission by vaccinated people by mild, asymptomatic infections.) Even very sophisticated statistical analyses struggle to extract meaningful signals from observational data on epidemics (and experts debate the validity of various methods).

Answer 2

For US counties, more rural areas have a sparser population that may be less susceptible to contagious illness spread, and also have lower vaccination rates than urban areas. This introduces a potential confound between cases and vaccination status unrelated to vaccine efficacy.

For countries around the world, there is tremendous variation in testing that makes countries difficult to compare. An under-resourced country is likely to have both fewer vaccinated people and fewer recorded infections; these associations are going to be confounding in any analyses by country.

It seems a lot more appropriate to go down to county-level (or, at a minimum, state-level) and look at individual people who are vaccinated versus not, and see how they fare. An example would be statistics for the US state of Wisconsin, available from their Department of Health Services.

They report 456.4 cases per 100k vaccinated, and 2255.1 cases per 100k unvaccinated in the month of October 2021. These numbers are age-adjusted, though a similar relationship is seen in the unadjusted numbers by age group. You can also look at past months, which show similar patterns.

There are larger gaps by ratio for hospitalizations (12.2 vaccinated versus 132.0 unvaccinated, per 100k) and deaths (1.8 vaccinated versus 27.3 unvaccinated, per 100k), again for the month of October 2021.

Answer 3

(comments turned into an answer as per @Oddthinking's request)

Summary: there are too many other important influencing factors to allow meaningful conclusions about vaccination efficacy from these population level diagrams.

those kind of analyses are useless w.r.t to showing the effects of vaccination on slowing down (or maybe not) a Covid-19 wave?

More direct studies on the effect of vaccination are available at the individual level, e.g. Vaccine effectiveness against infection and onwards transmission of COVID-19: Analysis of Belgian contact tracing data, January-June 2021,
Which reports the following reduction in transmission compared to unvaccinated index patient - unvaccinated high risk contact person:

update Wrt delta, the CDC reports vaccine efficacy against infection for Moderna and Pfizer of 53 %**. They do not distinguish probability of being infected vs. infecting, the 53 % is the total reduction (diagonal of the matrix in the Belgian paper).

A UK preprint has similar findings (and time dependency, recommended to read), and similar to the Belgian study, the reduction is largely in lower probability of getting infected if you are vaccinated. A sizeable reduction in the probability of onward transmission is found only with very fresh 2nd vaccination.

I.e. unless you have very high vaccination rates with mRNA (or recovered, which the Belgian paper finds to be comparable), the effect on ongoing transmission in the population will only be a moderate slowing (much less than an order of magnitude, for delta maybe half - that is, delta spreads in vaccinated populations only a little bit slower than the original variant in unvaccinated).

---

You'd need to distinguish what other factors are controlled for. E.g. a region may be reacting to high levels of vaccination by accepting high incidence since the risk per infection is much lowered. High vaccination rates alone may not be suffient to quench ongoing transmission (consider delta with an R0 of 5. A population 100 % vaccinated with Astra or Johnson (according to the Belgian estimates) would still have to expect R > 1 without further measures to reduce transmission).

OTOH, if regions have largely similar measures in place, higher vaccination rates would be expected to slow down infections more.

Further considerations:

• Percentage vaccinated may be a rather incomplete picture without knowing the percentage of the population in question which is immune via recovery but without additional vaccination.

  • In Germany, recovered means "had a positive PCR test > 1 and < 6 months ago. Regardless of e.g. antibody titer. Which means that right now in Nov 21, we have few "officially" recovered since the last winter and spring waves are outside the 6 month window by now. From an epidemiological perspective, however, their immunity is not suddenly switched off after 6 months (the study linked above in J.Vaccine says "There was no significant difference between protection by full-dose vaccination and previous infection.", previous infection being defined as positive antigen test > 3 months ago). Also, the standing committee on vacciation's recommendation treats those with positive specific antibody titer even without knowing when the infection occured (i.e. recovered, but dark figure) as equal to the officially recovered.
    I'm not aware of data saying how many of the recovered from any of the previous waves are now vaccinated and how many are not.
  • Relevant for this question: news report about GPs sending not everyone of the same household for to get their own PCR test The region (Erzgebirge/Saxony) discussed is one that is in the news now due to very low percentage of vaccinated and high infection rates.
    My take on the situation: I expect they have a higher dark figure of people immune due to previous infection than other regions. Not to the extent that (nearly) everyone is immune there now (until summer, close to 10 % of Saxons have officially had Corona, even with dark figure of 100 - 200 % this cannot nearly close the gap in immunity), but it may very well be sufficient to shift them (the lowest vaccination rate data point) sufficiently into the center of the % vaccinated range to substantially reduce the observed correlation.
    For the other extreme, Bremen, there is some discussion how many of the vaccinations were given to commuters from Niedersachsen [they vaccinated whoever came - whereas e.g. in Hesse one had to go to the vaccination center of one's home Kreis (county).] - which may shift them also a bit further into the "big crowd".

• Are the case rates notification rates (like the official numbers for Germany) that have a huge dependendy on rules for testing (e.g. vaccinated being exempt from regular testing) or incidences including dark figure estimates?

• Also, "new cases in the last 7 days" (obviously) would undergo important changes over the course of a wave. And unless these happen synchronousĺy (on a time scale with 1 week averages) in all Länder - which would be a ridiculous assumption - this will easily mix up the diagram.
  Had you done your diagram for Germany end of August/beginning of September, Saxony and Thuringia would also have shown up at the low vaccination rate end of the diagram - but with low case rates instead of high case rates - whereas Bremen would have been at high vaccination high incidence (vs. high vaccination low incidence now [missing in OP's diagram]):

I found a table from Sept. 8th in a newspaper

Bundesland	Inzidenz	Impfquote
Baden-Württemberg	91.0	60.4
Bayern	77.5	59.5
Berlin	81.0	61.3
Bremen	117.6	71.4
Brandenburg	37.5	55.8
Hamburg	78.9	65.1
Hessen	113.6	60.6
Mecklenburg-Vorpommern	36.4	60.2
Niedersachsen	71.6	63.4
Nordrhein-Westfalen	109.4	64.2
Rheinland-Pfalz	103.0	61.6
Saarland	89.2	67.7
Sachsen	32.2	52.4
Sachsen-Anhalt	25.3	58.1
Schleswig-Holstein	49.1	65.7
Thüringen	38.3	56.2

The corresponding diagram is:

Putting this together with OP's table (which I can not(!) confirm looking at the recent historical data available at the RKI):

Seeing all this, I don't think "no relationship found" across a diverse set of populations should be that surprising. Nor should it be surprising that looking at a smallish subset of regions at one point in time can exhibit a pattern in one direction or the other.

Oh, and careful with significant: significance doesn't imply that the effect size is sufficient to be of practical importance. @BenBolker's point estimate of 1.4 additional cases /(100k population ⋅ week ⋅ additional % vaccinated) i.e. about 20 cases / (100k population ⋅ week) difference over the ≈ difference of 15 %-points in fraction vaccinated in your diagram would be barely visible as an increase.

The data from J.Vaccine. means that in terms of transmission the effect of vaccination is in the same order of magnitude as other measures like testing (and immediately quarantining if positive), proper vs. improper wearing of medical masks, meeting outdoors instead of indoors etc.

From a statistical point of view, that's about the most difficult situation we can find ourselves in: several influencing factors that have effects in the same order or magnitude. Such a situation can only be sorted out by very carefully designed studies, not by an overview glance at raw data. The latter would only be possible if all but one dominating factor could be dropped.

Answer 4

This feels like a "me too" answer given Bryan Krause's, but I'll just point out there are more reports of a negative correlation (between cases and vaccination level) in more geographically limited studies (which have the advantage of reducing the nosiness of the data due to e.g. reporting methodology across countries). I won't try to delve into the details of each:

• California (September 2021)

• New York state (April 2021)

The two-dose vaccination rate was a significant negative predictor of cases per 100K population in NYS counties (β= -.866, P=.031), with each percentage point of completed vaccination nearly equating to one case less in the daily count when controlling for county population size (β =2.732, P<.001).

• Israel (December 2020 - March 2021)

by analyzing vaccination records and test results collected during the rapid vaccine rollout in a large population from 177 geographically defined communities, we find that the rates of vaccination in each community are associated with a substantial later decline in infections among a cohort of individuals aged under 16 years, who are unvaccinated. On average, for each 20 percentage points of individuals who are vaccinated [...] the positive test fraction for the unvaccinated population decreased approximately twofold. These results provide observational evidence that vaccination not only protects individuals who have been vaccinated but also provides cross-protection to unvaccinated individuals in the community.

• EU (November 2021) deaths not cases though!

• On that outcome (deaths not cases) US July-September 2021 state-level data seems to agree

On the other hand, an even earlier (and even less cited) study (Fukutani et al.) in the same vein as Subramanian's also found divergence across countries.

Using the date available up to April 23, 2021, we performed a correlation analysis between the numbers of new cases with the daily vaccinations. As a result, 60 countries presented positive correlations (Table 1) and 27 countries with negative correlation (Table 1). [giant table omitted here]

So what's my "original research" conclusion? Looking at cross-country data needs more covariates. When comparing accross countries in different regions of the globe, one might have to consider that different (kinds of) vaccines predominate in different regions (to name just one covariate). And the other more obvious confounder across jurisdictions is that lockdown measures are/were intiated at different points in time in various countries (as well as being of different intensity/degree and length), and those affect the spread/reduction as well; in countries with enough local political power to make such decisions in a decentralized fashion, that effect (of lockdowns) can be observed at sub-national level as well, such as in a study on US counties. So when one compares two countries/regions even with similar rates of vaccination at a random point in time, (prior or present) lockdowns can be a substantial part of the difference (or "noise" if one doesn't account for it).

Some papers (like the one further above from Israel) tested their conclusions for robustness by changing the reference time (see fig 1d), but Subramanian's paper didn't do that... and in fairness, most other links/studies above also didn't do that, but it's probably more of an issue when the time window for the study in just 1-2 weeks rather than months. In Subramanian's it was one week.