6

The EPA and other regulatory bodies have long concluded there is a casual relationship between poor (ambient) air quality and death.

This news article challenges that claim:

researchers have finally published a study claiming to have debunked science the Environmental Protection Agency (EPA) has used to justify imposing costly regulations on U.S. industries.

The study led by veteran statistician Stan Young found “little evidence for association between air quality and acute deaths” in California between 2000 and 2012.

Is it true that this study 'Air quality environmental epidemiology studies are unreliable' by Stan Young debunks the EPA's claim that there a causal relationship between poor air quality and death?

Community
  • 1
Liron Yahdav
  • 257
  • 1
  • 6

2 Answers2

8

This is effectively a partial answer. It does not seek to respond to the overarching question of whether or not there is a causal relationship between poor air quality and death. I'm not currently aware of any notable sources who have claimed that it does not. It only seeks to address the direct claim that Stan Young's paper "debunked" the existing scientific work the EPA had used to determine it. The answer to that is... not really. He's at best pointed out an indicator of potential concern that should be checked further.

This paper is probably technically correct in its assertions. It assumes bad faith on the part of fellow researchers (effectively claiming that they first acquire their data sets and then poke through them in order to find a particularly juicy conclusion, without including that in their calculations of statistical significance). It was not able to find publication for over a year. When it was published, it was published in a journal with a probable pro-corporate bias. This paper may well be making a valid point about statistical rigor, but "debunked" is a rather severe overstatement for the strength of the conclusions that can actually be reasonably drawn.

  • The Daily Caller is a Conservative News and Opinion site, reporting on politically charged science news. "Science News" of all sorts is chronically bad at actually knowing what papers are talking about, and "News and Opinion" sites tend to be a little more fast and loose than most, when reporting things that support their ideological position. This finding supports the conservative position. This does not inherently undermine the findings of the paper in and of itself, but it does mean that the article itself doesn't add much (if any) credibility. If you include the tendency towards both sensationalism and inaccuracy of science news in general, it means that the claims that initiate in the article may require particular scrutiny.

  • Regulatory Toxicology and Pharmacology is a monthly peer-reviewed scientific journal. The editor is Gio B. Gori. Wikipedia suggests that he's spend the past few decades working more or less directly for cigarette companies, and googling his name seems to support that. The journal is published by the International Society of Regulatory Toxicology and Pharmacology (http://www.isrtp.org/) which openly acknowledges that they have as one of their major objectives "identifying and engaging the policy makers and scientists who would be most concerned with developing, modifying, or applying those regulations, or affected by their development, modification, or application." - or, in short, lobbying. The Journal clearly describes itself as being intended to support policy decisions. Wikipedia suggests that the institute is financed in part by the tobacco, pharmaceutical, and chemical industries. Their own webpage merely states "Sponsors to be announced soon", and looks like it has done so for multiple years. None of this is proof of anything, but the most likely state of affairs given the available evidence is that the institute is largely a pro-corporate lobbying group that does not wish to make that obvious on their website. That in turn would imply rather heavily that the journal would be at least somewhat biased. It is peer-reviewed, and these conclusions are not nearly enough to suggest that it will knowingly publish things that are false or lack academic rigor, but there is a good chance that the choice of which papers to consider may be influenced by political slant.

  • The paper, on direct observation of the abstract https://www.sciencedirect.com/science/article/pii/S0273230017300673, appears to be selecting eight papers that were used in meta-analysis papers, looking at the available information, determining how many things could have been tested for with the given data, and then asserting that random chance would have produced a reasonable number of apparently statistically significant results, and thus that the meta-analysis results aren't as compelling as they might be. It was also mentioned (in the Daily Caller) that he had to shop it around for a few years before he got anyone to publish the thing - suggesting that no one was willing to publish it until he got to the industry-friendly ears at Regulatory Toxicology and Pharmacology.

So, short answer is that it looks like the paper is probably technically correct, and it may even have a valid point (saying that people are acting with more certainty than they really ought to have without a few more tests), but it's not anything like as strong as "debunked".

To clarify further, at least by my understanding of Young's abstract, he's essentially assuming bad faith on the part of the researchers of the papers he is studying. His thesis appears to be, in essence, there are enough possible questions that could be asked of the datasets in question that someone who was willing to take their time to pick and choose could easily find questions that would appear to be statistically significant, even if the input data was noise. He concludes from that that the papers in question, and all meta-analysis papers based off of them, are unreliable. It is not clear from the abstract how how chose his eight example papers, given that there are apparently over 100 papers on air quality available.

Ben Barden
  • 3,657
  • 2
  • 15
  • 20
  • 1
    Your bullet points go straight to the second bottom rung of https://en.m.wikipedia.org/wiki/Ad_hominem#/media/File%3AGraham's_Hierarchy_of_Disagreement-en.svg – daniel Aug 31 '18 at 22:04
  • 4
    I'm fairly certain that all US courts would admit the sources of financial support of Dr Gori as evidence of potential bias. I'm not sure Skeptics should have a higher threshold. – Andrew Lazarus Aug 31 '18 at 23:49
  • 2
    @AndrewLazarus: Not higher. Different, because science has different goals to courts. – Oddthinking Sep 01 '18 at 05:58
  • 4
    (+1) Also https://www.heartland.org/about-us/who-we-are/s-stanley-young seems to suggest the author of that paper (S. StanleyYoung) is getting some support or has some kind of affiliation with the [Heartland Institute](https://en.wikipedia.org/wiki/The_Heartland_Institute). They are skeptical of evertything from the effects of smoking to the existence of global warming. – Fizz Sep 01 '18 at 09:35
  • There's also [an exchange in JAMA](https://jamanetwork.com/journals/jama/article-abstract/2681730) between Young and the authors of a recent study. The technicalities are probably beyond what would interest most people on SE, but authors' reply ends with "The studies in [the meta-analysis](https://www.ncbi.nlm.nih.gov/pubmed/24706041) used multiple different modeling techniques and arrive at similar results to ours, indicating the robustness of the results, despite the outlier study he [Young] cites." – Fizz Sep 01 '18 at 10:14
  • Although this answer contains some valuable information, it is incomplete for it does not address the substance on its merits. – gerrit Sep 03 '18 at 11:26
  • @gerrit I've made an edit, in an attempt to address your comment. Did this help? If not, could you please clarify what would be necessary to address it on its merits? I don't have access to the contents of the paper itself. – Ben Barden Sep 04 '18 at 13:59
  • @BenBarden It does help (I had already upvoted) – gerrit Sep 04 '18 at 14:24
  • 5
    @daniel It's true that I'm including analysis of the bias of the news site and the journal, but I think it's pertinent. Specifically, under normal circumstances, journals and news sites could be expected to add credence by filtering out unreliable/biased/non-pertinent papers. In this particular case, though, the probable biases of the publications undermine that benefit to a degree (though not entirely). I've tried to make it clear that my bias analysis is intended in that light. – Ben Barden Sep 04 '18 at 16:07
  • @daniel I've edited the answer to both expand on and make more prominent my response to the substance of the question. Does that answer your concern? If not, what modifications/additions would you suggest? – Ben Barden Sep 04 '18 at 16:09
  • @BenBarden I think my concern is still valid, and now I can't tell if you are answering yes or no to 'Is there a causal relationship between poor air quality and death?' It's a tricky one to answer and the strength of epidemiological studies gets flimsy depending on the readers mood, and what you count as air quality. – daniel Sep 04 '18 at 22:25
  • @daniel Thank you. I have my answer to better clarify what it is intended to address. Unfortunately, the question is unclear on what exactly it's asking/which claim it's challenging. My answer is probably only a partial answer to the question as intended, but it covers the topic of almost all of the question body, so it would seem to be at least somewhat significant. Given that clarification, are your concerns addressed? If not, could you please explain in more detail what the remaining issue is? There must be at least some value in considering the reliability and bias of sources. – Ben Barden Sep 07 '18 at 17:20
  • @BenBarden theres no value there, the source of information doesn't have anything to do with how true it is – daniel Sep 07 '18 at 21:52
  • 4
    @daniel information can be true or false regardless of where its found, but the source of information says a great deal about its credibility, and credibility is one of the few things we have to go on when figuring out how true something is. That's why we don't just blindly reference Wikipedia and assume that it's correct. That's why scientific journals are given more credence than random websites. Provenance matters. – Ben Barden Sep 09 '18 at 19:20
  • Provenance matters, when facts don't. Your answer is like an opinion piece, well cool opinion, I don't agree. – daniel Sep 09 '18 at 20:33
  • I do find it interesting that he talks about bias in sample selection, then you aptly point out, he doesn't detail his own sample selection. Definitely suspicious. – JMac Sep 12 '18 at 19:39
  • @JMac he doesn't detail his own sample selection in the abstract. It's entirely possible that he has good and well-explained reasons for the ones he picks detailed thoroughly in the body of the paper. I'm just not in a position to evaluate it. – Ben Barden Sep 12 '18 at 20:20
  • @BenBarden That's what I'm getting at. He seems to leave the exact same gap in his argument that he's used as evidence in his own article. One the one hand, I doubt they would put all that effort in to leave such a silly and potentially blatant flaw. On the other hand... you would think they would have the foresight to explain their process; due to the very nature of the argument they are making. All I can conclude is that it is definitely interesting. – JMac Sep 12 '18 at 20:24
1

The larger (heading) question: 'Is there a causal relationship between poor air quality and death?'
It is undebatable that some amount of pollution in the air will cause death, we all need air to breathe and this has happened in history, Most agree there should be regulation on air pollution to prevent these deaths. An example from the comments is from 1952 when thousands died from the great smog of London.

The smaller (body) question: 'Is there evidence for association between air quality and acute deaths in California between 2000 and 2012?'
No, there is no good solid evidence of deaths caused by this current level of pollution, there is no smoking gun, like with Mesothelioma for asbestos. Its possible someone will find solid evidence for this time period in the future, possibly right after I post this answer.

What about the Epidemiological studies? I will borrow from the previous answer here to show why they are not solid evidence. Those types of studies are strong enough to show the link of smoking and lung cancer (again for most people, some die hard smokers still think the studies are inconclusive), but comparing the smoking studies to the air pollution studies has a huge difference of the relative risk.

There is this study on PM pollutants in outside air with a relative risk factor of 1.1.
Compared to this study on smoking, which shows a relative risk at least 10 times larger than the previous study.
I was looking into particulate matter in particular with the last answer so if anyone else finds comparisons of epidemiological studies for smoking cigarettes vs air pollution in California between 2000 and 2012 with comparable relative risk factors please feel free to add them.

daniel
  • 735
  • 7
  • 13
  • 3
    You seem to be saying "The evidence doesn't suggest that it's as bad as smoking, so it's not meaningful." I'm perplexed why the standard should be "as bad as smoking". There are a *lot* of things that aren't as bad as smoking, but are still bad. Also, you make the assertion "No, there is no good solid evidence of deaths..." and you don't say what evidence you base this assertion on. Was it a three-day dig through available research journals? A five minute google search? Off the top of your head? It's a pretty core assertion. It deserves at least some indication of where you got it. – Ben Barden Sep 11 '18 at 14:45