9

ArsTechnica used a headline (and this has been reported elsewhere):

Apollo astronauts dying of heart disease at 4-5X the rate of normal humans

What is the evidence relating to the headline's claim?

This story is based on a sample of 77 astronauts and an animal model analogous to the radiation exposure felt when outside the earth's magnetosphere. 7 astronauts in the subsample of 44 who went to space got beyond the magnetosphere.

matt_black
  • 56,186
  • 16
  • 175
  • 373
  • 8
    Only 24 men have been to the moon, and they were all male with far better than average health. The sample size is too small and not sufficiently representative of the general population to draw any meaningful conclusions. – GordonM Mar 23 '18 at 11:29
  • And if there is a difference then no one knows why. Astronauts were exposed to a vast array of "unnatural" conditions in their training (not to mention military experience prior to entering the astronaut corps). – Daniel R Hicks Sep 03 '18 at 12:15

3 Answers3

14

The basic answer is: we don't know. If you look at the original paper, you will find table 2, which shows that the number of people in the Apollo Lunar Astronauts group was 7 (n=7). Of those, 43% (or a total of 3) died from cardiovascular disease. In general you can't use such small sample sizes to draw conclusions in this sort of situation.

According to the paper's methods, they use the "Fisher’s exact probability test". However, this test assumes you know the totals of each measure beforehand. In this case, they don't (they don't know beforehand how many people would die from each cause), which renders the test results unreliable. They claim that the test is too conservative, which is controversial even according to the references they cite to support its use (41, 42, and 43). But they then use those references to justify using less strict confidence levels than is normally accepted for these tests (again according to those references). If they hadn't raised those confidence intervals, many of their results would be negative.

TheBlackCat
  • 1,171
  • 1
  • 7
  • 11
  • 1
    Where did you get the numbers 3, [7](https://en.m.wikipedia.org/wiki/List_of_Apollo_astronauts) and the analysis that that use to small to draw a conclusion? – Oddthinking Aug 01 '16 at 06:33
  • @oddthinking: from the paper. That us the number of lunar astronauts that based their conclusion on. – TheBlackCat Aug 01 '16 at 11:47
  • 8
    Upvoted; this is why I do not like this part of the SE network. One does not need a journal citation (or any citation, for that matter) that says that **NO** conclusions should be drawn from such a ridiculously small sample size. The seven and three to which theBlackCat is referring are the number of Apollo lunar astronauts who had died at the time of this study (7) and the number who had died of cardiovascular disease (3). Yes, 3/7 is about 43%. It also is just about meaningless. – David Hammen Aug 01 '16 at 13:17
  • 3
    @DavidHammen if we don't ask for references to conclude that no conclusions can be drawn, how can we ask for references when someone does draw a conclusion nonetheless? It's, pardon me, very easy to argue that no citations are needed to support something we know to be true, while forgetting the same rules prevent people from posting something we know to be wrong. – Sklivvz Aug 02 '16 at 23:19
  • @Sklivvz -- In this case, the [obligatory xkcd](https://xkcd.com/882/) would have been more than sufficient reference. What that group did was statistically invalid. For this to have passed peer review in one of the most prestigious journals of all (*Nature*) is even worse. – David Hammen Aug 03 '16 at 02:33
  • 1
    A sample population of just seven is inevitably going to show all sorts of extreme deviations from the mean. – TheMathemagician Aug 03 '16 at 11:52
  • @DavidHammen: In fact with Fisher’s exact test, the size of the sample is indeed unimportant – that’s why it’s called *exact.* For small sample sizes, Fisher’s exact test still exactly tells you how unlikely it is to obtain the results by chance (assuming that the test’s condition are met) – the respective numbers just aren’t very small. This also happens with the data in question, where hardly any *p* values are below 0.05. — By the way: The biggest problem I see with that table (Table 2) is that it does not account for multiple testing. They make 40 comparisons after all. – Wrzlprmft Aug 03 '16 at 19:45
  • @DavidHammen: This was published in Scientific Reports, not in Nature. – Wrzlprmft Aug 03 '16 at 20:01
  • @Wrzlprmft: as I said in my answer, that is only true if the totals are known ahead of time. That isn't the case here, so the test is not exact. – TheBlackCat Aug 03 '16 at 22:56
  • @TheBlackCat: I do not dispute you there. I just dispute directly arguing against the sample size. – Wrzlprmft Aug 04 '16 at 05:48
5

If the question is:

Are the Apollo astronauts experiencing heart disease at a higher than average rate?

The answer is possibly, but the data is not rock-solid.

Otherwise, if you ask if the following is true:

Apollo astronauts dying of heart disease at 4-5X the rate of normal humans

This is a misreading of the original study, and is false strictly speaking, but is up for interpretation (see figure below)

The original scientific article (linked within the ArsTechnica article you link, is titled

Apollo Lunar Astronauts Show Higher Cardiovascular Disease Mortality: Possible Deep Space Radiation Effects on the Vascular Endothelium

It reports the following figure:

Caption:

The proportional mortality rate due to cardiovascular disease in the United States among individuals age 55–64 years, non-flight astronauts, astronauts that flew only low Earth orbit missions, all flight astronauts, and Apollo astronauts that flew missions to the Moon.

Notes:

*Significantly different from the US population 55–64 years of age at the time of death, P ≤ 0.05. †Significantly different from the non-flight astronaut group, P ≤ 0.05. ‡Significantly different from the low Earth orbit astronaut group, P ≤ 0.1.

They then proceed with other analysis (that you can read at the link above), but already from here it can be seen that:

  1. astronauts tend to be healthier than the general population (as it might have been expected, given that the selection also looks at medical data)
  2. Astronauts that have flown to the moon seem to have a significantly higher mortality rate w.r.t. other astronauts (P ≤ 0.05 vs non-flight group, P ≤ 0.1 vs Low-Earth orbit group)

NOTE: the 4-5X ratio mentioned in the ArsTechnica title, is the one between Lunar astronauts and Non-flight/Low-Earth Orbit astronauts. It could be argued that Astronauts are a selected subset of "normal humans", thus making the title true, but a strict interpretation of the data provided does not confirm this.

ADDENDUM: to follow-up on the discussion in the comments, one could add the death of Edgar Mitchell to the Lunar group, making the bar drop from 43% to 37.5%, a 5.5% decrease that is about 2/3 the height of the non-flight group. This obviously affects the P values listed in the paper (but I am unable to re-compute them). It is also worth noting that 17 Lunar astronauts are still alive, and their future deaths might completely reverse the statistic cited above.

SEP. 2018 UPDATE: Since the writing of this answer, four more Lunar astronauts have died, Alan Bean, John Young, Richard F. Gordon Jr., and Gene Cernan.
The cause of death is specified only for John Young (complications from pneumonia), I am thus unable to update the computation of the mortality rate figure.

Federico
  • 1,810
  • 3
  • 18
  • 25
  • Could you add confidence intervals to the graph? – Christian Aug 01 '16 at 09:23
  • 1
    @Christian I have not created the graph (I've lifted it from the Nature article), and they do not seem to be reported in the article, so I'm sorry to say that currently I am unable to do it. – Federico Aug 01 '16 at 10:06
  • @Christian -- The Apollo Lunar bar is exact. Of the exactly seven Apollo lunar astronauts who had died at the time of this bad study, exactly three of them (43%) died because of cardiovascular disease. The problem isn't the uncertainty. The problem is that seven is an absolutely ridiculously small sample size. – David Hammen Aug 01 '16 at 13:15
  • 1
    @DavidHammen : If you actually see the seven as a small sample size then you can calculate an interval based on 3 of 7. – Christian Aug 01 '16 at 13:23
  • 4
    @Christian, my stats book says you can't. "To be on the safe side [when calculating a confidence interval for a proportion], one should require both *np̂* and *nq̂* to be greater than or equal to 5" (*Probability and Statistics for Engineers and Scientists*, sixth edition, Walpole, Myers, and Myers). In this case, *np̂* is 3 while *nq̂* is 4. – Mark Aug 02 '16 at 03:32
  • @Mark: They use Fisher’s exact test, which does not impose any constraint on sample size (that’s why it’s called exact), so there is no need to apply such a rule of thumb, which I presume to be intended for non exact test based on assumptions that are not valid for small sample sizes. If your sample size is too small to make any conclusive statement, the test accounts for this and yields a high *p* value. — There are however other problems with the statistical analysis (see the other answer an comments on it). – Wrzlprmft Aug 03 '16 at 19:54
  • @Mark: the $n p$ and $ n (1 - p) \geq 5$ rule is usually the limit for normal approximation. Also for confidence intervals of proportions, there are other approximations that work for smaller $n$. While they also differ, they roughly indicate 95 % confidence intervals for 3 heart deaths out of 7 deaths in total ranging from 10 or 15 % all the way to somewhere around 75 or 80 % (normal approximation is 6 % - 80 %). Considering the width of the confidence interval, IMHO these differences do not matter in terms of the interpretation: you cannot say much from 7 cases. – cbeleites unhappy with SX Aug 05 '16 at 21:43
  • (3 out of 8 gives about 10 or 13 to about 70 %). Calculations were done in R 3.2.3 using package `binom` v. 1.1-1: `binom.confint (3, 7)`. – cbeleites unhappy with SX Aug 05 '16 at 21:44
1

Some problems with that article...

A total of 30 astronauts traveled outside the magnetosphere of the earth: three astronauts on ten missions: Apollo 8, 10, 11, 12, 13, 14, 15, 16, and 17. Not all landed on the moon, but all traveled to the moon, outside of the earth's magnetic influence, which is the premise of the article.

The study used seven lunar orbit astronauts, as seven out of those thirty have died. So, not only is the pool small enough to be statistically suspect, other factors such as age of the astronaut are not being considered.

The Apollo astronauts are all very old now, and at an age where heart disease is not unusual, whereas the total pool of astronauts studied includes a high percentage of much younger people, as NASA hired a substantially higher number of astronauts for the large crews for the shuttles plus the much higher number of shuttle missions flown as compared to Mercury, Gemini and Apollo.

This list of astronauts, along with hiring dates, shows the preponderance of much younger non lunar astronauts in the study. During Apollo, there were about 20-25 active astronauts depending on who was retiring and who was being hired, while at the peak of the shuttle years, 2000, there were 109 active astronauts. Same holds true for actual flying astronauts: Apollo had a crew of three and flew 15 missions including spacelab, while the shuttle had a crew of 5-7, and flew over 120 missions. So the non lunar astronauts studied tend to be much younger than the lunar orbit astronauts, due to the simple fact that we haven't sent a crew to the moon since 1972.

This PDF was the most complete age related heart disease study I could find. It shows under one percent for under 40, around six percent for middle aged people, rising to around 20 percent for 60+, and 32% for 80 and older... the lunar orbit astronauts are all older than 60, and a few are around 80. That should have been a big factor in any such study. It is not considered in this one.

Also not considered is the untimely and premature deaths of 14 non moon mission astronauts, on Apollo 1, and the Challenger and Columbia disasters. Those deaths were not natural and not related to health conditions, so it can't be determined if any of them would also have suffered from heart disease had they not died in the losses of the spacecraft. As the total non lunar flying astronauts analyzed was 37, over one third of that study group died in those three accidents, which certainly would have thrown off any conclusions since the study was related to long term health effects of lunar (or further) space travel.

So to get back to the original question - are lunar astronauts dying of heart disease at a higher rate? It is impossible to determine from that rather shoddy piece of work.

tj1000
  • 195
  • 1
  • 4