Has the 9th Circuit Court been overturned 80% of the time?

Question

This claim is being made in some parts of the media:

The 9th Circuit Court has been overturned 80% of the time.

Similar versions abound. Here is another one

Why? Because the “reversal rate” of the 9th Circuit is hovering at a solid 80 percent.

What does that mean? It means that this court is filled to the brim with individuals who have no regard for the constitution and who look to further an agenda that suits their personal politics — also known as “activist judges.”

This is in the context of their ruling against the Trump administration in relation to his executive order banning immigrants from 7 countries.

Is this true?

It might be worth noting that the Supreme Court doesn't tend to review cases that have no chance of being overturned. I.e., there's a selection bias in effect on the cases they review. — Ben Hocking, Feb 10 '17 at 12:59
To @BenHocking's point: I modified the question. If that is **not** the claim that the Trump administration made, please include a reference to what the spokesperson said. — Oddthinking, Feb 10 '17 at 13:49
Note that your first sentance and the quoted bit say two completely different things which one are you asking? — mmmmmm, Feb 10 '17 at 14:06
@Oddthinking Actually that is the claim they made. I think you have hit the nail on the head. What they should have said is what you have edited my question to. — Simd, Feb 10 '17 at 15:05
@BenHocking You are quite right. I see now that some places omit "that are reviewed" part and some don't. To make an answer complete, what is the percentage of all 9th circuit rulings that are overturned? — Simd, Feb 10 '17 at 15:12
@BenHocking That's not right. They tend to decide cases where there is jurisdictional conflict, i.e. the 9th versus any other district has a different legal interpretation — , Feb 10 '17 at 15:23
@user37696 here is a quantitative study of that: http://scholarship.shu.edu/cgi/viewcontent.cgi?article=1070&context=circuit_review — DavePhD, Feb 10 '17 at 17:31
@Ben Hocking: And per today's news, the administration isn't planning to appeal this ruling to the Supreme Court, either: http://thehill.com/homenews/administration/318973-trump-officials-scramble-after-travel-ban-defeat — jamesqf, Feb 11 '17 at 04:50
@user37696, that sounds to me like potato/potahto. Jurisdictional conflict implies more likely to be overturned, no? Here is a more complete discussion of factors influencing whether the Supreme Court will hear a case: http://litigation.findlaw.com/legal-system/how-does-the-u-s-supreme-court-decide-whether-to-hear-a-case.html — Ben Hocking, Feb 11 '17 at 15:24
"Regard for the constitution" or "agendas" are all myths. [MYTH OF THE RULE OF LAW](http://webcache.googleusercontent.com/search?q=cache:yykQx1K7k88J:faculty.msb.edu/hasnasj/GTWebSite/MythWeb.htm+&cd=3&hl=en&ct=clnk&gl=us&client=ubuntu) — Chloe, Feb 12 '17 at 21:29
Before you ask this question, you would be wise to do a minimum of investigating of the reputation of the sources you cite. In both cases, their postings are often debunked by many reputable research outlets. — Jesse C. Slicer, Feb 14 '17 at 15:04
This begs the question... How does this rate for the 9th circuit compare to that over other courts of the same level? — dotnetengineer, Feb 14 '17 at 19:44
@JesseC.Slicer Is that for the original question or a comment? There is a newly found problem with only looking at respectable sources which is that half of the US + the president now trusts other sources. — Simd, Feb 16 '17 at 12:10
@felipa the original question. It would be better stated as "knowing these two sources have often published falsehoods which have been debunked, is there any truth when they say ..." — Jesse C. Slicer, Feb 16 '17 at 15:12
@JesseC.Slicer Well that would be the left wing way to put it... :) Next you will be telling me not to trust Breitbart. — Simd, Feb 16 '17 at 16:29

score 249 · Accepted Answer · edited Oct 20 '18 at 15:19

Summary

It depends on your interpretation of the claim.

Were 80% of all 9th Circuit Court decisions overturned? NO, not even close.
Were 80% of the small minority of 9th Circuit Court decisions that made it to review by the Supreme Court overturned? Yes.

Only 0.12% of cases ruled on by the 9th circuit were overturned. 99.85% of cases were not heard by the Supreme Court. 0.03% of cases were confirmed by the Supreme Court.

In the remaining 99.88% of cases, the Supreme Court either wasn't brought in, chose not to take the case or approved of the 9th circuit's rulings.

This is from the data found by DavePhD. The total number of cases decided by the 9th circuit court of appeals was 114,199. Of that number, 107 were reversed, 33 were vacated, and 35 affirmed by the Supreme Court.

A total of 0.15% of 9th circuit cases were heard by the Supreme Court

Of those cases, 80% were overturned (reversed or vacated). This is the "Reversal Rate" and is the number they are using. The quote omits the fact that the "Reversal Rate", as understood by lawyers, does not include cases that weren't reviewed by the Supreme Court.

Cases referred to the Supreme Court are not a random sample of cases decided. They are selected by one of the parties to the case (usually the loser) because that party believes that there is an error of law in the decision and are willing to spend a great deal of money to have that perceived error verified. As such, cases referred to the Supreme Court are preferentially the ones most likely to contain a reviewable error of law and therefore be overturned.

Nevertheless, with data from Brythan (see their excellent answer), the Supreme Court sees approximately 7,000-8,000 petitions for writs of certiorari each term, only 80 of which they agree to take. Assuming the larger number for simplicity, that means 1% of cases requested for review are reviewed. Given that court reviewed 175 cases in the given time frame for the given court, we can extrapolate an approximate 17,500 cases were requested for review within those parameters.

This would give an approximate breakdown of 84.7% of cases weren't even considered by the Supreme Court, 15.1% of cases were declined by the Supreme Court, 0.12% of cases were overturned, and 0.03% of cases were confirmed.

Therefore, the biggest deciding factor from a purely statistical point of view, is "Will the Supreme Court hear the case"? 99% odds they won't, even with a petition. If you hit that 1% odds, then you have 80% odds of the case being overturned. Of course, there's actually an 84.7% chance you won't even seek the Supreme Court. But these numbers assume a randomly selected case, or that the decisions in a future case will be randomly made according to the past-performance statistics.

The case in question isn't going to be decided by statistics, but by judges making non-random decisions. The chances of the Supreme Court taking the case, and in fact whether they would overturn the decision of the lower court, cannot be determined by the past performance of all cases. The particulars of the case have a significant impact on the outcome, and neither the original articles, nor the statistics given here, take the particulars into account.

In short, they've taken a number that applies to a very small sliver of data and claimed it applies to all the data.

They've built an argument around that small sliver in order to make a claim that applies to the entirety of the 9th circuit. Sometimes they acknowledge that the data only applies to a subset, but they fail to clarify just how minuscule that subset is.

This is a common fallacy called the Statistics of Small Numbers fallacy or the Hasty Generalization fallacy, and also shows Selection Bias.

Comments are not for extended discussion; this conversation has been [moved to chat](http://chat.stackexchange.com/rooms/53455/discussion-on-answer-by-kevin-fee-has-the-9th-circuit-court-been-overturned-80). — Sklivvz, Feb 11 '17 at 11:52
Just deleted a large debate that should have happened in the chat room. Most were bickering about the correct interpretation of words. In an attempt to stop the debate, I added a summary. Still got a problem? Take it to chat. — Oddthinking, Feb 15 '17 at 12:46

score 73 · Answer 2 · edited Jun 17 '20 at 09:41

73

According to the American Bar Association article Supreme Court Reversal Rates: Evaluating the Federal Courts of Appeals:

The reversal rates in Figure 2 range between 55% and 84%. Interestingly, this comparison of reversal rates reveals that the Federal Circuit has the highest reversal rate at about 83.33%, and the Ninth Circuit has the second highest reversal rate at 80%. The Seventh Circuit has the lowest reversal rate at 55.26%. The median reversal rate is 68.29%.

The American Bar Association is defining "reversal rate" as (decisions reversed or vacated by the Supreme Court)/(decisions reversed, vacated, or affirmed by the Supreme Court).

In the time period of the bar association study (1999-2008), 114,199 cases were decided by the 9th circuit and 107 were reversed 33 vacated and 35 affirmed by the Supreme Court.

80% reversal rate does not mean that 80% of all the 9th circuit's decisions are reversed. For only about 1% of decisions is review by the Supreme Court requested by one of the parties, and of that 1% only a small fraction are heard by the Supreme Court.

For more recent information:

Percent reversed for the 9th circuit:

2015 80%

2014 63%

2013 92%

2012 86%

2011 71%

2010 79%

2009 73%

edited Jun 17 '20 at 09:41

Community

1

answered Feb 10 '17 at 13:27

DavePhD

103,432
24
436
464

Do these figures include cases in which the court refused even to hear the case? The vast majority of petitions to the supreme court are rejected with a one-line order. In such cases, of course, the lower court's ruling stands. – phoog Feb 10 '17 at 14:48
1

@phoog The 80% is only (reversed + vacated)/(reversed + vacated + affirmed), it does not include cases that are not heard. – DavePhD Feb 10 '17 at 15:02
1

Thank you. I think the key detail is what is the percentage that are heard? – Simd Feb 10 '17 at 15:06
6

This doesn't answer the question title "Are 80% of Ninth Circuit judgements overturned by the Supreme Court?", nor the first quote "The 9th Circuit Court has been overturned 80% of the time." – 410 gone Feb 10 '17 at 15:16
@felipa The supreme court only decides to hear about 80 out of 8000 request. https://www.supremecourt.gov/faq.aspx I don't have numbers for how many requests are from the 9th circuit (though about 1/4 of those heard are from the 9th circuit). So a very small percentage. – DavePhD Feb 10 '17 at 15:17
1

@EnergyNumbers an answer on skepticsSE should address the claim itself, not the title of someone's question. Only a tiny fraction of cases are heard by the Supreme Court. – DavePhD Feb 10 '17 at 15:19
24

@DavePhD well, quite - so please can you address the first cited claim "the 9th Circuit Court has been overturned 80% of the time". All you've addressed is how often they've been overturned **when an appeal has been heard**. The original claim is practising lying through half-truths; and one of the things we try to do here, is expose it when we find it; unfortunately, as it stands, your answer instead reinforces it. – 410 gone Feb 10 '17 at 15:23
@EnergyNumbers In the time period of the bar association study, 114,199 cases were decided by the 9th circuit and 107 were reversed 33 vacated and 35 affirmed. – DavePhD Feb 10 '17 at 15:26
2

@DavePhD: I think we are agree on the facts. The question has been through a couple of changes (title AND claim), so I think it is well worth emphasizing in this answer that it only includes those that are reviewed - and hence the (poorly worded) claims are false. – Oddthinking Feb 10 '17 at 15:45
2

So based on that data, the answer is "No. The 9th circuit has been overturned in 0.12% of cases. It has been affirmed in 0.03% of cases. The remaining 99.85% of cases the court's findings were left untouched." Admittedly this is independent analysis of the data, though. – Kevin Fee Feb 10 '17 at 15:50
6

@Oddthinking The term "reversal rate" is a standard term and the American Bar Association is a neutral source. I think quoting the ABA plus giving the raw numbers is the most neutral, correct answer possible. – DavePhD Feb 10 '17 at 15:53
The problem with that statistic is that it assumes that the cases that each circuit court hears are equivalent. It's possible that the cases of the 7th circuit court are routine, while the one in the 9th are ones that deal with bleeding edge legal theory that's not yet settled. Given that the 9th includes California, I bet that's the case. Also, just because the SCOTUS overturns a case does not mean that the circuit court got it wrong. – ventsyv Feb 10 '17 at 15:59
@ventsyv You are addressing a different question. This is really just a question of fact. – Simd Feb 10 '17 at 16:02
1

@felipa The question as written now asks us to interpret the meaning of those numbers so it's not just a question of fact. – ventsyv Feb 10 '17 at 16:05
18

I think a perfect answer would be "No. The 9th circuit has been overturned in 0.12% of cases. It has been affirmed in 0.03% of cases. The remaining 99.85% of cases the court's findings were left untouched." – Simd Feb 10 '17 at 16:08
3

@felipa You could always add an answer saying that, and accept your own answer. I'm sure you'll get lots of votes. It think if you search the term "reversal rate" or ask about it on lawSE, no one will define it the way you are proposing. https://www.theatlantic.com/politics/archive/2016/02/a-circuit-split/435567/ ; http://www.slate.com/articles/news_and_politics/jurisprudence/2007/07/in_defense_of_the_9th_circuit.html – DavePhD Feb 10 '17 at 16:19
I suggest making more prominent the percentage of cases heard that are reversed, which by my calculation is 0.09% – DJClayworth Feb 10 '17 at 16:31
3

@Dave Then your answer needs to define it. For a person who doesn't know exactly what it means reading the claim in the question could very easily be interpreted as 80% of all their cases are overturned. – Feb 10 '17 at 16:34
@DavePhD The first part of claim (as it now is) says nothing about reversal rates - OK the second part does say reversal rate but that is not the headline (and I suspect is not the headline anywhere else) – mmmmmm Feb 10 '17 at 16:35
@DJClayworth that would be percent of cases terminated by the 9th circuit which were reversed by the Supreme Court (not including cases vacated) – DavePhD Feb 10 '17 at 16:36
@fredsbend ok, I added that – DavePhD Feb 10 '17 at 16:39
2

@DavePhD: this still fails to 'popularise' the answer into something clear enough for someone not familiar with the terms. I don't want to edit against your will, but explaining that the intuitive reading is false is important, even if you don't want to include an invented metric. No-one should walk away under the misapprehension that the court had 80% of ALL of its decisions overturned. – Oddthinking Feb 10 '17 at 17:12
@Oddthinking See if that's more clear. – DavePhD Feb 10 '17 at 17:19
2

I upvoted because you finally added "80% reversal rate does not mean that 80% of all the 9th circuit's decisions are reversed." – Feb 10 '17 at 18:38
Congrats, Dave, you've become an authoritative source of information for answers on Skeptics.SE... or so it seems. – Mindwin Remember Monica Feb 10 '17 at 19:14
3

+1 for this answer. Explains why the figure is correct, explains what it means (and thus why it's misleading out of context), and gives a good baseline for what the reversal rates are for other courts. ("The median reversal rate is 68.29%.") So, yes the 80% reversal rate is high, but not as outrageously high as it might seem on first glance. – Ajedi32 Feb 10 '17 at 20:23
@felipa The remaining cases were not necessarily left untouched. It's not uncommon for several court cases to turn on a particular question of law. That question only needs to be answered once by the Supreme Court for the entire country, not once for every single case from every single district. So, several of the other cases could have been effectively overturned as a result of a ruling in either the challenge to another case from the 9th or a challenge to a case from another district. – reirab Feb 10 '17 at 20:51
@reirab That's a very reasonable point but perhaps too sophisticated to handle in this type of forum? – Simd Feb 11 '17 at 15:20
9

With the changes that have been made, this is clearly the most helpful answer that has been posted. It both states the context of the claim (that it's for cases heard by the Supreme Court), and it links to an article putting the 80% into context with other court circuits. After all, while it's true that some people will misinterpret the 80% number, thoughtful people want a way to compare one circuit with another, when the cases end up before the Supreme Court. Good answer! – TheGerm Feb 13 '17 at 20:06

score 29 · Answer 3 · edited Oct 20 '18 at 15:20

The original question title asked:

Are 80% of Ninth Circuit judgements overturned by the Supreme Court?

No. Asked like that, the rate is between .1% and .2%.

The quoted claim is

The 9th Circuit Court has been overturned 80% of the time.

Which would be better stated as: of the cases from the 9th Circuit that have been reviewed by the Supreme Court, 80% are overturned. This is misleading in that many people will hear the first claim. The wording is overly ambiguous.

As @DavePhD posted the "overturn" rate statistic of 80% is correct. The problem is that not all cases are reviewed. So there might be a 114,199 cases, 175 are reviewed, and 140 are overturned. 140 is 80% of 175, but it is far less than 1% of the overall cases. That rate is between .1% and .2%.

USCourts.gov says:

In fact, the Court accepts 100-150 of the more than 7,000 cases that it is asked to review each year.

Obviously 150/7500 is only 2%. 80% of that would only be 1.6%. And of course, not every case is brought to the Supreme Court for Review. Numbers just for illustration -- the actual rate may differ somewhat.

The Supreme Court says:

The Court receives approximately 7,000-8,000 petitions for a writ of certiorari each Term. The Court grants and hears oral argument in about 80 cases.

It's not clear why the difference in numbers between the two sources. Regardless, the point is that the court does not hear most of the cases where review is requested, much less most of the cases issued by the circuits.

Note that in the context of the travel pause stay, it's not as misleading as it first seems. While only a small percentage of the overall judgments are overturned, a high percentage of those requiring review are overturned. This will almost certainly require review, as there is a conflict in the district courts of different circuits. A court in Seattle issued the stay while a court in Boston ruled the actions constitutional. The only way to resolve conflicts in different circuits is for the national court to rule. The Supreme Court is the national court. The term is circuit split.

The locution used in the quotes is skipping a step. In a random case, it's a big step. In this particular case, not quite as big. But 80% is still a high estimate at this point in the process.

As a side note, one of the reasons why the 9th Circuit has a high overturn rate is that it is too large for en banc review to proceed easily. So more litigants try to get a Supreme Court review without the step of en banc review. The 9th might have better statistics if it overturned some of its three judge panels itself, like the other circuits do. There have been some recent reforms that may help with that going forward, but that's unlikely to affect this particular case.

The Supreme Court itself says 80 out of 7000-8000 https://www.supremecourt.gov/faq.aspx — DavePhD, Feb 10 '17 at 18:26
Perhaps the difference between "about 80" and "100-150" are cases where the court grants certiorari but in the same breath (that is, without argument or opinion) remands for reconsideration "in light of" a recently decided case. — hmakholm left over Monica, Feb 10 '17 at 18:54
"Note that in the context of the travel pause stay, it's not as misleading as it first seems." - Except it's not really talking about how likely this decision is to be overturned, it's passing moral judgement on the judges of the ninth circuit. — Random832, Feb 11 '17 at 20:38
"80% of reviewed cases" is a bit misleading. In addition to upholding or overturning a ruling by a circuit court, there are some other things the Supreme Court can do. They can decline to rule on the matter (essentially, saying "we shouldn't have accepted this for review"). Or, they can return the case to the circuit court for reconsideration in light of specific facts (usually functionally equivalent to overturning it, but not counted as such; occasionally a court will say "no, we got it right the first time"). I think the "80%" is 80% of cases where the court issued a conclusive ruling. — Mark, Feb 11 '17 at 21:57
Is 80% a high estimate? Given that we know which circuits are involved in the split, I think a rather good estimate could be found using the conditional probability, specifically "The probability that a law is found to be constitutional on review, after the 9th Circuit previously held it was unconstitutional and the District of Massachusetts held it was constitutional", or even better, a slight variation dealing with probabilities conditioned on temporary injunctions. — Ben Voigt, Feb 14 '17 at 01:11

score 6 · Answer 4 · answered Feb 10 '17 at 18:28

6

Yes. The cited claims are correct but the simplified/paraphrased post title is incorrect.

Both cited articles do qualify their respective "80% overturned" statements.

The first notes(with my emphasis):

If Trump brings makes his plea to the Supreme Court — with Justice Neil Gorsuch onboard — the “rogue” 9th Circuit decision will be overturned easily.

The second, quoting National Review, states:

The Ninth Circuit’s best showing in recent years was October Term 2009, with a 60 percent reversal rate in the 15 cases on which certiorari was granted.

I agree that someone not reading beyond the headlines might come to the mistaken conclusion that 80% of all of the 9th Circuit's decisions have been overturned, but we do not do that here, right?

answered Feb 10 '17 at 18:28

Michael J.

239
1
7

[Welcome to Skeptics](http://meta.skeptics.stackexchange.com/q/1505)! Please [provide some references](http://skeptics.stackexchange.com/help/how-to-answer) to support your claims. – Larian LeQuella Feb 10 '17 at 23:43
1

Yes, explaining how something is misleading is something we do here. – Oddthinking Feb 11 '17 at 05:31
The only sentence on this entire page of any value is this one: "Both cited articles do qualify their respective "80% overturned" statements." – Fattie Feb 11 '17 at 14:36
Both linked articles *exactly* misrepresent the data as is presented in the question. What you quote as qualification doesn't make the article(s) an ounce less misleading. I certainly couldn't have interpreted the numbers if not for the explanation in the good answers here. – Martin Feb 11 '17 at 21:47
`If Trump brings makes his plea to the Supreme Court ` - this implies, to my reading, that it is Trump's decision whether the court hears the case. It is the court's. Otherwise he is one of those 7-8000. – SomeoneSomewhereSupportsMonica Feb 12 '17 at 09:31
@Oddthinking Why do you think that the referenced articles are misleading? I read both of them and came away with no other conclusion that 80% of the cases _that were heard by the Supreme Court_ had been overturned. Certainly, it could have been stated more plainly, but I am not willing to judge the truth or falsehood of a statement because I _think_ it could have been stated better. – Michael J. Feb 16 '17 at 13:46
You write "I agree that someone not reading beyond the headlines might come to the mistaken conclusion that [...] but we do not do that here, right?" I read that as a bit dismissive of the important Skeptics.SE role of helping people who have come to mistaken conclusions with a shallow reading of a misleading claim. – Oddthinking Feb 16 '17 at 14:09

Has the 9th Circuit Court been overturned 80% of the time?

4 Answers4

Summary