1

I am not sure how to interpret the confidence interval obtained when using the CausalImpact function in the CausalImpact R package.

I am confused because I think there is a contradiction - the model is returning a very low p-value (0.009) which indicates that there is a casual effect, and yet the "actual" line (the solid line) appears to be well within the 95% confidence band of the counterfactual. If there was a causal impact, wouldn't you expect the line to be outside the blue band?

These are my results:

graphs

and here are the model summary results (my apologies for the large text) summary

What's happening here?

Tom Roth
  • 1,954
  • 17
  • 25
  • 1
    I just want to add the package actually defines it to be a [credible (central) interval](https://en.wikipedia.org/wiki/Credible_interval), a not confidence interval. In Bayesian context, it means 95% of the probability density lies within that interval, which contrasts with the interpretation of a confidence interval. – Booley Jan 29 '18 at 18:54

1 Answers1

4

The two results answer different questions.

  • The plot shows daily effects. The fact that the CIs contain zero means that the effect wasn't significant on any day by itself.

  • The table shows overall effects. Unlike the plot, the table pools information over time, which increases statistical power. The fact that effects were consistently negative throughout the post-period provides evidence that, overall, there probably was a negative effect. It's just too subtle to show up on any day by itself.

A side note: There seems to be a strong dip in the gap between pre- and post-period. You may want to be extra careful here and think about whether the effect in the post-period could have been caused by whatever happened in the gap rather than by the treatment.

Kay Brodersen
  • 227
  • 2
  • 2