Causal inference - IPTW vs nearest neighbour matching

Question

I am doing a quasi experimentation and am interested in getting ATT. i have a data with 260k entries where Ti = 0 and 5k entries where Ti = 1. I am calculating ATT with iptw technique where I achieve a great balance and the treatment effect -on treated as -ve 450 euros but not significant.

Weight calculation: (If treatment = 1, weight = 1 else propensity score / (1-propensity score)

Then, to compare against other methodology, I use nearest neighbour matching with ratio = 1, the balance is again achieved. I get treatment effect (which is ATT by default in matching) as +very 750 and significant.

Shouldn't both method generate similar result ? Which method should I go for in this case and why?

Answer 1

When you match, is there any treated individuals without a match?

In expectation, IPTW and matching should both give the same answer. One possible explanation is that some treated individuals don't have a close match, so they are dropped. When this happens, the population for which the causal effect is defined changes. This could results in different answer between the methods

Answer 2

Each of these methods needs to be evaluated differently.

For IPW you need to check that you did not get samples with extremely low (or extremely high) propensity. If they are close to 0 or 1 then you need to evaluate why this happened and probably remove samples like that from the data. Since your labels are very unbalanced this could certainly happen.

For matching, like @pzivich said, you need to check if there are samples that did not get matched (similar to a very low propensity)

Finally, I like checking the balancing on held-out data to check that there is no over-fitting.