2

It is common to want to append the results of predictions to the dataset used to make the predictions, but the statsmodels predict function returns (non-indexed) results of a potentially different length than the dataset on which predictions are based.

For example, if the test dataset, test, contains any null entries, then

mod_fit = sm.Logit.from_formula('Y ~ A B C', train).fit()
press = mod_fit.predict(test)

will produce an array that is shorter than the length of test, and cannot be usefully appended with

test['preds'] = preds

And since the result of predict is not indexed, there is no way to recover the rows to which the results should be attached.

What is the idiom for associating predict results to the rows from which they were generated? Is there, perhaps, a way to get predict to return a dataframe that preserves the indices of its argument?

orome
  • 45,163
  • 57
  • 202
  • 418
  • Why does `predict` even work this way? Why not return a dataframe with indices that match those of the rows from which the predictions are made? – orome Mar 22 '14 at 16:47
  • what version of statsmodels are you using? – behzad.nouri Mar 22 '14 at 17:44
  • 1
    I tried with `0.6.0.dev`, and there though you do not get a dataframe back, missing values are not dropped from predict output. – behzad.nouri Mar 22 '14 at 18:08
  • @behzad.nouri: So in `0.6.0.dev` `len(train) == len(preds)`, regardless of missing values in `train`? What is returned in `press` where there are missing values in `train`? – orome Mar 22 '14 at 20:59

1 Answers1

2

Predict shouldn't drop any rows. Can you post a minimal working example where this happens? Preserving the pandas index is on my radar and should be fixed in master soon.

https://github.com/statsmodels/statsmodels/issues/1501

Edit: Nevermind. This is a known issue. https://github.com/statsmodels/statsmodels/issues/1352

jseabold
  • 7,903
  • 2
  • 39
  • 53
  • Cool. As long as it's known and not some part of the philosophy that I wasn't grasping. I look forward to the update. Any thoughts on timeframe? – orome Mar 22 '14 at 21:32