Appending arrays to dataframe (python)

Question

So I ran a time series model on a small sales data set, and forecasted sales for next 12 periods. With the following code:

 mod1=ARIMA(df1, order=(2,1,1)).fit(disp=0,transparams=True)
    y_future=mod1.forecast(steps=12)[0]

where df1 contains the sales values with months being the index. Now I'm storing the predicted values in the following manner:

pred.append(y_future)

Now, I need to append the forecasted values to the original dataset df1, preferably with the same index. I'm trying to use the following code:

df1.append(pred, ignore_index=False)

But I'm getting the following error:

TypeError: cannot concatenate a non-NDFrame object

I've tried converting pred variable to list and then appending, but to no avail. Any help will be appreciated. Thanks.

The problem here seems to be that your pred is not if the df type but probably of the list type. What you need to do is convert pred to dataframe. Something like this `new_pred = pd.DataFrame(pred)` — Imre_G, Jan 24 '18 at 10:57
@Pratik Kumar, no I'd like to append it as rows to the existing dataframe. — IndigoChild, Jan 24 '18 at 11:08
@Imre_G, thanks I've tried that but its not getting appended as rows. — IndigoChild, Jan 24 '18 at 11:10
@Vivek getting this error: TypeError: Index does not support mutable operations — IndigoChild, Jan 24 '18 at 12:47

score 16 · Answer 1 · answered Jan 24 '18 at 11:09

16

One solution could be appending the new array to your dataFrame to the last position using df.loc

df.loc[len(df)] = your_array

But this is not efficient cause if you want to do it several times, it will have to get the length of the DataFrame for each new append.

A better solution would be to create a dictionary of the values that you need to append and append it to the dataFrame.

df = df.append(dict(zip(df.columns, your_array)), ignore_index=True)

answered Jan 24 '18 at 11:09

saloua

2,433
4
27
37

hi@saloua thanks for this, but i wanted to append the forecasted values as rows to the original sales dataframe, the method you've stated would probably hold for columns I believe. – IndigoChild Jan 24 '18 at 11:16

score 3 · Answer 2 · answered Dec 12 '18 at 23:14

You can append your results into a dictionary list and then append that dictionary list to data frame.

Let's assume that you want to append your ARIMA forecasted results to the end of actual data frame with two columns "datetime" (YYYY-MM-DD) and "value" respectively.

Steps to follow

First find the max day in datetime column of your actual data frame and convert it to datetime. We want to assign future dates for our forecasting results.
Create an empty dictionary list and inside a loop fill it by incrementing datetime value 1 day and place a forecasted result subsequently.
Append that dictionary list to your dataframe. Don't forget to reassign it to itself as left hand value since append function creates a copy of appended results data frame.
Reindex your data frame.

Code

lastDay = dfActualData[dfActualData['datetime'] == dfActualData['datetime'].max()].values[0][0]
dtLastDay = lastDay.to_pydatetime("%Y-%m-%d")

listdict = []
for i in range(len(results)):
    forecastedDate = dtLastDay + timedelta(days = i + 1)
    listdict.append({'datetime':forecastedDate , 'value':results[i]})

dfActualData= dfActualData.append(listdict, ignore_index=True)
dfActualData.reset_index(drop=True)

Appending arrays to dataframe (python)

2 Answers2

Steps to follow

Code

Linked