Python warning confusion

Question

I have the following code which correlates data brought in from PgSQL.

if wd is not None and dd is not None:
    alldata=np.concatenate((wd,dd))
    alldat_df=pd.DataFrame(alldata, index=None, columns=['datetime','rain', 'raindiff'])

    alldat_df.drop(alldat_df.loc[2708:2738].index, inplace=True)
    alldata=np.array(alldat_df)
    alldata[0,2]=0
    mask = (alldat_df['datetime'] > fdate) & (alldat_df['datetime'] <= tdate)
    ndf=alldat_df.loc[mask]
    ndf.loc[0,['raindiff']]=0
    ndf.index=ndf['datetime']
    ndf.drop(columns=['datetime'], inplace=True)
    davisdfnew=ndf.resample(bs, offset=bs, origin=fdate).sum()
    davisdfnew.rename(columns={'rain':'rain sum','raindiff':'raindiff sum'}, inplace=True)
   

if dd is None:
    alldat_df=pd.DataFrame(wd, index=None, columns=['datetime', 'rain', 'raindiff'])
    mask = (alldat_df['datetime'] > fdate) & (alldat_df['datetime'] <= tdate)
    ndf=alldat_df.loc[mask]
    ndf.loc[0,['raindiff']]=0
    ndf.index=ndf['datetime']
    ndf.drop(columns=['datetime'], inplace=True)
    davisdfnew=ndf.resample(bs, offset=bs, origin=fdate).sum()
    davisdfnew.rename(columns={'rain':'rain sum','raindiff':'raindiff sum'}, inplace=True)
    

if wd is None:
    alldat_df=pd.DataFrame(dd, index=None, columns=['datetime', 'rain', 'raindiff'])
    mask = (alldat_df['datetime'] > fdate) & (alldat_df['datetime'] <= tdate)
    ndf=alldat_df.loc[mask]
    ndf.loc[0,['raindiff']]=0
    ndf.index=ndf['datetime']
    ndf.drop(columns=['datetime'], inplace=True)
    davisdfnew=ndf.resample(bs, offset=bs, origin=fdate).sum()
    davisdfnew.rename(columns={'rain':'rain sum','raindiff':'raindiff sum'}, inplace=True)

When it runs and the first two if conditions are met it throws the following warning

SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  ndf.loc[0,['raindiff']]=0

but when the condition if wd is None is met there is no warning in all cases the value at ndf.loc[0,['raindif']] is a none type object

I would appreciate it if someone could shed some light on this!

edited as per @john giorgio comment

wd=

array([[datetime.datetime(2021, 5, 20, 10, 45), 0.0, None],
       [datetime.datetime(2021, 5, 20, 11, 0), 0.0, 0.0],
       [datetime.datetime(2021, 5, 20, 11, 15), 0.0, 0.0],
       ...,
       [datetime.datetime(2021, 6, 17, 22, 30), 96.6, 0.0],
       [datetime.datetime(2021, 6, 17, 22, 45), 96.6, 0.0],
       [datetime.datetime(2021, 6, 17, 23, 0), 96.6, 0.0]], dtype=object)

dd=

array([[datetime.datetime(2021, 6, 17, 15, 30, 42), 96.6, None],
       [datetime.datetime(2021, 6, 17, 15, 35, 42), 96.6, 0.0],
       [datetime.datetime(2021, 6, 17, 15, 40, 42), 96.6, 0.0],
       ...,
       [datetime.datetime(2021, 6, 30, 23, 45, 41), 113.8, 0.0],
       [datetime.datetime(2021, 6, 30, 23, 50, 41), 113.8, 0.0],
       [datetime.datetime(2021, 6, 30, 23, 55, 41), 113.8, 0.0]],
      dtype=object)

as I said, the error occurs when wd exists. If both wd and dd exist they are combined, and duplicate datetimes removed to give ndf. if only wd exists ndf is formed from it, in both these cases the error occurs.

If only dd exists ndf is formed from this, and the error does not occur

Does this answer your question? [A value is trying to be set on a copy of a slice from a DataFrame-warning even after using .loc](https://stackoverflow.com/questions/29888341/a-value-is-trying-to-be-set-on-a-copy-of-a-slice-from-a-dataframe-warning-even-a) — JeffUK, Feb 11 '22 at 11:10
@JeffUK, not really, as ndf is created in the same way in each if statement, but the warning only comes up when the condition 'if wd is none' is met — matt cooper, Feb 11 '22 at 12:23

score 1 · Answer 1 · answered Feb 11 '22 at 18:37

1

What you could try to do is resetting the index, .reset_index(drop=True) each time you take a subsample of your original dataset, before performing any other action.

answered Feb 11 '22 at 18:37

John Giorgio

634
3
10

This is not an answer, as it does not explain why the warning occurs in two conditions, but not the third. Plus I cannot reset the index as the index is the datetime value on which the .resample method is performed on. – matt cooper Feb 12 '22 at 17:06
Sure, then please give us examples of wd and dd dataframes – John Giorgio Feb 12 '22 at 19:08

Python warning confusion

1 Answers1