Get 'TypeError: data type not understood' When Replacing Zeros With Previous Non-zero Values in Pandas

Question

I get 'TypeError: data type not understood' when trying to execute a line of code that looks like this:

df['c'].replace(0, method='ffill', inplace=True)

The code above is basically to replace every zero value with its previous non-zero value in column c. The data type of values in column c is integer with only two NaNs. The error still occurs after replacing the NaNs with zeros, but it works fine when I add the following line before the above code:

df.dropna(inplace = True)

So what does actually happen here? How can I replace every zero value with its previous non-zero value in column c without deleting any row with NaNs in my DataFrame?

Can you post some portion of dataframe representing the input & also expected output? — moys, Sep 11 '19 at 04:50
@SH-SF Basically it looks like [this](https://stackoverflow.com/q/13445241/3397819). — Anastasiya-Romanova 秀, Sep 11 '19 at 04:54
@Vishnudev's answer replaces not only your zeros with the previous value, but also the other NaNs. Is this desired? — Itamar Mushkin, Sep 11 '19 at 04:59
I think handling `NaN` should be done beforehand rather than persisting it. What I am expecting is a non-NaN column. — Vishnudev Krishnadas, Sep 11 '19 at 05:01
@Trenton_M Yes, it does work. But it does not answer the 'why' yet. And I prefer to wait for a moment before accepting an answer. — Anastasiya-Romanova 秀, Sep 11 '19 at 05:10
There's no way to really answer why, unless you post all your real data. — Trenton McKinney, Sep 11 '19 at 05:11

Vishnudev Krishnadas · Accepted Answer · 2019-09-11T05:04:27.223

2

Assuming all the missing data are handled i.e. NaNs replaced with some desired value,

some_desired_value = 0 # Maybe different in your case
df['c'] = df['c'].fillna(some_desired_value)

Replace zeros with NaN and then forward-fill

df['c'].replace(0, pd.np.nan).ffill()

Even your code should work if there are no undesired value in column

df['c'].replace(0, method='ffill', inplace=True)

edited Sep 11 '19 at 05:04

answered Sep 11 '19 at 04:57

Vishnudev Krishnadas

10,679
2
23
55

This works fine. But why did the error occur using my code? – Anastasiya-Romanova 秀 Sep 11 '19 at 05:08
As I said there might be some undesired value in the column. Maybe it's not visually seen. It may be some non-ASCII character. You'll need to evaluate the data more carefully for this. – Vishnudev Krishnadas Sep 11 '19 at 05:10
Also, try using `pd.to_numeric` and see if raises an error. – Vishnudev Krishnadas Sep 11 '19 at 05:12
I use dtype 'Int32' so that it can handle the NaNs – Anastasiya-Romanova 秀 Sep 11 '19 at 05:13
What do you mean? Did you use `astype`? – Vishnudev Krishnadas Sep 11 '19 at 05:14
Basically I use this `df['c'] = df.groupby(...).diff().fillna(...).astype('timedelta64[D]').astype('Int32')` when trying to create `df['c']`. Afterward, I use the above code. – Anastasiya-Romanova 秀 Sep 11 '19 at 05:17
You cant aggregate and assign to same dataframe as the size will be different – Vishnudev Krishnadas Sep 11 '19 at 05:19
I think you have different issue. That's a totally different question. For now, I am ending it here – Vishnudev Krishnadas Sep 11 '19 at 05:20

Get 'TypeError: data type not understood' When Replacing Zeros With Previous Non-zero Values in Pandas

1 Answers1