1

I can't for the life of me figure out why something that used to be so simple no longer works. Often I might map a dataframe column to a dictionary and have some null values appear as they aren't found in the dictionary keys. So the resulting column will be floats + null. Typically I convert .astype("Int64") and boob, the non-nulls are now ints and not floats, with everything else untouched.

Now I'm running into issues where, I treat my data, use Int64 conversions, acceptance tests are passed, yet later down the road in the pipeline data deployment fails because floats are found in these columns.

Just to make sure I'm not insane, I open jupyter notebook, and initializew a basic dataframe, map it to a dictionary for which some dataframe values don't exist in dictionary keys, then cast as "Int64".....and I still gert this issue!! What's going on? I'm sure this used to be so simple....

df = pd.DataFrame({"keys": [5, 10, 15, 20]})

df["after_mapping"] = df["keys"].map({1: 0, 2: 2, 5: 25, 15: 305})

df["after_mapping"] = df["after_mapping"].astype("Int64")

ValueError: Cannot convert non-finite values (NA or inf) to integer

gosuto
  • 5,422
  • 6
  • 36
  • 57
Armon
  • 113
  • 6

1 Answers1

0

Works fine on my machine:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 4 entries, 0 to 3
Data columns (total 2 columns):
 #   Column         Non-Null Count  Dtype
---  ------         --------------  -----
 0   keys           4 non-null      int64
 1   after_mapping  2 non-null      Int64
dtypes: Int64(1), int64(1)
memory usage: 196.0 bytes

Are you sure about your version number?

pd.__version__
gosuto
  • 5,422
  • 6
  • 36
  • 57
  • You're absolutely right. I guess that takes me back to my deeper question: why am I successfully casting as Int64, writing to csv, then running into issues later.... I'll go ahead and close this question for now, but I might be back very soon :) – Armon Apr 22 '20 at 06:46