2

How can one downcast columns with nullable integers in pandas DataFrames?

# input DataFrame
import pandas as pd, numpy as np
d = pd.DataFrame({'x':[1,3]}, dtype=np.int64)
d['y'] = pd.Series([2,np.nan], dtype=pd.Int64Dtype())

# downcasting 'x' works
d['x'] = pd.to_numeric(d['x'], downcast='unsigned')

# downcasting 'y' does not work
d['y'] = pd.to_numeric(d['y'], downcast='unsigned')
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
#   File ".../Python-3.6.3/lib/python3.6/site-packages/pandas/core/tools/numeric.py", line 165, in to_numeric
#     elif downcast == "unsigned" and np.min(values) >= 0:
#   File ".../Python-3.6.3/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 2618, in amin
#     initial=initial)
#   File ".../Python-3.6.3/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction
#     return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
#   File ".../Python-3.6.3/lib/python3.6/site-packages/pandas/core/arrays/integer.py", line 375, in __array_ufunc__
#     raise NotImplementedError("The 'reduce' method is not supported.")
# NotImplementedError: The 'reduce' method is not supported.

Thank you for your help!

S.V
  • 2,149
  • 2
  • 18
  • 41
  • Though this may not be the most satisfactory answer, you could always convert this column to a float. For example, `pd.to_numeric(d['y'].astype(np.float32), downcast='unsigned')`. – lmo Sep 03 '19 at 22:47
  • @Imo Thank you for your reply, but this is not the answer which would solve my problem. I need integers to stay integers. In one of my data sets, most of the columns are integers and many of them suffice to be represented by 8 or 16 bits. So, the goal of down-casting is to: 1) be safe when I do groupby by the columns (which is not the case with floats), 2) save memory. I guess, I will have to do it manually, i.e. `d['y'] = d['y'].astype(pd.UInt8Dtype())` – S.V Sep 04 '19 at 13:18
  • for pandas 1.2.0 this doesn't throw any error but `d.y` remains `Int64` – Stef Jan 02 '21 at 16:42

0 Answers0