176

I have table x:

        website
0   http://www.google.com/
1   http://www.yahoo.com
2   None

I want to replace python None with pandas NaN. I tried:

x.replace(to_replace=None, value=np.nan)

But I got:

TypeError: 'regex' must be a string or a compiled regular expression or a list or dict of strings or regular expressions, you passed a 'bool'

How should I go about it?

smci
  • 32,567
  • 20
  • 113
  • 146
AdamNYC
  • 19,887
  • 29
  • 98
  • 154

7 Answers7

252

You can use DataFrame.fillna or Series.fillna which will replace the Python object None, not the string 'None'.

import pandas as pd
import numpy as np

For dataframe:

df = df.fillna(value=np.nan)

For column or series:

df.mycol.fillna(value=np.nan, inplace=True)
Guillaume Jacquenot
  • 11,217
  • 6
  • 43
  • 49
  • 5
    If you imported data from an SQL database, you can combine this with the answer below. This converts None (which isn't a string) to NaN. Then you can `df['column'].replace(nan, "", inplace=True)` if say you wanted None to be empty string. – VISQL Jul 04 '18 at 12:52
  • 1
    This does answer does not work for me; it does not replace `None`. Max's answer works. – Daniel Oct 11 '18 at 08:14
  • I found this column-specific solution to be the most effective: `df['website'].replace(pd.np.nan, 0, inplace=True)`. It also does not require Numpy to be included, relying on Pandas' inbuilt reference. – CodeMantle Dec 28 '19 at 18:29
  • 2
    `pd.np.nan` now gives `FutureWarning: The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead`. – Alex Povel Jun 22 '20 at 17:14
32

Here's another option:

df.replace(to_replace=[None], value=np.nan, inplace=True)
Nickolai
  • 1,579
  • 2
  • 12
  • 19
  • 3
    Please beware when you run df.replace([None], np.nan, inplace=True), this changed all datetime objects with missing data to object dtypes. So now you may have broken queries unless you change them back to datetime which can be taxing depending on the size of your data. – Doubledown Oct 02 '19 at 16:54
  • Do you know why the brackets are required? Leaving them away does not work. But I don't get that from the documentation. – Make42 Jul 02 '21 at 08:02
26

The following line replaces None with NaN:

df['column'].replace('None', np.nan, inplace=True)
Maz Izadi
  • 317
  • 3
  • 3
  • Just double-checked it, it does work for me. Do you get any errors or the 'None' values don't get replaced? – Maz Izadi Apr 30 '18 at 00:35
  • 1
    NB: this method uses `np.nan`, which has a float dtype (e.g.: `float64`), as opposed to pandas's default dtype of `object` for a nan column. – tehfink Jun 20 '18 at 11:37
  • 18
    Be aware: This replaces strings with the text "None", but not the explicit None values ([None as in the constant](https://docs.python.org/3/library/constants.html#None)). – Gregor Müllegger Jul 26 '19 at 08:02
8

If you use df.replace([None], np.nan, inplace=True), this changed all datetime objects with missing data to object dtypes. So now you may have broken queries unless you change them back to datetime which can be taxing depending on the size of your data.

If you want to use this method, you can first identify the object dtype fields in your df and then replace the None:

obj_columns = list(df.select_dtypes(include=['object']).columns.values)
df[obj_columns] = df[obj_columns].replace([None], np.nan)
Doubledown
  • 458
  • 1
  • 6
  • 13
6

This solution is straightforward because can replace the value in all the columns easily.
You can use a dict:

import pandas as pd
import numpy as np

df = pd.DataFrame([[None, None], [None, None]])
print(df)
      0     1
0  None  None
1  None  None

# replacing
df = df.replace({None: np.nan})
print(df)
    0   1
0 NaN NaN
1 NaN NaN
igorkf
  • 3,159
  • 2
  • 22
  • 31
1

Its an old question but here is a solution for multiple columns:

values = {'col_A': 0, 'col_B': 0, 'col_C': 0, 'col_D': 0}
df.fillna(value=values, inplace=True)

For more options, check the docs:

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.fillna.html

Mitzi
  • 2,652
  • 1
  • 20
  • 15
0
DataFrame['Col_name'].replace("None", np.nan, inplace=True)
4b0
  • 21,981
  • 30
  • 95
  • 142
Ashish Sharma
  • 9
  • 1
  • 1
    Hi and welcome to stackoverflow, and thank you for answering. While this code might answer the question, can you consider adding some explanation for what the problem was you solved, and how you solved it? This will help future readers to understand your answer better and learn from it. – Plutian Feb 07 '20 at 10:16
  • 1
    @Plutian This actually doesn't answer the question. The question asks about None, this only works for the string "None". It's also duplicating a previous answer so not exactly adding any value. – DU_ds Jan 18 '21 at 22:12