Pandas read_csv() gives DtypeWarning

Question

I created a .csv file from a dataframe as below:

df.to_csv('partial.csv', sep=',')

Datatypes in the dataframe

df.dtypes gives:

Contact_ID      int64
Src_Sys_Cd     object
First_Name     object
Last_Name      object
Src_Sys_Key    object
Full_Name      object
Office_No      object
Mobile         object
Email          object
dtype: object

When I try to read the newly created .csv file using read_csvit gives me error:

new_df =  pd.read_csv('partial.csv')

DtypeWarning: Columns (5) have mixed types. Specify dtype option on import or set low_memory=False. interactivity=interactivity, compiler=compiler, result=result)

How can I avoid this error? Is this error coming up because I did something wrong while to_csv or read_csv ?

score 6 · Accepted Answer · edited Feb 15 '18 at 09:27

6

Please give the below a shot. It might work well,

new_df = pd.read_csv('partial.csv', low_memory=False)

edited Feb 15 '18 at 09:27

Martin Valgur

5,793
1
33
45

answered Feb 15 '18 at 07:14

MuraliSunil

126
2

6

What does `low_memory = False` do? – Tom J Muthirenthi Feb 15 '18 at 08:48
2

@TomJMuthirenthi from the documentation `Internally process the file in chunks, resulting in lower memory use while parsing, but possibly mixed type inference. To ensure no mixed types either set False, or specify the type with the dtype parameter. Note that the entire file is read into a single DataFrame regardless, use the chunksize or iterator parameter to return the data in chunks. (Only valid with C parser) ` – alex Aug 27 '20 at 20:10

score 3 · Answer 2 · answered Feb 08 '18 at 17:43

3

What are the data types of your original columns? You can try specifying the data types in read_csv by putting an argument dtype:

types = {‘your_col_name01’: your_dtype01, ‘your_col_name02’: your_dtype02}
new_df = pd.read_csv('partial.csv', dtype=types)

answered Feb 08 '18 at 17:43

idchiang

71
5

I have added the datatypes in the question – Tom J Muthirenthi Feb 08 '18 at 17:48

score 2 · Answer 3 · edited Mar 02 '22 at 20:35

2

pd.read_csv('input.csv', engine='python')

does the trick for me.

edited Mar 02 '22 at 20:35

Henry Ecker

34,399
18
41
57

answered Mar 02 '22 at 03:35

Ka Wa Yip

2,546
3
22
35

Pandas read_csv() gives DtypeWarning

3 Answers3