33

I have a csv file that contains some data with columns names:

  • "PERIODE"
  • "IAS_brut"
  • "IAS_lissé"
  • "Incidence_Sentinelles"

I have a problem with the third one "IAS_lissé" which is misinterpreted by pd.read_csv() method and returned as �.

What is that character?

Because it's generating a bug in my flask application, is there a way to read that column in an other way without modifying the file?

In [1]: import pandas as pd

In [2]: pd.read_csv("Openhealth_S-Grippal.csv",delimiter=";").columns

Out[2]: Index([u'PERIODE', u'IAS_brut', u'IAS_liss�', u'Incidence_Sentinelles'], dtype='object')
farhawa
  • 10,120
  • 16
  • 49
  • 91
  • Looks like Pandas can't handle unicode characters in the column names. Try converting the column names to ascii. Note that you'll lose the accent. – Sohier Dane Sep 23 '16 at 00:07
  • The comment above is not true and wasn't true as of its posting - see any of the answers below for the proper way to handle non-ASCII (generally by setting encoding to utf-8 or latin1). – bsplosion Aug 06 '20 at 15:37

4 Answers4

28

I found the same problem with spanish, solved it with with "latin1" encoding:

import pandas as pd

 pd.read_csv("Openhealth_S-Grippal.csv",delimiter=";", encoding='latin1')

Hope it helps!

Francisco del Valle Bas
  • 474
  • 4
  • 5
13

You can change the encoding parameter for read_csv, see the pandas doc here. Also the python standard encodings are here.

I believe for your example you can use the utf-8 encoding (assuming that your language is French). 

df = pd.read_csv("Openhealth_S-Grippal.csv", delimiter=";", encoding='utf-8')

Here's an example showing some sample output. All I did was make a csv file with one column, using the problem characters.

df = pd.read_csv('sample.csv', encoding='utf-8')

Output:

    IAS_lissé
0   1
1   2
2   3
shawnheide
  • 797
  • 4
  • 11
  • Oups ! I got an error ( the same one ) `'utf8' codec can't decode byte 0xe9 in position 8: unexpected end of data` – farhawa Sep 23 '16 at 07:28
  • 11
    That is because your data is not encoded to `utf-8`. Try `latin1`: `pd.read_csv("Openhealth_S-Grippal.csv", delimiter=";", encoding='latin1')`... – Kartik Sep 23 '16 at 07:33
  • Yeah, `latin1` is what I had at first, but changed it to `utf-8`. @farhawa, if you want a better answer you can post your csv or a sample of it with the header so that we know what your encoding is. – shawnheide Sep 23 '16 at 20:15
11

Try using:

import pandas as pd    
df = pd.read_csv('file_name.csv', encoding='utf-8-sig')
Gil Baggio
  • 13,019
  • 3
  • 48
  • 37
  • 1
    This ended up working for me. UTF-8 wasn't throwing an error - but it was turning "é" into "é". latin1 didn't work - it threw an error on "ś". – jeffsdata Sep 20 '21 at 20:59
4

Using utf-8 didn't work for me. E.g. this piece of code: 

    bla = pd.DataFrame(data = [1, 2])
    bla.to_csv('funkyNamé , things.csv')
    blabla = pd.read_csv('funkyNamé , things.csv', delimiter=";", encoding='utf-8')
    blabla

Ultimately returned: OSError: Initializing from file failed

I know you said you didn't want to modify the file. If you meant the file content vs the filename, I would rename the file to something without an accent, read the csv file under its new name, then reset the filename back to its original name. 

    originalfilepath = r'C:\Users\myself\\funkyNamé , things.csv'
    originalfolder = r'C:\Users\myself'
    os.rename(originalfilepath, originalFolder+"\\tempName.csv")
    df = pd.read_csv(originalFolder+"\\tempName.csv", encoding='ISO-8859-1')
    os.rename(originalFolder+"\\tempName.csv", originalfilepath)

If you did mean "without modifying the filename, my apologies for not being helpful to you, and I hope this helps someone else.

pantherenoire
  • 49
  • 3