Importing csv embedding special character with numpy genfromtxt

Question

I have a CSV containing special characters. Some cells are arithmetic operations (like "(10/2)"). I would like to import these cells as string in numpy by using np.genfromtxt. What I notice is that it actually import them in UTF8 (if I understood). For instance everytime I have a division symbol I get this code in the numpy array :\xc3\xb7

How could I import these arithmetic operations as readable string?

Thank you!

[According to the docs](https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.genfromtxt.html), `np.genfromtxt`'s `encoding` parameter defaults to `"bytes"`, which "enables backward compatibility workarounds that ensure that you receive byte arrays when possible and passes latin1 encoded strings to converters." Since you have UTF-8 text, I would guess that setting this parameter to `"utf8"` would produce the output you expect. — Jordan Running, Apr 16 '18 at 14:46

hpaulj · Answer 1 · 2018-04-16T22:51:13.433

Looks like the file may have the 'other' divide symbol, the one we learn in grade school:

In [185]: b'\xc3\xb7'
Out[185]: b'\xc3\xb7'
In [186]: _.decode()
Out[186]: '÷'

Recent numpy version(s) handle encoding better. Earlier ones tried to work entirely in bytestring mode (for Py3) to be compatible with Py2. But now it takes an encoding parameter.

In [68]: txt = '''(10/2), 1, 2
    ...: (10/2), 3,4'''

In [70]: np.genfromtxt(txt.splitlines(), dtype=None, delimiter=',')
/usr/local/bin/ipython3:1: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
  #!/usr/bin/python3
Out[70]: 
array([(b'(10/2)', 1, 2), (b'(10/2)', 3, 4)],
      dtype=[('f0', 'S6'), ('f1', '<i8'), ('f2', '<i8')])

In [71]: np.genfromtxt(txt.splitlines(), dtype=None, delimiter=',',encoding=None
    ...: )
Out[71]: 
array([('(10/2)', 1, 2), ('(10/2)', 3, 4)],
      dtype=[('f0', '<U6'), ('f1', '<i8'), ('f2', '<i8')])

Admittedly this simulated load from a list of strings is not the same as loading from a file. I don't have earlier numpys installed (and not on Py2), so can't show what happened before. But my gut feeling is that "(10/2)" shouldn't have given problems before, at least not in an ASCII file. There aren't any special characters in the string.

With the other divide:

In [192]: txt = '''(10÷2), 1, 2
     ...: (10÷2), 3,4'''
In [194]: np.genfromtxt(txt.splitlines(), dtype=None, delimiter=',',encoding='ut
     ...: f8')
Out[194]: 
array([('(10÷2)', 1, 2), ('(10÷2)', 3, 4)],
      dtype=[('f0', '<U6'), ('f1', '<i8'), ('f2', '<i8')])

Same thing in a file:

In [200]: np.genfromtxt('stack49859957.txt', dtype=None, delimiter=',')
/usr/local/bin/ipython3:1: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
  #!/usr/bin/python3
Out[200]: 
array([(b'(10\xf72)', 1, 2), (b'(10\xf72)', 3, 4)],
      dtype=[('f0', 'S6'), ('f1', '<i8'), ('f2', '<i8')])

In [199]: np.genfromtxt('stack49859957.txt', dtype=None, delimiter=',',encoding=
     ...: 'utf8')
Out[199]: 
array([('(10÷2)', 1, 2), ('(10÷2)', 3, 4)],
      dtype=[('f0', '<U6'), ('f1', '<i8'), ('f2', '<i8')])

In earlier versions, encoding could be implemented in a converter. I've helped with that task in previous SO questions.

Importing csv embedding special character with numpy genfromtxt

1 Answers1

Linked