Using numpy.genfromtxt gives TypeError: Can't convert 'bytes' object to str implicitly

Question

I have a project in python that is from kaggle.com. I am having problems reading in the data set. It has one csv file. We need to read it in and put the target and train parts of it in arrays.

Here are the first 3 rows of data set (the target column is the 19th column and the features are the first 18 columns):

user    gender  age how_tall_in_meters  weight  body_mass_index x1  
debora  Woman   46  1.62    75  28.6    -3  
debora  Woman   46  1.62    75  28.6    -3  

The target column which is not shown here has string values.

from pandas import read_csv
import numpy as np
from sklearn.linear_model.stochastic_gradient import SGDClassifier
from sklearn import preprocessing
import sklearn.metrics as metrics
from sklearn.cross_validation import train_test_split

#d = pd.read_csv("data.csv", dtype={'A': np.str(), 'B': np.str(), 'S': np.str()})

dataset = np.genfromtxt(open('data.csv','r'), delimiter=',', dtype='f8')[1:]
target = np.array([x[19] for x in dataset])
train = np.array([x[1:] for x in dataset])

print(target)

The error I'm getting is:

Traceback (most recent call last):
  File "C:\Users\Cameron\Desktop\Project - Machine learning\datafilesforproj\SGD_classifier.py", line 12, in <module>
    dataset = np.genfromtxt(open('data.csv','r'), delimiter=',', dtype='f8')[1:]
  File "C:\Python33\lib\site-packages\numpy\lib\npyio.py", line 1380, in genfromtxt
    first_values = split_line(first_line)
  File "C:\Python33\lib\site-packages\numpy\lib\_iotools.py", line 217, in _delimited_splitter
    line = line.split(self.comments)[0]
TypeError: Can't convert 'bytes' object to str implicitly

Answer 1

What worked for me was changing the line

dataset = np.genfromtxt(open('data.csv','r'), delimiter=',', dtype='f8')[1:]

to

dataset = np.genfromtxt('data.csv', delimiter=',', dtype='f8')[1:]

(unfortunately, I'm not quite sure what the underlying problem was)

Answer 2

This is in fact a bug in numpy, cf. issue #3184.

I'll just copy the workaround that I presented over there:

import functools
import io
import numpy as np
import sys

genfromtxt_old = np.genfromtxt
@functools.wraps(genfromtxt_old)
def genfromtxt_py3_fixed(f, encoding="utf-8", *args, **kwargs):
  if isinstance(f, io.TextIOBase):
    if hasattr(f, "buffer") and hasattr(f.buffer, "raw") and \
    isinstance(f.buffer.raw, io.FileIO):
      # Best case: get underlying FileIO stream (binary!) and use that
      fb = f.buffer.raw
      # Reset cursor on the underlying object to match that on wrapper
      fb.seek(f.tell())
      result = genfromtxt_old(fb, *args, **kwargs)
      # Reset cursor on wrapper to match that of the underlying object
      f.seek(fb.tell())
    else:
      # Not very good but works: Put entire contents into BytesIO object,
      # otherwise same ideas as above
      old_cursor_pos = f.tell()
      fb = io.BytesIO(bytes(f.read(), encoding=encoding))
      result = genfromtxt_old(fb, *args, **kwargs)
      f.seek(old_cursor_pos + fb.tell())
  else:
    result = genfromtxt_old(f, *args, **kwargs)
  return result

if sys.version_info >= (3,):
  np.genfromtxt = genfromtxt_py3_fixed

After putting this at the top of your code, you can just use np.genfromtxt again and it should work fine in Python 3.

Answer 3

As per https://mail.python.org/pipermail/python-list/2012-April/622487.html you probably need

import io
import sys
inpstream = io.open('data.csv','rb')
dataset = np.genfromtxt(inpstream, delimiter=',', dtype='f8')[1:]

In the examples shown in http://docs.scipy.org/doc/numpy/reference/generated/numpy.genfromtxt.html the objects used as file are of class StringIO. Nevertheless, from the specification of the function, I guess that passing the file name should work.

Answer 4

You need to decode the bytes object to a str object explicitly, as the TypeError implies.

# For instance, interpret as UTF-8 (depends on your source)
self.comments = self.comments.decode('utf-8')

Answer 5

Instead of:

dataset = np.genfromtxt(open('data.csv','r'), delimiter=',', dtype='f8')[1:]

try this:

dataset = np.genfromtxt('C:\\\\..\\\\..\\\train.csv', delimiter=',', dtype='None')[1:]

Note that you have to use an extra '\' to escape the other.