Using converter function in genfromtxt fails

Question

When I try to read a space separated file using genfromtxt and using a converter function to convert numbers with a comma as decimal separator, I get a type error. It seems there is something wrong with my converter function. However, when I use it on a single value, it does work properly.

This is my code (I'm using Matplotlib/Pylab):

t = dtype([('Date', 'U12'), ('Time', 'U10'), ('Cond', 'f4'), ('Temp', 'f4')])

conv = lambda valstr: float(valstr.replace(',','.'))

c = {2:conv, 3:conv}

data = genfromtxt('Example.csv', dtype = t,
    skip_header=1, delimiter = ' ', converters = c)

The data looks like this:

Date Time Cond Temp
11-10-2012 00:00:14 5,430583 29,5107
11-10-2012 00:00:15 5,431812 29,45066
11-10-2012 00:00:16 5,435501 29,43862
11-10-2012 00:00:17 5,436732 29,43862
...

And this is part of the error message:

TypeError                                 Traceback (most recent call last)
<ipython-input-41-c65c2d17c55d> in <module>()
      5 c = {2:conv, 3:conv}
      6 
----> 7 data = genfromtxt('Example.csv', dtype = t, skip_header=1, delimiter = ' ', converters = c)


...


<ipython-input-41-c65c2d17c55d> in <lambda>(valstr)
      1 t = dtype([('Date', 'U12'), ('Time', 'U10'), ('Cond', 'f4'), ('Temp', 'f4')])
      2 
----> 3 conv = lambda valstr: float(valstr.replace(',','.'))
      4 
      5 c = {2:conv, 3:conv}

TypeError: expected an object with the buffer interface

Am I doing something wrong here, or is this some kind of bug in genfromtxt?

I'am using Python 3.2 on Win7 x64. Numpy version is 1.6.2.

FWIW, your code works fine with Python 2.7.3 in Ubuntu 12.04 (64 bit). — Warren Weckesser, Oct 13 '12 at 04:16
After googling around a bit, this specific type error seems to occur when a unicode string function is being applied on a byte string. A small test confirms this: b'test'.replace('t','r') gives the exact same error. Seems to be a Py3-conversion issue and that's probably why my code is working on your Py2 configuration. — Puggie, Oct 15 '12 at 08:29

score 4 · Accepted Answer · answered Oct 15 '12 at 08:39

Apparently, genfromtxt feeds read columns to converter functions as byte strings, not as unicode strings.

The problem was solved for me by changing the code of the converter function as follows:

conv = lambda valstr: float(valstr.decode("utf-8").replace(',','.'))

Using converter function in genfromtxt fails

1 Answers1