Loading data file with too many commas in Python

Question

I am trying to collect some data from a .txt file into my python script. The problem is that when the data was collected, it could not collect data in one of the columns, which has given me more commas than normally. It looks like this:

0,0,,-2235
1,100,,-2209
2,200,,-2209

All I want is to load the data and remove the commas but when I try with numpy.loadtxt it gives me a value error. What do I do?

are the commas there because there are some values missing? – Padraic Cunningham Jan 15 '15 at 11:07 — Padraic Cunningham, Jan 15 '15 at 11:07

dragon2fly · Accepted Answer · 2015-01-15T11:00:37.597

1

You can use regular expression module to split

In[1]: import re
In[2]: re.split(',,|,', '0,0,,-2235 1,100,,-2209 2,200,,-2209')
Out[2]: ['0', '0', '-2235 1', '100', '-2209 2', '200', '-2209']

',,|,' means it firstly splits at ,, and then in the result, it continues to split at ,.

So if you want to get -2235 and 1 instead of -2235 1 you can use ',,|,| ' or ',,|,|\s' to ease the eyes (\s means space).

edited Jan 15 '15 at 11:00

answered Jan 15 '15 at 10:51

dragon2fly

2,309
19
23

score 1 · Answer 2 · answered Jan 15 '15 at 10:56

1

It really depends on what you're trying to do. I'd need to see a code example to see what you're trying to do really. You could just replace the double comma with a single one

inputstr = "0,0,,-2235 1,100,,-2209 2,200,,-2209"
inputstr = inputstr.replace(",,",",")

Or, if you don't want to lose the position of the data in the string, maybe replace the double commas with a null value of sorts

inputstr = "0,0,,-2235 1,100,,-2209 2,200,,-2209"
inputstr = inputstr.replace(",,",",0,")

The key thing is that you don't want to corrupt the data, and introduce values that shouldn't be there. It really is about what the data means and what you're trying to do.

answered Jan 15 '15 at 10:56

cyram

820
8
22

Nice touch with the replace. – ShellFish Jan 15 '15 at 11:14
Well the I just need to put the data into an array with the command inputfile = np.loadtxt("data.txt") but when I do than, I get: ValueError: invalid literal for float(): 0,0,,-2235. It seems like the format of the file is incorrect since the commas are so close together. What can I do to help that? – Jacob Hermann Olesen Jan 16 '15 at 10:08

score 0 · Answer 3 · answered Jan 15 '15 at 10:53

I don't know if this is an option but you could pre-process it using tr -s ',' file.txt. This is a shell command so you'd have to do it either before calling python or using system. The latter might not be the best way since dragon2fly solved the issue using a python function.

Loading data file with too many commas in Python

3 Answers3