How do I do what strtok() does in C, in Python?

Question

I am learning Python and trying to figure out an efficient way to tokenize a string of numbers separated by commas into a list. Well formed cases work as I expect, but less well formed cases not so much.

If I have this:

A = '1,2,3,4'
B = [int(x) for x in A.split(',')]

B results in [1, 2, 3, 4]

which is what I expect, but if the string is something more like

A = '1,,2,3,4,'

if I'm using the same list comprehension expression for B as above, I get an exception. I think I understand why (because some of the "x" string values are not integers), but I'm thinking that there would be a way to parse this still quite elegantly such that tokenization of the string a works a bit more directly like strtok(A,",\n\t") would have done when called iteratively in C.

To be clear what I am asking; I am looking for an elegant/efficient/typical way in Python to have all of the following example cases of strings:

A='1,,2,3,\n,4,\n'
A='1,2,3,4'
A=',1,2,3,4,\t\n'
A='\n\t,1,2,3,,4\n'

return with the same list of:

B=[1,2,3,4]

via some sort of compact expression.

Answer 1

How about this:

A = '1, 2,,3,4  '
B = [int(x) for x in A.split(',') if x.strip()]

x.strip() trims whitespace from the string, which will make it empty if the string is all whitespace. An empty string is "false" in a boolean context, so it's filtered by the if part of the list comprehension.

Answer 2

For the sake of completeness, I will answer this seven year old question: The C program that uses strtok:

int main()
{
    char myLine[]="This is;a-line,with pieces";
    char *p;
    for(p=strtok(myLine, " ;-,"); p != NULL; p=strtok(NULL, " ;-,"))
    {
        printf("piece=%s\n", p);
    }
}

can be accomplished in python with re.split as:

import re
myLine="This is;a-line,with pieces"
for p in re.split("[ ;\-,]",myLine):
    print("piece="+p)

Answer 3

Generally, I try to avoid regular expressions, but if you want to split on a bunch of different things, they work. Try this:

import re
result = [int(x) for x in filter(None, re.split('[,\n,\t]', A))]

Answer 4

Mmm, functional goodness (with a bit of generator expression thrown in):

a = "1,2,,3,4,"
print map(int, filter(None, (i.strip() for i in a.split(','))))

For full functional joy:

import string
a = "1,2,,3,4,"
print map(int, filter(None, map(string.strip, a.split(','))))

Answer 5

This will work, and never raise an exception, if all the numbers are ints. The isdigit() call is false if there's a decimal point in the string.

>>> nums = ['1,,2,3,\n,4\n', '1,2,3,4', ',1,2,3,4,\t\n', '\n\t,1,2,3,,4\n']
>>> for n in nums:
...     [ int(i.strip()) for i in n if i.strip() and i.strip().isdigit() ]
... 
[1, 2, 3, 4]
[1, 2, 3, 4]
[1, 2, 3, 4]
[1, 2, 3, 4]

Answer 6

How about this?

>>> a = "1,2,,3,4,"
>>> map(int,filter(None,a.split(",")))
[1, 2, 3, 4]

filter will remove all false values (i.e. empty strings), which are then mapped to int.

EDIT: Just tested this against the above posted versions, and it seems to be significantly faster, 15% or so compared to the strip() one and more than twice as fast as the isdigit() one

Answer 7

Why accept inferior substitutes that cannot segfault your interpreter? With ctypes you can just call the real thing! :-)

# strtok in Python
from ctypes import c_char_p, cdll

try: libc = cdll.LoadLibrary('libc.so.6')
except WindowsError:
     libc = cdll.LoadLibrary('msvcrt.dll')

libc.strtok.restype = c_char_p
dat = c_char_p("1,,2,3,4")
sep = c_char_p(",\n\t")
result = [libc.strtok(dat, sep)] + list(iter(lambda: libc.strtok(None, sep), None))
print(result)

Answer 8

Why not just wrap in a try except block which catches anything not an integer?

Answer 9

I was desperately in need of strtok equivalent in Python. So I developed a simple one by my own

def strtok(val,delim):
    token_list=[]
    token_list.append(val)  
    for key in delim:       
        nList=[]        
        for token in token_list:            
            subTokens = [ x for x in token.split(key) if x.strip()]
            nList= nList + subTokens            
        token_list = nList  
    return token_list

Answer 10

I'd guess regular expressions are the way to go: http://docs.python.org/library/re.html