Using strtok twice to separate a line into "words" and the "words" into smaller words?

Question

I've got a text file that looks something like this

44,12,4,5 2,45,3,1,2,45 6,77,5,3,5,44

I would like to use strtok() to separate this file into just numbers and read them into a char**, with spaces included, such that

arr[0] = "44"
arr[1] = "12"
arr[2] = "4"
arr[3] = "5"
arr[4] = " "
arr[5] = "2"
...

This is my code so far:

    int i = 0;
    char line[6000], **arr = calloc(200, sizeof(char*)), *token = calloc(50, sizeof(char)), *token2 = calloc(8, sizeof(char));
    FILE* textFile = openFileForReading(); //Simple method, works fine.
    fgets(line, sizeof line, textFile);
    token = strtok(line, " ");
    token2 = strtok(token, ",");
    arr[i] = token2;
    while((token2 = strtok(NULL, ",")) != NULL)
    {
            i++;
            arr[i] = token2;
    }

    i++;
    arr[i] = " "; //adds the space once we're done looping through the "word"

    while((token = strtok(NULL, " ")) != NULL) //PROGRAM BREAKS HERE
    {
            token2 = strtok(token, ",");
            i++;
            arr[i] = token2;
            while((token2 = strtok(NULL, ",")) != NULL)
            {
                    i++;
                    arr[i] = token2;
            }
            i++;
            arr[i] = " ";
    }

At the beginning of the second while loop, it's never executed. I'm sure it's got something to do with passing a NULL argument into strtok, but I'm not sure really how to get around this. If any of you folks have advice, suggestions, or critique, I'd love to hear it.

`strtok` will not work because it keeps hidden state between calls. POSIX has `strtok_r` which will work because it has no hidden state. — Ian Abbott, Mar 26 '19 at 17:11
[`strtok`](https://en.cppreference.com/w/c/string/byte/strtok) is not reentrant, you can't use it to parse more than one string. Some systems have a reentrant `strtok_r` function, but it's not portable. Best (portable) solution is to do it in two passes. — Some programmer dude, Mar 26 '19 at 17:12
You can use `strcspn` instead of `strtok`. The difference is that `strcspn` won't modify the input string, so you can examine the location where it found the delimiter, to see whether it's a comma or a space. Then you can write the `'\0'` to terminate the substring. — user3386109, Mar 26 '19 at 17:15
Windows has `strtok_s()` which is compatible except in name with POSIX `strtok_r()`, and is mostly unrelated to the `strtok_s()` defined in C11 Annex K (though the function in Annex K was designed using the Windows version as a starting point). — Jonathan Leffler, Mar 26 '19 at 17:20

score 1 · Answer 1 · answered Mar 26 '19 at 17:17

strtok() maintains state between calls when it's parsing a single string, so it can't be used as you've outlined above.

You have two choices: either use strtok_r() which is re-entrant and therefore can be used as you've written, or use strtok() but complete the initial parse into space separated lists first, then iterate over the resulting strings, treating them as comma separated numbers.

Using strtok twice to separate a line into "words" and the "words" into smaller words?

1 Answers1