strtok() tokenizes only once when nested

Question

Suppose I have the following string: 0:1,2,3.

I want to separate first using : as the delimiter and when it gets to the second part (i.e 1,2,3) and try to use strtok on that (with ,) it does not work as expected.

#include <stdio.h>
#include <stdbool.h>

int main(void){
    char s[10];
    strcpy(s, "0:1,2,3");
    char* token1 = strtok(s, ":");
    //To let me know it is on the second part
    bool isSecondToken = false;
    while (token1) {
        printf("token1: %s\n", token1);
        if(isSecondToken == true){
            char* token2 = strtok(token1, ",");
            while (token2) {
                printf("token2: %s\n", token2);
                token2 = strtok(NULL, " ");
            }
        }
        token1 = strtok(NULL, " ");
        isSecondToken = true;
    }
}

Output I get:

token1: 0
token1: 1,2,3
token2: 1
token2: 2,3

Expected output:

token1: 0
token1: 1,2,3
token2: 1
token2: 2
token2: 3

Your call to strtok inside the loop says you want to split by space, not comma, so it'll return until next space is found which is never, hence "2,3" — Sami Kuhmonen, Sep 23 '18 at 15:07

score 3 · Accepted Answer · answered Sep 23 '18 at 15:11

When updating the token1 and token2 pointers you need to use the same token splitter:

#include <stdio.h>
#include <stdbool.h>
#include <string.h>

int main(void){
    char s[10];
    strcpy(s, "0:1,2,3");
    char* token1 = strtok(s, ":");
    //To let me know it is on the second part
    bool isSecondToken = false;
    while (token1) {
        printf("token1: %s\n", token1);
        if(isSecondToken == true){
            char* token2 = strtok(token1, ",");
            while (token2) {
                printf("token2: %s\n", token2);
                token2 = strtok(NULL, ",");
            }
        }
        token1 = strtok(NULL, ":");
        isSecondToken = true;
    }
}

Also strcpy requires the string.h library, so you were probably also getting some warnings of implicit declaration.

strtok() tokenizes only once when nested

1 Answers1