strtok weird behavior on tokenization

Question

The below code gives unexpected behaviour.

#include <stdio.h>
#include <string.h>

int main()
{
    
    char s[] = "hello $";
    char *t;
    t = strtok(s, "$$");
    printf("%s", t);

    return 0;
}

Why does this output hello and not hello $?

The function tokenizes using the ***set*** of characters in the second argument, It will use only one of the characters in the second arguyment string. — Some programmer dude, Aug 18 '21 at 11:40
The second parameter is a string containing multiple indepent single-character delimiters, not a substring. — Gerhardh, Aug 18 '21 at 11:41
"strtok weird behavior" is a truism. Its weirdness is by design. — William Pursell, Aug 18 '21 at 11:53

score 2 · Answer 1 · edited Aug 18 '21 at 12:02

From the C Standard (7.23.5.8 The strtok function)

2 A sequence of calls to the strtok function breaks the string pointed to by s1 into a sequence of tokens, each of which is delimited by a character from the string pointed to by s2

In this call

t = strtok(s, "$$");

the parameter s2 is set as having two identical delimiters. So the call is equivalent to

t = strtok(s, "$");

If you need to find the substring "$$" in a string you can use the standard C string function strstr. For example

#include <stdio.h>
#include <string.h>

int main(void) 
{
    char s[] = "hello $";
    
    char *p = strstr( s, "$$" );
    
    if ( p ) *p = '\0';
    
    puts( s );
    
    return 0;
}

Steve Summit · Accepted Answer · 2021-08-18T12:15:56.757

In a comment you wrote:

so how should I split by a string?

There is no standard way (no predefined library function) for doing this, that I am aware of.

You could write your own split-on-string loop, using strstr. Here is one way:

#include <stdio.h>
#include <string.h>

int main()
{
    char s[] = "hello$$this$$is$$a$$test";
    char *sep = "$$";
    char *p = s;
    char *p2;
    int i = 1;
    do {
        p2 = strstr(p, sep);
        if(p2 != NULL) *p2 = '\0';
        printf("%d: %s\n", i++, p);
        if(p2 != NULL) p = p2 + strlen(sep);
    } while(p2 != NULL);
}

This could be improved, but it works, and it should get you started.

strtok weird behavior on tokenization

2 Answers2