C free variables used with strtok

Question

I already saw post similar to this one, but I found a little difference that drive me in the wrong way.

I have this code:

char * token_one = strtok(my_buffer, " ,.-");
char * token_two = strtok(NULL, " ,.-");

free(token_one);
free(token_two);

I saw post where people says a variable used with strtok should not be freed but why while executing this code I get this:

free(token_one) no errors

free(token_two) I get "invalid pointer"

Why don't I get error in free(token_one)? What is the correct way to handle this?

`strtok`'s return values point to the original string. If `my_buffer` is an allocated object, then it makes sense that this might not fail. However, this is another example of causing undefined behaviour and expecting that you'll always get errors, just like expecting a segfault when accessing memory outside of any objects. The correct way to handle this is to stop trying to `free` the return values, and read the documentation on `strtok`. — Thomas Jager, Aug 28 '19 at 13:32
If anything needs to be `free`d it would be `my_buffer` but only if that was dynamically allocated, and only after you finish accessing the tokens. If `token_one` is the same as `my_buffer` then that `free` was valid. — Weather Vane, Aug 28 '19 at 13:33
In general, you cannot `free` the result of `strtok`. You could copy the string returned by `strtok` to a buffer allocated by `malloc` (or the non-standard `strdup` function) for example, and `free` that buffer. — Ian Abbott, Aug 28 '19 at 13:34
You need to understand that strtok modifies your original string, It does not allocate new strings. — jarmod, Aug 28 '19 at 13:35
The contract with malloc+free does not specify that free will detect errors. It might, sometimes, but you can't rely on it. — mevets, Aug 28 '19 at 13:50

score 5 · Accepted Answer · answered Aug 28 '19 at 13:35

5

If you look how strtok() works, it becomes immediately clear:

The first call to strtok() returns the given pointer, modifying the string pointed to by it by replacing the next separation character with a NUL byte.
Any subsequent call, done with NULL as the first argument, operates on the saved pointer from before which points to one after the replaced character.

So the first free() call succeeds iff your my_buffer came from malloc(). The second one fails because, well, why shouldn't it? It doesn't come from malloc() et al., so calling free() on it is undefined behaviour.

answered Aug 28 '19 at 13:35

glglgl

89,107
13
149
217

strtok everytime give me the pointer to the next token and put a termination simbol at the end of the token? – user4789408 Aug 28 '19 at 13:37
@user4789408 That's roughly what its definition says, yes. – glglgl Aug 28 '19 at 13:38
2

@user4789408 Yes, but that piece of memory is not a new chunk of memory but it is part of the initial string. As soon as you free your first pointer, any access via the second will be illegal because the memory area where it points to is not yours any more. – Gerhardh Aug 28 '19 at 13:45

Vlad from Moscow · Answer 2 · 2019-08-28T13:42:38.723

It seems that the pointer my_buffer points to a dynamically allocated character array.

If the string stored in the array does not start from one of this characters " ,.-" then the pointer token_one returned from the function strtok will have the same value as the pointer my_buffer. So it can be used to free the allocated memory.

The pointer token_two points to the middle of the string. So its value does not point to the previously allocated memory. So the program after this call

free(token_two);

has undefined behavior.

This statement

free(token_two);

would be valid if the pointer token_two returned from the call of the function strtok was a null-pointer. But in this case no action will be performed by the call of free.

Consider two examples.

The first one is

#include <stdio.h>
#include <stdlib.h>

int main(void) 
{
    char *p1  = malloc( 10 );
    char *p2 = p1;

    free( p2 );

    return 0;
}

You can use the pointer p2 to free the memory the address of which initially was stored in the pointer p1.

On the other hand, you may not use the pointer p2 to free the allocated memory if its value points to inside the allocated memory

This program is invalid.

#include <stdio.h>
#include <stdlib.h>

int main(void) 
{
    char *p1  = malloc( 10 );
    char *p2 = p1 + 5;

    free( p2 );

    return 0;
}

C free variables used with strtok

2 Answers2