10

I will be continually splitting strings in a multithreaded application, I've read that strtok is not suitable for this, but why?

Should I consider using a semaphore around the portion of my code that calls strtok?

andandandand
  • 21,946
  • 60
  • 170
  • 271
  • duplicate: http://stackoverflow.com/questions/4031075/strtok-function-thread-safety – Necrolis Nov 30 '10 at 05:16
  • 3
    Please don't close this as a duplicate. The answers are related but it's not the same. Here, OP is asking for advice on how to handle the non-reentrancy of `strtok` rather than trying to debug a problem due to it. – R.. GitHub STOP HELPING ICE Nov 30 '10 at 05:35

3 Answers3

14

You should consider not using strtok or strtok_r at all. It's trivial to write your own functions similar to these but better-tailored to the exact way you want to use them, and of course have the caller store all the state and pass a pointer to the state for thread-safety/reentrancy.

As for your question about using a semaphore (or other locking primitive) around calls to strtok, that will not help if you just put it around the actual call. You'd have to hold the lock during the whole process of parsing the string to protect the internal state of strtok. I believe this is what many people refer to as locking code in place of data and it's generally considered A Bad Thing.

R.. GitHub STOP HELPING ICE
  • 208,859
  • 35
  • 376
  • 711
9

Use strtok_r() for thread safety.

fastcodejava
  • 39,895
  • 28
  • 133
  • 186
4

You'll need to use strtok_r.

In some non-standard implementations (most prominently Microsoft's), strtok stores its values in TLS (thread-local storage), so it should be fine to use in multiple threads at the same time. However, you cannot split your tokenization for one and the same string across multiple threads.

EboMike
  • 76,846
  • 14
  • 164
  • 167
  • 4
    A version of `strtok` that stores its state in TLS is non-conformant. A conformant program may, as long as it ensures proper synchronization, split calls to `strtok` on the *same string* across multiple threads. If the state is in TLS, this will obviously not work. Please file bug reports for whichever implementation does that. – R.. GitHub STOP HELPING ICE Nov 30 '10 at 05:28