Is there a way to count tokens in C?

Question

I'm using strtok to split a string into tokens. Does anyone know any function which actually counts the number of tokens?

I have a command string and I need to split it and pass the arguments to execve() .

Thanks!

Edit

execve takes arguments as char**, so I need to allocate an array of pointers. I don't know how many to allocate without knowing how many tokens are there.

And `realloc` should solve the problem of not knowing the size in advance. — Kerrek SB, Oct 25 '12 at 23:46

nneonneo · Accepted Answer · 2012-10-26T01:36:28.673

11

One approach would be to simply use strtok with a counter. However, that will modify the original string.

Another approach is to use strchr in a loop, like so:

int count = 0;
char *ptr = s;
while((ptr = strchr(ptr, ' ')) != NULL) {
    count++;
    ptr++;
}

If you have multiple delimiters, use strpbrk:

while((ptr = strpbrk(ptr, " \t")) != NULL) ...

edited Oct 26 '12 at 01:36

answered Oct 25 '12 at 23:54

nneonneo

171,345
36
312
383

strchr() gets cumbersome when there are multiple field delimiters, for example punctuation. – jim mcnamara Oct 26 '12 at 01:35
You can use `strpbrk` in that case. – nneonneo Oct 26 '12 at 01:35
1

OK, I have amended my answer to include `strpbrk`. Thanks for your feedback. – nneonneo Oct 26 '12 at 01:37
Sorry to barge in like this but is this seemed like the best answer I've found. Is there a way to get the lengths of each segment between the delimiter. Thanks again for this answer. Cheers @nneonneo – Isabel Inc Sep 11 '16 at 22:28
You can just keep track of the previous value of `ptr` and take the difference between the two - the pointers point to different parts of the same string. – nneonneo Sep 11 '16 at 23:24

router · Answer 2 · 2014-10-15T14:52:46.930

4

As number of tokens is nothing but one more than the frequency of occurrence of the delimiter used. So your question boils down to find no. of times of occurrence of a character in a string

say the delimiter used in strtok function in c is ' '

int count =0,i;
char str[20] = "some string here";

for(i=0;i<strlen(str);i++){
    if(str[i] == ' ')
        count++;
}

No. of tokens would be same as count+1

edited Oct 15 '14 at 14:52

answered Oct 15 '14 at 14:39

router

582
5
16

Does the job but gets cumbersome if more than one delimiter is allowed. – Carey Gregory Sep 19 '15 at 16:49
I guess we can use the same procedure and should take care of the corner cases – router Sep 20 '15 at 07:07
This would count two spaces in a row as two separate word-separators. Doesn't `strtok()` treat consecutive delimiters as a single delimiter? – jvriesem Apr 29 '20 at 04:29

score 1 · Answer 3 · answered Jan 30 '19 at 10:10

Here is a version based on strtok which does not modify the original string, but a temporal copy of it. This version works for any combination of tabs and space characters used as tokens separators. The function is

unsigned long int getNofTokens(const char* string){
  char* stringCopy;
  unsigned long int stringLength;
  unsigned long int count = 0;

  stringLength = (unsigned)strlen(string);
  stringCopy = malloc((stringLength+1)*sizeof(char));
  strcpy(stringCopy,string);

  if( strtok(stringCopy, " \t") != NULL){
    count++;
    while( strtok(NULL," \t") != NULL )
        count++;
  }

  free(stringCopy);
  return count;
}

A function call could be

char stringExample[]=" wordA 25.4 \t 5.6e-3\t\twordB 4.5e005\t ";
printf("number of elements in stringExample is %lu",getNofTokens(stringExample));

Output is

number of elements in stringExample is 5

Is there a way to count tokens in C?

3 Answers3

Linked