strtok: How to store tokens in two different buffers

Question

I have a string containing datatypes and addresses of variables. These values are separated by "///" and they are alternating (type /// address /// type /// address ...). The amount of these tuples is not fixed and can vary from execution to execution.

Now my problem is how to process the string in a loop, as strtok needs to be called first with the original string and then with the NULL parameter but in the loop it has to be called twice. So after the first loop strtok is called three times which leads to an uneven count of strtok executions whereas it should be an even count. I tried to solve this problem by processing the first tuple outside the loop (because strtok has to be called with the original string) and process the remaining tuples inside the loop.

char mystring[128];
char seperator[] = "///";
char *part;
int type [128];
int address [128];
number_of_variables = 0;

part = strtok(mystring, seperator);
type[number_of_variables] = (int) atoi(part);
part = strtok(NULL, seperator);
address[number_of_variables] = (int)strtol(part, NULL, 16);

while(part != NULL){
    part = strtok(NULL, seperator);
    type[number_of_variables] = (int) atoi(part);


    part = strtok(NULL, seperator);
    address[number_of_variables] = (int)strtol(part, NULL, 16);

    number_of_variables++;
}

So now I have an even count of strtok executions but if my strings contains for example 2 tuples it will enter the loop for a second time so strtok is called for a fifth time which causes the program to crash as atoi() gets a bad pointer.

EDIT: Example for mystring:

"1///0x37660///2///0x38398"

1 and 2 are type identifiers for the further program.

Answer 1

I can suggest the following loop as it is shown in the demonstrative program below.

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

int main(void) 
{
    char mystring[128] = "1///0x37660///2///0x38398";
    char separator[] = "/ ";
    int type [128];
    int address [128];

    size_t number_of_variables = 0;

    for ( char *part = strtok( mystring, separator ); part; part = strtok( NULL, separator ) )
    {
        type[number_of_variables] = atoi(part);
        part = strtok( NULL, separator );
        address[number_of_variables] = part ? (int)strtol(part, NULL, 16) : 0;
        ++number_of_variables;
    }

    for ( size_t i = 0; i < number_of_variables; i++ )
    {
        printf( "%d\t%x\n", type[i], address[i] );
    }

    return 0;
}

The program output is

1   37660
2   38398

Answer 2

You can write a robust and fast parser, that is guaranteed to work and has no bugs like this

File: lexer.l

%{
#include <stdio.h>
#include "parser.tab.h"
int yyerror(const char *const message);
%}

%option noyywrap
%x IN_ADDRESS

DECIMAL [0-9]+
HEX "0x"[a-fA-F0-9]+
DELIMITER "///"

%%

<*>{DELIMITER} { return DELIMITER; }

<INITIAL>{DECIMAL} {
        char *endptr;
        // Make the lexer know that we are expecting a
        // hex number
        BEGIN(IN_ADDRESS);
        // Asign the value to use by bison
        yylval = strtol(yytext, &endptr, 10);
        // Check conversion's success
        if (*endptr != '\0')
            return ERROR;
        return TYPE;
    }

<IN_ADDRESS>{HEX} {
        char *endptr;
        // Restore the initial state
        BEGIN(INITIAL);
        // Asign the value to use by bison
        yylval = strtol(yytext, &endptr, 16);
        // Check conversion's success
        if (*endptr != '\0')
            return ERROR;
        return ADDRESS;
    }

%%

File: parser.y

%{
#include <stdio.h>
extern int yylex();
extern FILE *yyin;

int yyerror(const char *const message);

#define YYSTYPE int
%}


%token TYPE
%token DELIMITER
%token ADDRESS
%token ERROR

%%

program:
       | program statement
       ;

command: TYPE DELIMITER ADDRESS { 
           fprintf(stdout, "type %d, address 0x%08x\n", $1, $3); 
       }
       ;

statement: command
         | statement DELIMITER command;
         ;

%%

int
yyerror(const char *const message)
{
    return fprintf(stdout, "error: %s\n", message);
}

int
main(void)
{
    yyin = fopen("program.txt", "r");
    if (yyin == NULL)
        return -1;
    yyparse();
}

File: program.txt

1///0x37660///2///0x38398

Compiling this with gcc, bison and flex is rather simple

bison -d parser.y
flex lexer.l
gcc -Wno-unused-function -Wall -Werror lex.yy.c parser.tab.c -o parserparser

Of course, this program needs some tweaking and adjusting it to your needs should be straightforward.

Just find a simple tutorial on bison and flex to help you fully understand this code.