Tokenizer only prints the first token

Question

I am having trouble building a tokenizer. I am new to c++ and was wondering if anyone could help.

When I run the program, I enter the user input as "x = a + 1". When i do this, the only token output is the x. I want to display "x\n = a\n +\n 1\n"

#include <iostream>
#include <string>
using namespace std;

int main(void)
{

 char *text = (char*)malloc ( 40 *sizeof( char ));
 cout << "Enter the first arrangement of data." << endl;
 cin >> text;
 char *token = strtok(text, " ");
 while ( token )
 {
     if ( strlen(token) > 0 )
     {
        printf(" %s", token);
     }
     token = strtok(NULL, " ");
 }
 return 0;
}

i thought that (char*)malloc ( 40 *sizeof( char )) created a size of 40, would creating an array help fix it do you think? — user4878925, May 08 '15 at 14:12
It does, but multiplying by `sizeof(char)` is redundant, that's all. — olovb, May 08 '15 at 14:32

olovb · Accepted Answer · 2015-05-08T14:27:47.147

0

You are not reading the entire input.

Change

cin >> text;

to

cin.getline(text, 40);

edited May 08 '15 at 14:27

answered May 08 '15 at 14:11

olovb

2,164
1
17
20

I input "x = a + 1". if i alter my code so that text = " x = a + 1", the same problem persists. – user4878925 May 08 '15 at 14:14
1

That done it, i understand it better now i was only reading the first part of the input. Thanks – user4878925 May 08 '15 at 14:31
You now read the entire input line, instead of just the first letter. Your tokenizer loop is fine. – olovb May 08 '15 at 14:33

Tokenizer only prints the first token

1 Answers1