Writing a lexer : lexer doesnt find "

Question

I am writing a small programming language that has only one keyword which is write. I haven't done a parser but lexer. It works fine till it gets to " token.

main.cpp:

#include <iostream>
#include <fstream>
#include <string>
#include <iomanip>
#include <vector>
using namespace std;

int main(int argc, char* argv[2])
{
    char text;
    string tok = "";
    string str = "";
    vector <string> tokens;

    fstream inFile(argv[1]);

    while (inFile >> noskipws >> text)
    {
        tok += text;

        if (tok == "write")
        {
            tokens.push_back(tok);
            tok = "";
        }
        else if(tok == "\"")
        {
            str += text;
            tokens.push_back("String:"+str);
            str = "";
            tok = "";
            tok += text;
            }

        for (auto const& c : tokens)
            std::cout << c << ' ';
    }

    return 0;
}

test.txt:

write "hello"

Output is:

write write write write write write write ...

Instead of,

String:hello

What can I do? Any ideas?

The right tool to solve such problems is your debugger. You should step through your code line-by-line *before* asking on Stack Overflow. For more help, please read [How to debug small programs (by Eric Lippert)](https://ericlippert.com/2014/03/05/how-to-debug-small-programs/). At a minimum, you should [edit] your question to include a [Minimal, Complete, and Verifiable](http://stackoverflow.com/help/mcve) example that reproduces your problem, along with the observations you made in the debugger. — πάντα ῥεῖ, Jun 28 '17 at 09:38
And what are the variable values when you stepped through the code line by line? Also `string tokens[];` looks wrong, did that even compile? — πάντα ῥεῖ, Jun 28 '17 at 09:43
Again what are the variable values when you were stepping through? — πάντα ῥεῖ, Jun 28 '17 at 09:51

score 0 · Answer 1 · answered Jun 28 '17 at 09:52

0

I think it should be skipws instead of noskipws, because you don't handle the case when it encounters whitespace between "write" and "hello"; specifically you don't clear tok, so the next loop it still thinks the token is a whitespace.

answered Jun 28 '17 at 09:52

meowgoesthedog

14,670
4
27
40

thnx very very much – Gameplayer Xd Jun 28 '17 at 10:09
@GameplayerXd no problem – meowgoesthedog Jun 28 '17 at 10:09

Writing a lexer : lexer doesnt find "

1 Answers1