C++ - Finding the proper design for this

Question

I'm writing a script interpreter and i first need to tokenize a string containing the source code. For that i've identified different things :

• Identifiers (variable names) ;
• Symbols (+, -, etc... including "alphabetic" operators such as "return") ;
• Litteral values (true, false, 1, 3.14, "foo").

To represent this I thought about two different ways : either creating a class hierarchy :

class Token
{
public:
    enum type_e { E_IDENTIFIER, E_SYMBOL, E_LITTERAL }
    const type_e type;
};
class Identifier : public Token
{
public:
    const string name
}
class Symbol : public Token
{
public:
    const symbol_e symbol;
}
class Litteral : public Token
{
public:
    const Value value;
}

Which i would use with down casts this way :

bla bla parseStatement( bla bla )
{
    // ...
    Token * curTok = tokens[ curPos ];
    if( curTok->type == E_SYMBOL && dynamic_cast< Symbol * >( curTok )->symbol == E_PLUS )
    {
       // ...
    }
    // ...
}

But I was told that down casting means that my design is probably flawn. And it is also against the principle of polymorphism.

Then i thought about a second method, using some kind of variant class containing everything :

class Token
{
private:
    type_e _type;
public:
    type_e getType()
    bool isIdentifier()
    bool isSymbol()
    bool isLitteral()
    string getName() // If is an identifier, else exception
    symbol_e getSymbol() // If is a symbol, else exception
    Value getValue() // If is a litteral, else exception
}

Which i would use this way :

bla bla parseStatement( bla bla )
{
    // ...
    Token curTok = tokens[ curPos ];
    if( curTok.isSymbol() && curTok.getSymbol() == E_PLUS )
    {
       // ...
    }
    // ...
}

But it doesn't seem to me that it's cleaner. It's basically the same thing, just a little shorter to write.

I was suggested to use the visitor design pattern, but I can't figure out a way to use this pattern for my problem. I want to keep the syntax analyzing logic outside of the tokens. The logic will be in a syntax analyzing class that will manipulate these tokens. I also need the tokens to either be of the same type or have a common base class so that I can store them in a single array.

Do you have an idea how I could design this ? Thank you :)

Answer 1

Here is a version with Visitor Pattern

#include <iostream>
using namespace std;

class Value { };
class Identifier;
class Symbol;
class Literal;
class ParseVisitor {
public:
    virtual void VisitAndParseFor(Identifier& d1) = 0;
    virtual void VisitAndParseFor(Symbol& d2) = 0;
    virtual void VisitAndParseFor(Literal& d1) = 0;
};

class Token {
public:
    virtual void ParseWith(ParseVisitor& v) = 0;
};

class Identifier : public Token {
public:
    virtual void ParseWith(ParseVisitor& v) {
        v.VisitAndParseFor(*this);
    }
    const string name;
};

class Symbol : public Token {
public:
    enum symbol_e {
        E_PLUS, E_MINUS
    };
    virtual void ParseWith(ParseVisitor& v) {
        v.VisitAndParseFor(*this);
    }
    symbol_e symbol;
};

class Literal : public Token {
public:
    virtual void ParseWith(ParseVisitor& v) {
        v.VisitAndParseFor(*this); 
    }
    Value value;
};


// Implementing custom ParseVisitor
class Parser : public ParseVisitor {
    virtual void VisitAndParseFor(Identifier& identifier) { 
        std::printf("Parsing Identifier\n"); 
    }
    virtual void VisitAndParseFor(Symbol& symbol) { 
        std::printf("Parsing Symbol\n"); 
        switch (symbol.symbol) {
            case Symbol::symbol_e::E_PLUS: std::printf("Found plus symbol\n"); break;
            case Symbol::symbol_e::E_MINUS: std::printf("Found minus symbol\n"); break;
        }
    }
    virtual void VisitAndParseFor(Literal& literal) { 
        std::printf("Parsing Literal\n"); 
    }
};

int main() {
    Parser p;

    Identifier identifier;
    Symbol symbol;
    symbol.symbol = Symbol::symbol_e::E_PLUS;
    Literal literal;

    identifier.ParseWith(p);
    symbol.ParseWith(p);
    literal.ParseWith(p);

    return 0;
}

If you need context data when parsing, then add argument to Token::ParseWith and ParseVisitor::VisitAndParseFor. Likewise, you can change the return signature if you need to pass state/data back.