ANTLR Tree Walker should not output AST?

Question

I have made an own language using ANTLR. The Grammar is complete, but I want to make a tree walker that verifies the input. But this doesn't work. When I (accidentally) used output=AST in the tree walker it compiles without error, but it doesn't accept any input. (It says org.antlr.runtime.tree.CommonTree cannot be cast to tools.ANode). But I think output=AST can not be used in a tree walker. However, it won't compile then. It gives a

internal error: /.../src/AwesomeChecker.g : java.lang.IllegalStateException: java.lang.NullPointerException
org.deved.antlride.runtime.AntlrErrorListener$DynamicToken.invokeMethod(AntlrErrorListener.java:59)
org.deved.antlride.runtime.AntlrErrorListener$DynamicToken.getLine(AntlrErrorListener.java:64)
org.deved.antlride.runtime.AntlrErrorListener.report(AntlrErrorListener.java:131)
org.deved.antlride.runtime.AntlrErrorListener.message(AntlrErrorListener.java:115)
org.deved.antlride.runtime.AntlrErrorListener.warning(AntlrErrorListener.java:99)
org.antlr.tool.ErrorManager.grammarWarning(ErrorManager.java:742)
org.antlr.tool.ErrorManager.grammarWarning(ErrorManager.java:757)
org.antlr.tool.Grammar.parseAndBuildAST(Grammar.java:655)
org.antlr.Tool.getRootGrammar(Tool.java:626)
org.antlr.Tool.process(Tool.java:459)
org.deved.antlride.runtime.Tool2.main(Tool2.java:24)

And when I try to compile using command-line, it says I should add output=AST. My first though was that the compiler sees some symbols that can only be used in the tree parser, but I can't find any 'illegal' symbols.

Can you please help me?

The grammar for the checker is:

//AwesomeChecker.g: ANTLR (context) checker voor Awesome++

tree grammar AwesomeChecker;

options
{
    tokenVocab = Awesome;       //import tokens from Awesome.tokens
    ASTLabelType = ANode;       //AST nodes are of type ANode
}

//Alter code generation so catch-clauses get replaced with this action.
//This disables ANTLR error handling: Exceptions are propagated upwards.
@rulecatch
{
    catch (RecognitionException re) 
    {
        throw re;
    }
}

@header
{
    import tools.*;
    import java.util.List;
    import java.util.ArrayList; 
}

@members
{
    //symbol table voor de identifiers
    private SymbolTable ST      = new SymbolTable();
    //voor de AST types
    private Checker checker     = new Checker();

    private void enter(ANode id, boolean isVar) throws AwesomeException {
        try {
            IdEntry ie = new IdEntry(id, isVar);
            this.ST.enter(id.getText(), ie);
        }
        catch (SymbolTableException e)  {
            throw new AwesomeException(id, "Al gedeclareerd");
        }
    }
}

start
    : program
    ;

program
        @init                   {   this.ST.openScope();    }
    :   ^(PROGRAM statement+)   {   this.ST.closeScope();   }
    ;

statement
    :   op=IDENTIFIER e=assignment_statement
            {
                checker.checkTypes($op, $e.start, op, checker.ANY_TYPE);
            }
    |   output_stat
    |   input_stat
    |   if_statement[false]
    |   do_while_statement
    |   declaration
    ;

assignment_statement
    : (op=BECOMES^ e=expr)+
            {
                op.setExprType($e.start.getExprType());
            }
    ;

declaration
    :   ^(DECL t=type c=CONST? id=IDENTIFIER (e=expr)?)
            {
                boolean isVar = (c == null); 
                int type = id.setExprType($t.start.getExprType());
                if(e == null) {

                } else {            
                    checker.checkType($e.start, type);
                }
                this.enter(id, isVar);
            }
    ;

type
    :   t=BOOLEAN
            {$t.setExprType(checker.BOOL_TYPE);}
    |   t=INTEGER
            {$t.setExprType(checker.INT_TYPE);}
    |   t=CHAR
        {$t.setExprType(checker.CHAR_TYPE);}
    ;

statements_block
        @init               {   this.ST.openScope();    }
    :   statement*
        {this.ST.closeScope();}
    ;

expr
    :   ^(op=COMPOUND (statement)* e=expr)
                {
                    op.setExprType($e.start.getExprType());
                }
    |   ^(op = (PLUS | MINUS | MULTIPLY)    e1=expr e2=expr)    
                            {   
                                checker.checkTypes($e1.start, $e2.start, op, checker.INT_TYPE); 
                                op.setExprType(checker.INT_TYPE);   
                            }
    |   ^(op = (DIVIDE | MODULUS)   e1=expr e2=expr)    
                            {   
                                checker.checkTypes($e1.start, $e2.start, op, checker.INT_TYPE);
                                if ($e2.text.equals("0"))
                                {
                                    throw new AwesomeException($e2.start, "Cannot divide by zero");
                                }
                                op.setExprType(checker.INT_TYPE);   
                            }
    |   ^(op = (SMALLER | SMALLEREQUAL | GREATEREQUAL | GREATER) e1=expr e2=expr)   
                            {   
                                checker.checkTypes($e1.start, $e2.start, op, checker.INT_TYPE); 
                                op.setExprType(checker.BOOL_TYPE);  
                            }
    |   ^(op = (EQUAL | NOTEQUAL) e1=expr e2=expr)  
                            {   
                                checker.checkTypes($e1.start, $e2.start, op, checker.ANY_TYPE); 
                                op.setExprType(checker.BOOL_TYPE);  
                            }
    |   ^(op = (AND | OR) e1=expr e2=expr)  
                            {   
                                checker.checkTypes($e1.start, $e2.start, op, checker.BOOL_TYPE);
                                op.setExprType(checker.BOOL_TYPE);  
                            }
    |   ^(op = UNARYEXPR (PLUS | MINUS) e1=expr)        
                            {   
                                checker.checkType($e1.start, checker.INT_TYPE);     
                                op.setExprType(checker.INT_TYPE);   
                            } 
    |   ^(op = UNARYEXPR NOTEQUAL e1=expr)      
                            {   
                                checker.checkType($e1.start, checker.BOOL_TYPE);    
                                op.setExprType(checker.BOOL_TYPE);  
                            }
    ;

input_stat
    :   ^(op=INPUT v=varlist)
                            {   
                                int type = $v.start.getExprType();
                                op.setExprType(type);
                            }
    ;

output_stat
    :   ^(op=OUTPUT v=varlist)
                            {
                                int type = $v.start.getExprType();
                                op.setExprType(type);
                            }
    ;

if_statement[boolean isExpr]
    :   ^(op=IF cond=expr (statements_block|op1=operand) (ELSE (statements_block|op2=operand))?)
            {
                checker.checkType($cond.start, checker.BOOL_TYPE);
                if(isExpr) {
                    int type = checker.checkTypes($op1.start, $op2.start, op, $op1.start.getExprType());
                    op.setExprType(type);
                } else {
                    op.setExprType(checker.VOID_TYPE);
                }
            }
    ;

do_while_statement
    :   ^(op=DO statements_block WHILE cond=expr)
        {
            checker.checkType($cond.start, checker.BOOL_TYPE);
            op.setExprType(checker.VOID_TYPE);
        }
    ;

varlist
    :   expr (COMMA expr)*
    ;

operand
    :   IDENTIFIER
    |   literal
    |   expr
    ;

literal
    :   NUMBER
    |   CHARLITERAL
    |   boolliteral  
    ;

boolliteral
        :    TRUE | FALSE
        ;

The grammar.g is:

grammar Awesome;

options {
    k=1;                                // LL(1) - do not use LL(*)
    language=Java;                      // target language is Java (= default)
    output=AST;                         // build an AST
}

tokens {
    COLON       =   ':'     ;
    COMMA       =   ','     ;
    SEMICOLON   =   ';'     ;
    LPAREN      =   '('     ;
    RPAREN      =   ')'     ;
    LBRACKET    =   '{'     ;
    RBRACKET    =   '}'     ;

    // keyword for AST
    DECL        =   'decl'  ;
    UNARYEXPR   =   'unexpr';
    COMPOUND    =   'compound';
    ASSIGNMENT  =   'ASSIGNMENT';

    // operators
    BECOMES     =   '='     ;
    PLUS        =   '+'     ;
    MINUS       =   '-'     ;
    MULTIPLY    =   '*'     ;
    DIVIDE      =   '/'     ;
    MODULUS     =   '%'     ;
    EXCLAMATION =   '!'     ;

    // comparators
    AND             =   'AND'   ;
    OR              =   'OR'    ;
    SMALLER         =   '<'     ;
    SMALLEREQUAL    =   '<='    ;
    GREATER         =   '>'     ;
    GREATEREQUAL    =   '>='    ;
    EQUAL           =   '=='    ;
    NOTEQUAL        =   '!='    ;  

    // keywords
    PROGRAM     =   'program'   ;
    RETURN      =   'return'   ;
    VAR         =   'var'       ;
    CONST       =   'const'     ;
    INPUT       =   'input'     ;
    OUTPUT      =   'output'     ;
    IF          =   'if'        ;
    THEN        =   'then'      ;
    ELSE        =   'else'      ;
    DO          =   'do'        ;
    WHILE       =   'while'     ;

    // types
    INTEGER     =   'int'   ;
    BOOLEAN     =   'bool'  ;
    CHAR        =   'char'  ;
    STRING      =   'string';

    TRUE        =   'true';
    FALSE       =   'false';

}

// Parser rules
start
    : program
    ;

program
    :   (statement)+ EOF
            ->  ^(PROGRAM statement+)
    ;

statement
    :   IDENTIFIER assignment_statement SEMICOLON!
    |   output_stat SEMICOLON!
    |   input_stat SEMICOLON!
    |   if_statement
    |   do_while_statement
    |   declaration SEMICOLON!
    ;

declaration
    :   type CONST? IDENTIFIER (BECOMES expr)?
        -> ^(DECL type CONST? IDENTIFIER (BECOMES expr)?)
    ;

type
    :   BOOLEAN
    |   INTEGER
    |   CHAR
    |   STRING
    ;

statements_block
    :   LBRACKET statement* RBRACKET
    ;

assignment_statement
    : (BECOMES^ expr)+
    ;

expr
    :   expr_or
    |   expr_compound
    |   if_statement
    ;

expr_unary
    :   operand
    |   PLUS operand -> ^(UNARYEXPR PLUS operand)
    |   MINUS operand -> ^(UNARYEXPR MINUS operand)
    |   EXCLAMATION operand -> ^(UNARYEXPR EXCLAMATION operand)
    ;

expr_multiply
    :   expr_unary ((MULTIPLY | DIVIDE | MODULUS)^ expr_unary)*
    ;

expr_plus
    :   expr_multiply ((PLUS | MINUS)^ expr_multiply)*
    ;

expr_compare
    :   expr_plus ((SMALLER | SMALLEREQUAL | GREATEREQUAL | GREATER | EQUAL | NOTEQUAL)^ expr_plus)*
    ;

expr_and
    :   expr_compare (AND^ expr_compare)* 
    ;

expr_or
    :   expr_and (OR^ expr_and)* 
    ;

expr_compound
    :   LBRACKET (statement)* RETURN expr SEMICOLON RBRACKET
        -> ^(COMPOUND (statement)* expr)
    ;

do_while_statement
    :   DO^ statements_block WHILE expr
    ;

if_statement
    :   IF^ expr (statements_block|operand) (ELSE (statements_block|operand))?
    ;

input_stat
    :   INPUT^ LPAREN! varlist RPAREN!
    ;

output_stat
    :   OUTPUT^ LPAREN! varlist RPAREN!
    ;

varlist
    :   expr (COMMA expr)*
    ;

operand
    :   IDENTIFIER
    |   literal
    |   LPAREN! expr RPAREN!
    ;

literal
    :   NUMBER
    |   CHARLITERAL
    |   boolliteral  
    ;

boolliteral
        :    TRUE | FALSE
        ;

// Lexer rules

IDENTIFIER
    :   LETTER (LETTER | DIGIT)*
    ;

CHARLITERAL
    :   '\'' LETTER '\''
    ;

NUMBER
    :   DIGIT+
    ;

COMMENT
    :   '[' .* ']' 
            { $channel=HIDDEN; }
    ;

WS
    :   (' ' | '\t' | '\f' | '\r' | '\n')+
            { $channel=HIDDEN; }
    ;

fragment DIGIT  :   ('0'..'9') ;
fragment LOWER  :   ('a'..'z') ;
fragment UPPER  :   ('A'..'Z') ;
fragment LETTER :   LOWER | UPPER ;

// EOF

EDIT: I found that

assignment_statement
        : (op=BECOMES^ e=expr)+

was the error, it should be

assignment_statement
        : (op=BECOMES e=expr)+

It compiles without errors! :D Thank you for your help!

score 1 · Accepted Answer · answered Jun 28 '12 at 12:34

I found that

assignment_statement
        : (op=BECOMES^ e=expr)+

was the error, it should be

assignment_statement
        : (op=BECOMES e=expr)+

I did not change anything in the options of the grammar files.

It compiles now without errors! :D Thank you for your help!

score 0 · Answer 2 · answered Jun 28 '12 at 10:23

But I think output=AST can not be used in a tree walker.

Sure it can. The idea is that your first grammar turns the input text into a tree of nodes, and then the tree grammar does some other operation. If you're making a compiler, then presumably the tree walker comes in between the parsing and code generation phases. Your output type has to be an AST so that the next phase of the process can do it's job... Even if you don't actually make any changes to the tree or do anything else with it.

Your problem is this line in the tree grammar:

ASTLabelType = ANode

In a tree grammar, it's expecting the incoming AST to have nodes of that type, but your first grammar doesn't create nodes of that type. It creates the default CommonTree nodes. So just set that same line in the first grammar.

I don't see a lot of documentation about ASTLabelType even in the official reference book, but there are a few notes about custom node types near the bottom of the "Tree construction" page on the antlr wiki.

ANTLR Tree Walker should not output AST?

2 Answers2