antlr identifier name same as pre-defined function name cause MismatchedTokenException

Question

I have defined a ANTLR grammer:

grammar test5;

stats_statement
:   
    STATS IDENT ASSIGN_SYM functions_stats

;

functions_stats 

:   COUNT LPAREN IDENT RPAREN   
;

STATS
:   'STATS'
;   

COUNT
:   'count'
;   

IDENT
:   (LETTER | '_') (LETTER | DIGIT | '_')*
;   

ASSIGN_SYM
: ':='
;

COMMA_SYM
: ','
;

SEMI_SYM
: ';'
;

LPAREN 
: '(' ;

RPAREN 
: ')' ;

fragment 
LETTER : ('a'..'z' | 'A'..'Z') ;

fragment 
DIGIT : '0'..'9';

which has one build-in function "count". But if I use the following test string:

STATS count:=count(col1)

the parser will return an error saying:

mismatched input 'count' expecting IDENT

any clue and/or hints on how to fix this problem?

Thanks Charles

score 1 · Accepted Answer · answered Feb 14 '13 at 20:52

1

Create an ident rule that matched both IDENT and COUNT and use that rule in your parser rules (instead of using IDENT):

stats_statement
 : STATS ident ASSIGN_SYM functions_stats
 ;

functions_stats 
 :   COUNT LPAREN ident RPAREN   
 ;

ident
 : COUNT
 | IDENT
 ;

answered Feb 14 '13 at 20:52

Bart Kiers

166,582
36
299
288

It works. But if I have lots of pre-defined function, does it mean I need to add all those function names to the 'ident'? – Aeris Feb 14 '13 at 20:56

score 0 · Answer 2 · answered Feb 22 '13 at 19:28

Do you really need COUNT to be recognized in the lexer? Consider defining a more general method to match functions.

function 
  :   fn=IDENT LPAREN IDENT (',' IDENT)* RPAREN   
  ;

Then you can add an action to handle your semantics, such as looking up the function name in a table and making sure it has the right number of parameters, etc.

If that doesn't fit with what you needed to do, you could instead eliminate your COUNT rule and rewrite your functions_stats rule:

functions_stats 
  :   {input.LT(1).getText().equals("count")}? IDENT LPAREN IDENT RPAREN   
  ;

the {...}? is a disambiguating semantic predicate (see page 295 of The Definitive ANTLR Reference by Terence Parr). Only if it evaluates to true will this match. ANTLR 3 has a feature called 'hoisted semantic predicates' which basically means that semantic predicate is put into the appropriate callers of functions_stats so that information is used to make the decisions of what to parse.

Page 296 of the book has a section called "Keywords as Variables", here is the sample code from that chapter: http://media.pragprog.com/titles/tpantlr/code/predicates/keywords/Pred.g which might give you a better understanding.

The book itself is quite useful if you are doing advanced ANTLR work. The online documentation doesn't cover this stuff well.

antlr identifier name same as pre-defined function name cause MismatchedTokenException

2 Answers2