Possible to format the token output in antlr4?

Question

In antlr when I run the following test command to grab tokens:

$ grun TestLexer tokens -tokens myfile.sql
[@0,0:5='SELECT',<'SELECT'>,1:0]
[@1,7:7='1',<NUMBER>,1:7]
[@2,9:12='with',<'WITH'>,2:0]
[@3,14:20='my_data',<IDENTIFIER>,2:5]
[@4,22:23='as',<'AS'>,2:13]
[@5,25:25='(',<'('>,2:16]
[@6,27:32='select',<'SELECT'>,2:18]
[@7,34:40=''text1'',<STRING_TOKEN>,2:25]
[@8,41:42='::',<'::'>,2:32]

Is there a way for it to actually name or format the token, such as for the last one to have:

[@8,41:42='::',<'TYPECAST(::)'>,2:32]

As it's been declared in my TextLexer.g4 file:

// Cast operator, '2014-01-01'::DATE
TYPECAST
    :'::'
    ;

score 2 · Answer 1 · answered Aug 16 '22 at 01:32

You would need to create a new implementation of the Token interface (waviest would be to subclass CommonToken And override the toString() Method. You’d also need to create a new TokenFactory to create Tokens of your type. At that point you can write your own code to print out the tokens. There’s not a way to override the TokenFactory in the `TestRig class.

There’s code to do just this in the “Tokens and Token Factories” section of The Definitive ANTLR 4 Reference

Based on the questions you’re posting, this book is going to prove well worth it’s price.

Possible to format the token output in antlr4?

1 Answers1