Catching (and keeping) all comments with ANTLR

Question

I'm writing a grammar in ANTLR that parses Java source files into ASTs for later analysis. Unlike other parsers (like JavaDoc) I'm trying to keep all of the comments. This is difficult comments can be used literally anywhere in the code. If a comment is somewhere in the source code that doesn't match the grammar, ANTLR can't finish parsing the file.

Is there a way to make ANTLR automatically add any comments it finds to the AST? I know the lexer can simply ignore all of the comments using either {skip();} or by sending the text to the hidden channel. With either of those options set, ANTLR parses the file without any problems at all.

Any ideas are welcome.

score 11 · Answer 1 · answered Jul 31 '13 at 02:43

Section 12.1 in "The Definitive Antlr 4 Reference" shows how to get access to comments without having to sprinkle the comments rules throughout the grammar. In short you add this to the grammar file:

grammar Java;

@lexer::members {
    public static final int WHITESPACE = 1;
    public static final int COMMENTS = 2;
}

Then for your comments rules do this:

COMMENT
    : '/*' .*? '*/' -> channel(COMMENTS)
    ;

LINE_COMMENT
    : '//' ~[\r\n]* -> channel(COMMENTS)
    ;

Then in your code ask for the tokens through the getHiddenTokensToLeft/getHiddenTokensToRight and look at the 12.1 section in the book and you will see how to do this.

Does not work. warning(155): vhdl.g4:1645:24: rule SPACE contains a lexer command with an unrecognized constant value; lexer interpreters may produce incorrect output error(164): vhdl.g4:26:0: custom channels are not supported in combined grammars — betontalpfa, Aug 31 '15 at 08:47

score 9 · Answer 2 · edited Mar 02 '18 at 03:32

first: direct all comments to a certain channel (only comments)

COMMENT
    : '/*' .*? '*/' -> channel(2)
    ;

LINE_COMMENT
    : '//' ~[\r\n]* -> channel(2)
    ;

second: print out all comments

      CommonTokenStream tokens = new CommonTokenStream(lexer);
      tokens.fill();
      for (int index = 0; index < tokens.size(); index++)
      {
         Token token = tokens.get(index);
         // substitute whatever parser you have
         if (token.getType() != Parser.WS) 
         {
            String out = "";
            // Comments will be printed as channel 2 (configured in .g4 grammar file)
            out += "Channel: " + token.getChannel();
            out += " Type: " + token.getType();
            out += " Hidden: ";
            List<Token> hiddenTokensToLeft = tokens.getHiddenTokensToLeft(index);
            for (int i = 0; hiddenTokensToLeft != null && i < hiddenTokensToLeft.size(); i++)
            {
               if (hiddenTokensToLeft.get(i).getType() != IDLParser.WS)
               {
                  out += "\n\t" + i + ":";
                  out += "\n\tChannel: " + hiddenTokensToLeft.get(i).getChannel() + "  Type: " + hiddenTokensToLeft.get(i).getType();
                  out += hiddenTokensToLeft.get(i).getText().replaceAll("\\s", "");
               }
            }
            out += token.getText().replaceAll("\\s", "");
            System.out.println(out);
         }
      }

This is not the answer to the literal question _Is there a way to make ANTLR automatically add any comments it finds to the AST?_, but this was the solution that I needed :-) Thanks — Juljan, Feb 06 '19 at 09:28

score 7 · Accepted Answer · answered Sep 19 '12 at 06:29

7

Is there a way to make ANTLR automatically add any comments it finds to the AST?

No, you'll have to sprinkle your entire grammar with extra comments rules to account for all the valid places comments can occur:

...

if_stat
 : 'if' comments '(' comments expr comments ')' comments ...
 ;

...

comments
 : (SingleLineComment | MultiLineComment)*
 ;

SingleLineComment
 : '//' ~('\r' | '\n')*
 ;

MultiLineComment
 : '/*' .* '*/'
 ;

answered Sep 19 '12 at 06:29

Bart Kiers

166,582
36
299
288

That's what I figured. Oh well. The real problem is that comments can be anywhere in source code, so every rule has to have "comments?" in every part of it. – T Suds Sep 19 '12 at 14:00
@TSuds, yeah, that is correct. Note that since my `comments` rule matches nothing or more comments, the `?` is not needed after it. – Bart Kiers Sep 19 '12 at 14:06
Depending on the use case, this might not be a good solution, see others. – zakjan Dec 22 '20 at 22:16

score 1 · Answer 4 · answered Aug 03 '15 at 11:00

1

The feature "island grammars" can also be used. See the the following section in the ANTLR4 book:

Island Grammars: Dealing with Different Formats in the Same File

answered Aug 03 '15 at 11:00

snorbi

2,590
26
36

score 0 · Answer 5 · answered Feb 26 '15 at 17:29

0

I did that on my lexer part :

WS  :   ( [ \t\r\n] | COMMENT) -> skip
;

fragment
COMMENT
: '/*'.*'*/' /*single comment*/
| '//'~('\r' | '\n')* /* multiple comment*/
;

Like that it will remove them automatically !

answered Feb 26 '15 at 17:29

Damiii

1,363
4
25
46

score 0 · Answer 6 · answered Mar 27 '18 at 09:27

0

For ANTLR v3:

The whitespace tokens are usually not processed by parser, but they are still captured on the HIDDEN channel.

If you use BufferedTokenStream, you can get to list of all tokens through it and do a postprocessing, adding them as needed.

answered Mar 27 '18 at 09:27

Jiri Tousek

12,211
5
29
43

Catching (and keeping) all comments with ANTLR

6 Answers6

Linked

Related