ANTLR4 lexer rules not matching correct block of text

Question

I am trying to understand how ANTLR4 works based on lexer and parser rules but I am missing something in the following example:

I am trying to parse a file and match all mathematic additions (eg 1+2+3 etc.). My file contains the following text:

start
4 + 5 + 22 + 1
other text other text test test
test test other text
55 other text
another text 2 + 4 + 255
number 44
end

and I would like to match

4 + 5 + 22 + 1

and

2 + 4 + 255

My grammar is as follows:

grammar Hello;
hi : expr+ EOF;
expr : NUM (PLUS NUM)+;

PLUS : '+' ;
NUM : [0-9]+ ;
SPACE : [\n\r\t ]+ ->skip;
OTHER : [a-z]+ ;

My abstract Syntax Tree is visualized as

Why does rule 'expr' matches the text 'start'? I also get an error "extraneous input 'start' expecting NUM"

If i make the following change in my grammar

OTHER : [a-z]+ ->skip;

the error is gone. In addition in the image above text '55 other text another text' matches the expression as a node in the AST. Why is this happening?

All the above have to do with the way lexer matches an input? I know that lexer looks for the first longest matching rule but how can I change my grammar so as to match only the additions?

sepp2k · Accepted Answer · 2018-07-20T14:48:31.217

Why does rule 'expr' matches the text 'start'?

It doesn't. When a token shows up red in the tree, that indicates an error. The token did not match any of the possible alternatives, so an error was produced and the parser continued with the next token.

In addition in the image above text '55 other text another text' matches the expression as a node in the AST. Why is this happening?

After you skipped the OTHER tokens, your input basically looks like this:

4 + 5 + 22 + 1 55 2 + 4 + 255 44

4 + 5 + 22 + 1 can be parsed as an expression, no problem. After that the parser either expects a + (continuing the expression) or a number (starting a new expression). So when it sees 55, that indicates the start of a new expression. Now it expects a + (because the grammar says that PLUS NUM must appear at least once after the first number in an expression). What it actually gets is the number 2. So it produces an error and ignores that token. Then it sees a +, which is what it expected. And then it continues that way until the 44, which again starts a new expression. Since that isn't followed by a +, that's another error.

All the above have to do with the way lexer matches an input?

Not really. The token sequence for "start 4 + 5" is OTHER NUM PLUS NUM, or just NUM PLUS NUM if you skip the OTHERs. The token sequence for "55 skippedtext 2 + 4" is NUM NUM PLUS NUM. I assume that's exactly what you'd expect.

Instead what seems to be confusing you is how ANTLR recovers from errors (or maybe that it recovers from errors).

I think the confusion comes from the expectation that ANTLR does island parsing by default. (Except that OP probably wouldn't use that terminology.) (See http://media.pragprog.com/titles/tpantlr2/islands.pdf) — rici, Jul 20 '18 at 16:29
@rici I'm not sure what you mean. The chapter you linked is about using lexical modes to basically ignore everything outside of certain sections ("islands") and then properly tokenize inside the sections. That does not seem to have anything to do with OP's question, nor is it something that ANTLR does by default without you telling it to. — sepp2k, Jul 20 '18 at 17:57
I know it doesn't do it by default. It seems to me that OP expects to find the valid expressions (as islands) and parse them, while leaving the ocean of non-expressions unparsed. That's how I interpret "I am trying to parse a file and match all mathematic additions" and the surprise when ANTLR attempts to match the whole input. — rici, Jul 20 '18 at 18:05
@rici Oh, now I see what you mean. I completely misread your first comment (I read "expectation" as "fact"). Yes, that probably is what the OP intended, now that you mention it. — sepp2k, Jul 20 '18 at 18:24

ANTLR4 lexer rules not matching correct block of text

1 Answers1