Antler matches a similar rule (but fails on the parts that differ)

Question

I'm creating an Xtext plugin and for some reason, the following line incorrectly matches the StringStatement rule when it should match the UnstringStatement rule:

UNSTRING test2 DELIMITED BY " " INTO test2 END-UNSTRING

Here is my grammar:

Program:
    (elements+=Elemental)*
    (s+=Statement)*
;

Variable_Name:  
    varName=ID ("-" ID)*
;

Variable_Reference: 
    varRef=ID ("-" ID)*
;

Elemental: 
    'VAR' var=Variable_Name
;

Statement:
    (us=UnstringStatement|s=StringStatement)
;

StringParam:
    Variable_Reference | STRING
;

StringStatement:
    'STRING' in=StringParam 'DELIMITED BY SIZE INTO' out=Variable_Reference 'END-STRING'   
;


UnstringStatement:
    'UNSTRING' in=StringParam 'DELIMITED BY' string2=STRING 'INTO' (outs+=Variable_Reference)* 'END-UNSTRING'
;

When I run the project as an Eclipse Application, the 'UNSTRING' token is highlighted (correctly), but the rest of the line has the error "Mismatched character '"' expecting 'S'." The 'S' that the error refers too, is from 'SIZE'.

Any idea why the two rules overlap like this?

EDIT, forgot the STRING rule:

terminal STRING : 
    '"' ( '\\' . /* 'b'|'t'|'n'|'f'|'r'|'u'|'"'|"'"|'\\' */ | !('\\'|'"') )* '"' |
    "'" ( '\\' . /* 'b'|'t'|'n'|'f'|'r'|'u'|'"'|"'"|'\\' */ | !('\\'|"'") )* "'"
;

EDIT 2: After stepping through some of the Lexer code, I discovered that the token "DELIMITED BY" is incorrectly matched to "DELIMITED BY SIZE INTO", which then fails.

EDIT 3 FIXED: I fixed this, but have no idea why it works. I just added a terminal DELIMITED_BY:

terminal DELIMITED_BY: 'DELIMITED BY'

StringStatement:
    'STRING' in=StringParam DELIMITED_BY 'SIZE INTO' out=Variable_Reference 'END-STRING'   
;

UnstringStatement:
    'UNSTRING' in=StringParam DELIMITED_BY string2=STRING 'INTO' (outs+=Variable_Reference)* 'END-UNSTRING'
;

The grammar is incomplete: What is the rule for the Token `STRING`? — CoronA, Mar 09 '15 at 19:49

CoronA · Answer 1 · 2015-03-10T02:57:29.690

0

The STRING Token looks too greedy. In ANTLR the expression should be

terminal STRING : 
'"' ( '\\' . | !('\\'|'"') )*? '"' |
"'" ( '\\' . | !('\\'|"'") )*? "'"
;

edited Mar 10 '15 at 02:57

answered Mar 09 '15 at 20:07

CoronA

7,717
2
26
53

Actually that does look correct, as it will allow the empty string. But, in my case, any string fails – Erik Hunter Mar 09 '15 at 20:25
Have you checked that " " is a token with type String? I have shortened the STRING definition. Does this one work? – CoronA Mar 09 '15 at 21:10

Antler matches a similar rule (but fails on the parts that differ)

1 Answers1