Antlr setText not working in the way I expected

Question

I have a requirement to convert an identifier into a beanutil string for retrieving an item from an object. The the identifiers to string conversions look like:

name                                        ==> name
attribute.name                              ==> attributes(name)[0].value
attribute.name[2]                           ==> attributes(name)[2].value
address.attribute.postalcode                ==> contactDetails.addresses[0].attributes(postalcode)[0].value
address[2].attribute.postalcode             ==> contactDetails.addresses[2].attributes(postalcode)[0].value
address[2].attribute.postalcode[3]          ==> contactDetails.addresses[2].attributes(postalcode)[3].value

Now I have decided to do this using antlr as I feel its probably going to be just as quick as using a set of 'if' statements. Feel free to tell me I'm wrong.

Right now, I've got this partial working using antlr, however once I start doing the 'address' ones, the setText part seems to stop working for Attribute.

Am I doing this the correct way or is there a better way of using antlr to get the result I want?

grammar AttributeParser;

parse returns [ String result ]
:  Address EOF { $result = $Address.text; }
|  Attribute EOF { $result = $Attribute.text; }
|  Varname EOF { $result = $Varname.text; }
;

Address
: 'address' (Arraypos)* '.' Attribute { setText("contactDetails.addresses" + ($Arraypos == null ? "[0]" : $Arraypos.text ) + "." + $Attribute.text); }
; 

Attribute
:  'attribute.' Varname (Arraypos)* { setText("attributes(" + $Varname.text + ")" + ($Arraypos == null ? "[0]" : $Arraypos.text ) + ".value"); }
;

Arraypos
: '[' Number+ ']'
;


Varname  
:  ('a'..'z'|'A'..'Z')+
;

Number
: '0'..'9'+
;

Spaces
:  (' ' | '\t' | '\r' | '\n')+ { setText(" "); }
;

Below are two unit tests, the first returns what I expect, the second doesn't.

@Test
public void testSimpleAttributeWithArrayRef() throws Exception {

    String source = "attribute.name[2]";
    ANTLRStringStream in = new ANTLRStringStream(source);
    AttributeParserLexer lexer = new AttributeParserLexer(in);
    CommonTokenStream tokens = new CommonTokenStream(lexer);
    AttributeParserParser parser = new AttributeParserParser(tokens);
    String result = parser.parse();
    assertEquals("attributes(name)[2].value", result);       
}

@Test
public void testAddress() throws Exception {

    String source = "address.attribute.postalcode";
    ANTLRStringStream in = new ANTLRStringStream(source);
    AttributeParserLexer lexer = new AttributeParserLexer(in);
    CommonTokenStream tokens = new CommonTokenStream(lexer);
    AttributeParserParser parser = new AttributeParserParser(tokens);
    String result = parser.parse();
    System.out.println("Result: " + result);
    assertEquals("contactDetails.addresses[0].attributes(postalcode)[0].value", result);       
}

Answer 1

No, you can't do (Arraypos)* and then refer to the contents as this: $Arraypos.text.

I wouldn't go changing the inner text of the tokens, but create a couple of parser rules and let them return the appropriate text.

A little demo:

grammar AttributeParser;

parse returns [String s]
  :  input EOF {$s = $input.s;}
  ;

input returns [String s]
  :  address   {$s = $address.s;}
  |  attribute {$s = $attribute.s;}
  |  Varname   {$s = $Varname.text;}
  ;

address returns [String s]
  :  Address arrayPos '.' attribute
     {$s = "contactDetails.addresses" + $arrayPos.s + "." + $attribute.s;}
  ;

attribute returns [String s]
  :  Attribute '.' Varname arrayPos 
     {$s = "attributes(" + $Varname.text + ")" + $arrayPos.s + ".value" ;}
  ;

arrayPos returns [String s]
  :  Arraypos      {$s = $Arraypos.text;}
  |  /* nothing */ {$s = "[0]";}
  ;

Attribute : 'attribute';
Address   : 'address';
Arraypos  : '[' '0'..'9'+ ']';
Varname   : ('a'..'z' | 'A'..'Z')+;

which can be tested with:

import org.antlr.runtime.*;

public class Main {
  public static void main(String[] args) throws Exception {
    String[][] tests = {
      {"name",                                "name"},
      {"attribute.name",                      "attributes(name)[0].value"},
      {"attribute.name[2]",                   "attributes(name)[2].value"},
      {"address.attribute.postalcode",        "contactDetails.addresses[0].attributes(postalcode)[0].value"},
      {"address[2].attribute.postalcode",     "contactDetails.addresses[2].attributes(postalcode)[0].value"},
      {"address[2].attribute.postalcode[3]",  "contactDetails.addresses[2].attributes(postalcode)[3].value"}
    };
    for(String[] test : tests) {
      String input = test[0];
      String expected = test[1];
      AttributeParserLexer lexer = new AttributeParserLexer(new ANTLRStringStream(input));
      AttributeParserParser parser = new AttributeParserParser(new CommonTokenStream(lexer));
      String output = parser.parse();
      if(!output.equals(expected)) {
        throw new RuntimeException(output + " != " + expected);
      }
      System.out.printf("in  = %s\nout = %s\n\n", input, output, expected);
    }
  }
}

And to run the demo do:

java -cp antlr-3.3.jar org.antlr.Tool AttributeParser.g
javac -cp antlr-3.3.jar *.java
java -cp .:antlr-3.3.jar Main

which will print the following to the console:

in  = name
out = name

in  = attribute.name
out = attributes(name)[0].value

in  = attribute.name[2]
out = attributes(name)[2].value

in  = address.attribute.postalcode
out = contactDetails.addresses[0].attributes(postalcode)[0].value

in  = address[2].attribute.postalcode
out = contactDetails.addresses[2].attributes(postalcode)[0].value

in  = address[2].attribute.postalcode[3]
out = contactDetails.addresses[2].attributes(postalcode)[3].value

EDIT

Note that you can also let parser rules return more than just one object like this:

bar
  :  foo {System.out.println($foo.text + ", " + $foo.number);}
  ;

foo returns [String text, int number]
  :  'FOO' {$text = "a"; $number = 1;}
  |  'foo' {$text = "b"; $number = 2;}
  ;