How to catch list of tokens in tree grammar of antlr3?

Question

I took a dummy language for example: It simply accepts one or more '!'. its lexer and grammar rules are:

grammar Ns;

options {
  output=AST;
  ASTLabelType=CommonTree;
}
tokens {
  NOTS;
}

@header { 
  package test;
}
@lexer::header {
  package test;
}

ns : NOT+ EOF -> ^(NOTS NOT+);

NOT : '!';

ok, as you can see, this represents a language which accept '!' or '!!!' or '!!!!!'...

and I defined some meaningful classes to build ASTs:

public class Not {
    public static final Not SINGLETON = new Not();

    private Not() {
    }
}



public class Ns {
    private List<Not> nots;

    public Ns(String nots) {
        this.nots = new ArrayList<Not>();
        for (int i = 0; i < nots.length(); i++) {
            this.nots.add(Not.SINGLETON);
        }
    }

    public String toString() {
        String ret = "";
        for (int i = 0; i < this.nots.size(); i++) {
            ret += "!";
        }
        return ret;
    }
}

and here's the tree grammar:

tree grammar NsTreeWalker;

options {
  output = AST;
  tokenVocab = Ns;
  ASTLabelType = CommonTree;
}
@header { 
  package test;
}
ns returns [Ns ret] : ^(NOTS n=NOT+) {$ret = new Ns($n.text);};

and the main class code with some sample data to test the generated classes:

public class Test {

    public static void main(String[] args) throws Exception {
        ANTLRInputStream input = new ANTLRInputStream(new ByteArrayInputStream("!!!".getBytes("utf-8")));
        NsLexer lexer = new NsLexer(input);
        CommonTokenStream tokens = new CommonTokenStream(lexer);
        NsParser parser = new NsParser(tokens);
        CommonTree root = (CommonTree) parser.ns().getTree();
        NsTreeWalker walker = new NsTreeWalker(new CommonTreeNodeStream(root));
        try {
            NsTreeWalker.ns_return r = walker.ns();
            System.out.println(r.ret);
        } catch (RecognitionException e) {
            e.printStackTrace();
        }
    }
}

but the final output printed is '!', other than the expecting '!!!'. that's mainly because this line of code :

ns returns [Ns ret] : ^(NOTS n=NOT+) {$ret = new Ns($n.text);};

the $n above captured only one '!', I don't know how to capture all three tokens of '!', in other words , a list of '!' with $n. Is there some one could help?thanks!

score 5 · Accepted Answer · answered Feb 12 '11 at 14:17

The fact that only one ! gets printed is because your rule:

ns returns [Ns ret] 
  :  ^(NOTS n=NOT+) {$ret = new Ns($n.text);}
  ;

gets more or less translated as:

Token n = null
LOOP
  n = match NOT_token
END
return new Ns(n.text)

Therefor, n.text will always be just a single !.

What you need to do is collect these NOT tokens in a list. In ANTLR you can create a list of tokens using the += operator instead of the "single token" operator =. So change your ns rule into:

ns returns [Ns ret] 
  :  ^(NOTS n+=NOT+) {$ret = new Ns($n);}
  ;

which gets translated as:

List n = null
LOOP
  n.add(match NOT_token)
END
return new Ns(n)

Be sure to change the constructor of your Ns class to take a List instead:

public Ns(List nots) {
    this.nots = new ArrayList<Not>();
    for (Object o : nots) {
        this.nots.add(Not.SINGLETON);
    }
}

after which the output of your test class would be:

!!!

Good luck!

How to catch list of tokens in tree grammar of antlr3?

1 Answers1