Add quoted strings support to Antlr3 grammar

Question

I'm trying to implement a grammar for parsing queries. Single query consists of items where each item can be either name or name-ref.

name is either mystring (only letters, no spaces) or "my long string" (letters and spaces, always quoted). name-ref is very similar to name and the only difference is that it should start with ref: (ref:mystring, ref:"my long string"). Query should contain at least 1 item (name or name-ref).

Here's what I have:

NAME: ('a'..'z')+;
REF_TAG: 'ref:';
SP: ' '+;

name: NAME;
name_ref: REF_TAG name;
item: name | name_ref;
query: item (SP item)*;

This grammar demonstrates what I basically need to get and the only feature is that it doesn't support long quoted strings (it works fine for names that doesn't have spaces).

SHORT_NAME: ('a'..'z')+;
LONG_NAME: SHORT_NAME (SP SHORT_NAME)*;
REF_TAG: 'ref:';
SP: ' '+;
Q: '"';

short_name: SHORT_NAME;
long_name: LONG_NAME;
name_ref: REF_TAG (short_name | (Q long_name Q));
item: (short_name | (Q long_name Q)) | name_ref;
query: item (SP item)*;

But that doesn't work. Any ideas what's the problem? Probably, that's important: my first query should be treated as 3 items (3 names) and "my first query" is 1 item (1 long_name).

Bart Kiers · Accepted Answer · 2012-01-22T08:43:32.150

ANTLR's lexer matches greedily: that is why input like my first query is being tokenized as LONG_NAME instead of 3 SHORT_NAMEs with spaces in between.

Simply remove the LONG_NAME rule and define it in the parser rule long_name.

The following grammar:

SHORT_NAME : ('a'..'z')+;
REF_TAG    : 'ref:';
SP         : ' '+;
Q          : '"';

short_name : SHORT_NAME;
long_name  : Q SHORT_NAME (SP SHORT_NAME)* Q;
name_ref   : REF_TAG (short_name | (Q long_name Q));
item       : short_name | long_name | name_ref;
query      : item (SP item)*;

will parse the input:

my first query "my first query" ref:mystring

as follows:

enter image description here

However, you could also tokenize a quoted name in the lexer and strip the quotes from it with a bit of custom code. And removing spaces from the lexer could also be an option. Something like this:

SHORT_NAME : ('a'..'z')+;
LONG_NAME  : '"' ~'"'* '"' {setText(getText().substring(1, getText().length()-1));};
REF_TAG    : 'ref:';
SP         : ' '+ {skip();};

name_ref   : REF_TAG (SHORT_NAME | LONG_NAME);
item       : SHORT_NAME | LONG_NAME | name_ref;
query      : item+ EOF;

which would parse the same input as follows:

enter image description here

Note that the actual token LONG_NAME will be stripped of its start- and end-quote.

Is there any way to get rid of `"` under `long_name` node? I want the for syntax, but I don't need them when working with results of parsing. Will really appreciate non-inline-Java solution. — Andrey Agibalov, Jan 22 '12 at 08:44
@loki2302, you mean from the first tree-image? EDIT: wait, I see you accepted my answer. This means you have no question(s) anymore? — Bart Kiers, Jan 22 '12 at 08:47
I've just responded before seeing you've updated your answer. Well, yes. Basically my question here is - is there a way to implement your second approach without Java string procedures? — Andrey Agibalov, Jan 22 '12 at 08:49

craigmj · Answer 2 · 2012-01-22T07:57:24.477

Here's a grammar that should work for your requirements:

  SP: ' '+;
  SHORT_NAME: ('a'..'z')+;
  LONG_NAME: '"' SHORT_NAME (SP SHORT_NAME)* '"';
  REF: 'ref:' (SHORT_NAME | LONG_NAME);

  item: SHORT_NAME | LONG_NAME  | REF;
  query: item (SP item)*;

If you put this at the top:

  grammar Query;

  @members {
      public static void main(String[] args) throws Exception {
          QueryLexer lex = new QueryLexer(new ANTLRFileStream(args[0]));
           CommonTokenStream tokens = new CommonTokenStream(lex);

          QueryParser parser = new QueryParser(tokens);

          try {
              TokenSource ts = parser.getTokenStream().getTokenSource();
              Token tok = ts.nextToken();
              while (EOF != (tok.getType())) {
                 System.out.println("Got a token: " + tok);
                 tok = ts.nextToken();
              }
          } catch (Exception e) {
              e.printStackTrace();
           }
      }
  }

You should see the lexer break everything apart nicely (I hope ;-) )

hi there "long name" ref:shortname ref:"long name"

Should give:

Got a token: [@-1,0:1='hi',<6>,1:0]
Got a token: [@-1,2:2=' ',<7>,1:2]
Got a token: [@-1,3:7='there',<6>,1:3]
Got a token: [@-1,8:8=' ',<7>,1:8]
Got a token: [@-1,9:19='"long name"',<4>,1:9]
Got a token: [@-1,20:20=' ',<7>,1:20]
Got a token: [@-1,21:33='ref:shortname',<5>,1:21]
Got a token: [@-1,34:34=' ',<7>,1:34]
Got a token: [@-1,35:49='ref:"long name"',<5>,1:35]

I'm not 100% sure what the problem is with your grammar, but I suspect the issue relates to your definition of a LONG_NAME without the quotes. Perhaps you can see what the distinction is?

Thanks for the answer, but it doesn't eliminate helper characters like quotes and `ref:` tag. — Andrey Agibalov, Jan 22 '12 at 08:20

Add quoted strings support to Antlr3 grammar

2 Answers2