2

I am new to ANTLR and want to clarify some basic concepts.

  1. Is the parser receiving a Token stream from the lexer? If yes, how does the hidden channel concept fit into this Token stream? Does it mean for each token in the stream, it got an attribute showing which channel it belongs to?

  2. I want to access the hidden channel tokens (white-spaces or comments) that is, say, preceding my own token in the parser. I think I must explicitly write some code. Is it through org.antlr.runtime.TokenStream.get()? What parameter value should be given to it?

Brad Mace
  • 27,194
  • 17
  • 102
  • 148
JavaMan
  • 4,954
  • 4
  • 41
  • 69

1 Answers1

4

JavaMan wrote:

Is the parser receiving a Token stream from the lexer?

A TokenStream is wrapped around the lexer. By default, the Parser "polls" this stream for tokens as needed (i.e. the input is not tokenized in one go, but is buffered).

JavaMan wrote:

If yes, how does the hidden channel concept fit into this Token stream? Does it mean for each token in the stream, it got an attribute showing which channel it belongs to?

The default TokenStream will only produce tokens placed on the DEFAULT channel. You can change the channel during parsing however (or read from more than one channel) [1].

JavaMan wrote:

I want to access the hidden channel tokens (white-spaces or comments) that is, say, preceding my own token in the parser. I think I must explicit code it.

Correct, see [1].


[1] How do I get an Antlr Parser rule to read from both default AND hidden channel

Community
  • 1
  • 1
Bart Kiers
  • 166,582
  • 36
  • 299
  • 288