1

I am using ASTs to perform certain lint styled checking in addition to some static analysis and would like to understand how I can access whitespace and other such formatting tokens adjacent to AST nodes. I am currently using Eclipse JDT to generate the ASTs.

I am looking for mechanism in Eclipse JDT, equivalent to the channels mechanism provided by ANTLR; which generates parse trees and has separate channels for tokens streams (hidden channel and the parse tree channel) or a workaround to achieve the same.

np20
  • 1,935
  • 3
  • 16
  • 24

1 Answers1

0

Each token in the AST comes with an index describing it's absolute position in the input stream (0..n). When you examine them you will see that they are not continuous. That is because the hidden tokens are missing. Knowing that you can use your token source to get any token by index, including the hidden ones.

Mike Lischke
  • 48,925
  • 16
  • 119
  • 181
  • Are you referring to `node.getStartPosition()` that gives the index of the first character of the node/token in the source? Meaning analyses of hidden tokens can be done only at character array level of abstraction and not at the 'tokens list' level of abstraction of source code? @MikeLischke – np20 Mar 19 '15 at 10:21
  • No, I mean `getTokenIndex()` in a common token struct/class and `get()` from the token source to get an arbitrary token for a given index. – Mike Lischke Mar 19 '15 at 13:46
  • Perhaps you are referring to the CommonToken class and its method in ANTLR, i.e.: http://www.antlr.org/api/Java/org/antlr/v4/runtime/CommonToken.html. I am looking for a functionality equivalent to the above in Eclipse JDT. Edited the question to clarify this. @MikeLischke – np20 Mar 20 '15 at 11:34
  • Ah, I see. Then my answer is probably of no help, sorry. – Mike Lischke Mar 21 '15 at 09:20