1

I am trying to implement language support plugin for a basic language. I am following jetbrains's tutorial for simple language support (.properties file support basically) and on the side I have rust plugin for reference. However the complexity gap between them is huge so some questions are hard to find answers to from both sources.

Here is my question: What is the best way to allow spaces between tokens which DO NOT require spaces, however force spacing between tokens which do?

i.e. class Foo{ <- here first space is mandatory, but the second one (before '{' symbol) can be ommited.

Levan Apakidze
  • 221
  • 2
  • 6
  • If you don't get the answer here, the best place to ask would be https://intellij-support.jetbrains.com/hc/en-us/community/topics/200366979-IntelliJ-IDEA-Open-API-and-Plugin-Development. – CrazyCoder Jul 28 '22 at 16:13
  • thank you! should've considered there would be some jetbrains platform to discuss this as well :D – Levan Apakidze Jul 28 '22 at 18:14

2 Answers2

0

That is a task for the Parser, not the Lexer. In fact the Lexer lexes the whitespaces but does not pass them to the Parser. To enforce or not enforce the whitespaces in your example you need to implement a .bnf file with the Grammar-Kit extension https://github.com/JetBrains/Grammar-Kit.

common sense
  • 3,775
  • 6
  • 22
  • 31
0

In general, parser doesn't handle whitespaces. In com.intellij.lang.ParserDefinition, you can find:

@NotNull
default TokenSet getWhitespaceTokens() {
    return TokenSet.WHITE_SPACE;
}

Returns the set of token types which are treated as whitespace by the PSI builder. Tokens of those types are automatically skipped by PsiBuilder.

Your Lexer finds all white characters and marks them as TokenType.WHITE_SPACE. If there is no whitespace character in your example, it will simply treat the whole classFoo as a single unknown token.

In your implementation of com.intellij.lang.ParserDefinition, you can override method:

default @NotNull SpaceRequirements spaceExistenceTypeBetweenTokens(ASTNode left, ASTNode right)

to return MAY, MUST, MUST_NOT or MUST_LINE_BREAK depending on the type of nodes.

You can also specify more advanced spacing for auto-formatting in your FormattingModelBuilder described in the tutorial here: https://plugins.jetbrains.com/docs/intellij/formatter.html

Slav
  • 786
  • 1
  • 13
  • 25