2

I'm currently testing how to create a VS Code extension that would allow me to write files using my own DSL defined in Xtext (*.sample). So far I've been able to create the VS Code extension and use it:

  • Open a file (*.sample) and load the extension.
  • See syntax errors, text prediction, renaming, etc (communication with Xtext LS works fine).
  • See syntax color highlighting by means of a TextMate grammar and a couple of VS Code UI themes I've created.

Now I would like to go a step further and enable semantic highlighting. So far I understand that this is a capability of the LS (see LSP specification 3.16) and the highlighting information comes from the server. However the only information I can find about semantic highlighting and VS Code is the creation of a DocumentSemanticTokensProvider in the VS Code extension, that is, client side (see VS Code documentation). As I understand the Semantic Token Provider returns the tokens with its classification (class, namespace, enum, ...) and modifier (declaration, declaration, ...). From my understanding this would imply some file analysis on the client side, which from my point of view is not the way to go.

On the other side, I'm not able to find any documentation on how to "activate" the semantic highlighting in the server side (Xtext) so the semantic tokens are returned when the client requests them.

As you can see I have lots of doubts on how the semantic highlighting would work in my scenario. Therefore I would like to ask the following questions.

  • Where do I need to define the logic that decides which are the semantic tokens of a file?
  • How does the VS Code extension triggers the calculation of semantic tokens in the server?

Any other comment on whether my assumptions are correct (or not) are welcome.

Many thanks in advance.

Alejandro González
  • 481
  • 9
  • 22
  • VSC can only communicate with the client side (the extension) when VSC wants to have the semantic tokens it calls the provider and that needs to produce all the semantic tokens by doing the analysis with or without the LS. Have you looked at the Semantic Highlight Sample Application – rioV8 Nov 03 '21 at 07:54
  • "when VSC wants to have the semantic tokens it calls the provider and that needs to produce all the semantic tokens by doing the analysis with or without the LS" okay, this means that the provider must be there. Now it would be a matter of triggering a request to the LSP so it returns the list of semantic tokens. I see that the sample application focuses on how the tokens are built and returned. To do so it parses a file which already indicates the type of tokens. I would be more interested moving this logic to the LSP. – Alejandro González Nov 03 '21 at 08:18
  • The XText doc says "LSP does not support syntax highlighting" [here](https://www.eclipse.org/Xtext/documentation/340_lsp_support.html), but LSP 3.16 does. If XText outputs TM files, it probably still doesn't support SH. I wrote a [tool](https://github.com/kaby76/uni-vscode) that inputs an Antlr4 grammar and a list of XPaths that specify SH classes, and outputs an LSP server and extension for VSCode. All the code for SH is in the server, as it should be (i.e., it reads the DSL file, parses it, uses XPath to find nodes in PT, return tokens with SH). Maybe read the code XText/Eclipse generates? – kaby76 Nov 03 '21 at 11:23
  • Yes, the Xtext does not support syntax highlighting and therefore this must be implemented in the VS Code extension by means of the TextMate grammar. My problem comes in the direction of semantic highlighting. I'll take a look to your tool and see if I can learn from it. Thanks for the comment! – Alejandro González Nov 03 '21 at 13:25
  • 1
    contributions to Xtext are welcome https://github.com/eclipse/xtext-core/issues/1612 – Christian Dietrich Nov 04 '21 at 09:23

1 Answers1

2

As stated on Cristian Dietrich's comment, the Xtext LSP implementation does no implement the semantic token feature of the LSP protocol yet (see issue). In my case, in order to be able to get "hardcoded" semantic tokens from the LSP (instead of using the DocumentSemanticTokensProvider from the VS Code extension), I had to extend the Xtext LSP implementation to set a semanticTokensProvider as part of the LSP capabilities:

public class CustomLanguageServerImpl extends LanguageServerImpl {
    
    @Override
    protected ServerCapabilities createServerCapabilities(InitializeParams params) {
        ServerCapabilities capabilities = super.createServerCapabilities(params);
        // Create server capabilities for semantic tokens.
        SemanticTokensWithRegistrationOptions options = new SemanticTokensWithRegistrationOptions();

        List<String> tokenTypes = new ArrayList<String>();
        tokenTypes.add(SemanticTokenTypes.Namespace);
        ...
        
        List<String> tokenModifiers = new ArrayList<String>();
        tokenModifiers.add(SemanticTokenModifiers.Declaration);
        ...

        options.setLegend(new SemanticTokensLegend(tokenTypes, tokenModifiers));
        options.setRange(false);
        options.setFull(true);

        capabilities.setSemanticTokensProvider(options);

        return capabilities;
    }
}

By using the new LSP implementation, I'm able to see hardcoded semantic tokens in my VS Code extension. My guess is that the "handshaking" process executed by the VS Code extension and the LSP (during initialization) makes the VS Code extension to request the semantic tokens based on the configuration (full/range). This can be seen in the messages exchanged between client and server:

   {jsonrpc: '2.0', id: 0, method: 'initialize', params: {…}}
   {jsonrpc: '2.0', id: 0, result: {…}}
   {jsonrpc: '2.0', method: 'initialized', params: {…}}
   {jsonrpc: '2.0', method: 'textDocument/didOpen', params: {…}}
   {jsonrpc: '2.0', method: 'textDocument/publishDiagnostics', params: {…}}
   {jsonrpc: '2.0', id: 1, method: 'textDocument/documentSymbol', params: {…}}
   {jsonrpc: '2.0', method: 'textDocument/publishDiagnostics', params: {…}}
   {jsonrpc: '2.0', id: 1, result: Array(1)}
   {jsonrpc: '2.0', id: 2, method: 'textDocument/documentSymbol', params: {…}}
   {jsonrpc: '2.0', id: 2, result: Array(1)}
-> {jsonrpc: '2.0', id: 3, method: 'textDocument/semanticTokens/range', params: {…}}
   {jsonrpc: '2.0', id: 4, method: 'textDocument/foldingRange', params: {…}}
   {jsonrpc: '2.0', id: 3, result: {…}}
   {jsonrpc: '2.0', id: 4, result: Array(2)}

Using the default Xtext LSP implementation, the textDocument/semanticTokens/* request is not even triggered.

Alejandro González
  • 481
  • 9
  • 22