tmLanguage syntax highlighting with begin-end rules without highlighting a begin that doesn't have an end

Question

I am creating a simple postfix programming language. The syntax is as follows: 2 3 add adds the two integers 2 and 3 together. "hello, world!" puts puts the string "hello, world!" to STDOUT.

The defining of new functions that can be used, is done as follows:

"fourth_pow"
  "This function raises the argument on the top of the stack to the fourth power."
  [
    dup dup dup mul mul mul
  ]
def

Whitespace and newlines are not important in the language grammar. This means that above definition could also have been made as "fourth_pow" "..." [ dup dup dup mul mul mul] def (but of course, this is less readable).

I now want to highlight the syntax of this language, such that in a definition statement as above, the newly defined function name and the def keyword are highlighted.

A snippet from the .tmLanguage file (in YAML-format) is:

definition:
    name: "entity.other.function.jux-beta_syntax"
    begin: ("([a-zA-Z_][\w]*[?!]?)")
    end: (def)
    beginCaptures:
      '1': {name: "entity.name.function.jux-beta_syntax"}
    contentName: "support.constant.jux-beta_syntax"
    patterns:
    - include: '#comment'
    - include: '#quotation_start'
    - include: '#quotation_end'
    - include: '#string'
    - include: '#identifier'
    - include: '#integer'

(See the whole file here)

This 'works', but it means that when at the end of the file, I start a new string, it is highlighted as if it would be a function definition. This because the 'begin' part of the #definition rule matches. But what I want to happen, is to only colour it if the match could be closed.

Is there a way to do so using the tmLanguage format?

score 1 · Answer 1 · answered Aug 25 '16 at 23:39

The major problem is that, at least in Sublime Text (I haven't tested TextMate or VSCode), the regex engine can only match one line at a time using .tmLanguage files. Therefore, something like this:

(?m:("([a-zA-Z_][\w]*[?!]?)")(.*)(def))

using the ?m: multiline option (. matches newlines) won't work, although it works just fine in Ruby (the regex engine is based on Oniguruma). Because syntax highlighting only goes one line at a time, your beginCaptures matches as soon as a double-quoted string is entered. Of course, once the body of the function and def are written, everything highlights appropriately.

Unfortunately, I'm not fluent enough in it to give you a relevant working example, but you may want to check out the new sublime-syntax format in the most recent versions of Sublime Text 3. It functions using a stack, so you can push and pop contexts, allowing for matching on multiple lines. In the Selected Examples section, look at Bracket Balancing as an example - it highlights closing parentheses ) without a matching opening paren (.

Good luck!

tmLanguage syntax highlighting with begin-end rules without highlighting a begin that doesn't have an end

1 Answers1