Syntax highlighter development guide

[duckdoom4]

Before you get started, I'd recommend first reading through the entire official syntax highlight guide. This guide will give you base knowledge about how the vscode provided syntax highlighter system works. I'd recommend also reading through the links provided in the 'TextMate grammars' section, as these do a really good job explaining how it works. I'll provide some additional information on top of those resources.

How patterns work

As mentioned in the guide, vscode uses oniguruma as regex engine (tokenizer). This provides additional pattern matching methods on top of normal regex.

• Match patterns are 'simple' regex matching.
• Repo patterns contain a list of (include) patterns
• Range patterns have a begin and end regex
  • When matching a range pattern, it attempts to match the begin regex in the document. If a match is found, it will then find the end regex.
    • If no end is found, it will match until the end of the file.
    • If it is found it will apply the patterns in it's patterns repo, to the contents meaning the source between the character after the end of begin regex, up to and including the last character of the end regex.
  • Note that this means that if the repo patterns match something that was part of the end regex result, the end match is invalidated and will attempt to find a new end match. This can be very helpful when attempting to match between closing and opening tags (eg, strings, arrays, parenthesis etc.) but it's important to understand this, as it will cause some headaches if this behaviour was unexpected
  • Back references in the end regex will be replaced with begin match results

(You can also use back references in scopes, which will be replaced with the match result.)

Oniguruma also includes some additional anchors that can help. For example\G is the position right after the last match. And thus (?!\G) means, only match if it's not right after the last match. These can be very helpful in some instances.

One warning though, we need these same patterns to also run in typescript for our custom tokenizer, which doesn't support all anchors. (?!\G) is supported in most cases and it will warn where it's not (yet). These anchors could be supported, but requires some custom behavior to be implemented.

After making any edits to the pattern files, please run the token patterns generator python script.

TODO

• Explain common practice in our files
• Explain the build in tokenizer (not used for syntax highlighting at all)

Additional resources

Resources:
Textmate scope/token naming convention:
https://macromates.com/manual/en/language_grammars#naming-conventions

Regex building:
https://regexr.com/