UPDATE (2024-08-06): After lots of discussion (internally, over Discord etc.) regarding not only the ideal syntax for annotations, but also the potential use cases for warn suppression, we've decided to prioritize other, more important improvements for diagnostics, so this PR won't move forward for the time being.

This PR implements warning suppression within Typst source code, and all the underlying features that it requires:

1. Warning identifiers (partially solves Assign Diagnostics with Identifiers #4217 - does not add identifiers for errors yet)
2. Decorator Annotation syntax (used to annotate where warn suppression occurs, indicating which warnings are suppressed)

Regarding internals, also implements some important lexing changes (might conflict a bit with #4364 , but hopefully not very much).

The idea is to make it possible to add more warnings without hurting UX too much, by allowing users to suppress them.

Draft: There are some rough edges in some places, so the PR shouldn't be merged just yet. (Also, many more tests are needed.)

Overview

This is what it looks like:

// @allow unnecessary-stars
#[**]

(Note: Used to be /! allow("unnecessary-stars") in the initial version of the PR)

We can see all three main elements of this PR here:

1. Decorator Annotation syntax: // @name("argument", "argument2") annotates something relevant to the compiler at the source code.
   • This is bikesheddable still, even the name ("directive" could perhaps be a more appropriate name). (Edit: The name has been changed to "Annotation")
   • The idea is that decorators annotations should be nearly as flexible as comments so that they can be inserted in various locations within the source code. Hence why the syntax resembles that of comments as well.
2. Warning identifiers: The warning regarding ** having no effect now has an identifier, "unnecessary-stars". This is bikesheddable and I chose some (in principle, reasonable) initial names for all existing warnings.
3. Warning suppression: The allow decorator, which takes warning identifiers, causes the given warning to be suppressed if it is raised under the decorator. That is, it won't be raised by the compiler at that point.

Of note, warning suppression applies to warnings raised not only directly under the decorator (or nested within what's under it), but also warnings raised in other files (or elsewhere in the same file) which have tracepoints that lead back to the decorator. For now, only calling functions (during eval) adds tracepoints to warnings, meaning that warnings raised during layout won't have tracepoints, or the tracepoints will be incomplete (e.g. warnings in a context { }).

This means that the unnecessary-stars warning will be suppressed in all of the following cases:

// @allow unnecessary-stars
#[**]

// @allow unnecessary-stars
#{
   [**]
   [**]
}

#let func() = {
    [**]
}

// Warning will have a tracepoint to this call
// @allow unnecessary-stars
#func()

But not in the cases below (must be up to one line above):

// @allow unnecessary-stars

#[**]

// @allow unnecessary-stars

#{
   [**]
   [**]
}

Implementation highlights

1. Identifiers were fairly simple to implement, and are contained entirely within diag.rs.
2. For decorator syntax, the following main changes were done:
   1. The lexer was updated such that next() returns a SyntaxNode. It's almost always a leaf node, except when there is an error (then an error is returned - this means take_error and such are now internal) or when lexing a decorator (which returns a node with a subtree).
      • I tried to make very minimal changes here for this PR. This means that most functions still return SyntaxKind, and as such I didn't change raw elements to use this new lexer API. I think this is better suited to a separate PR to better gauge if there could be any possible regressions when implementing that.
   2. The former change was because all decorator parsing is done entirely within the lexer, as it's simpler to work with characters directly in that case.
   3. The parser had to be updated accordingly.
3. For warning suppression itself:
   1. Added Engine::tracepoint which is used to run some code and then immediately attach a tracepoint to the returned errors (through a returned SourceDiagnostic) and warnings (through a temporary Sink, which is created in the function and then merged into the outer engine's Sink).
   2. This is used in impl Eval for FuncCall to add a tracepoint to all warnings raised during a function's evaluation. This allows tracking which function calls led to a warning being raised.
   3. In the Sink, we now store duplicate warnings (relaxed warning deduplication), but only as long as they have differing tracepoints. This is because we need to keep track of all sets of tracepoints for a warning, since an allow decorator above a particular tracepoint might suppress that warning at that point, but the warning might still be raised because it wasn't suppressed at other tracepoint chains.
   4. Created LinkedNode::prev_attached_decorator which searches for the nearest sibling decorator before the given syntax node, but only up to one line before it. That is, decorators two lines above a given node don't apply to it. (To properly implement this, syntax nodes now keep track of if they have 0, 1 or 2 newlines within them.)
   5. Created Sink::suppress_and_deduplicate_warnings which performs suppression by searching the syntax tree near where each warning was raised - as well as near each tracepoint - by checking if there is any decorator sibling before the node at the warning's span that suppresses it (see the previous item). If not, checks the node's parent, and keeps checking parents and their attached decorators until either a decorator suppressing this warning is found, or we reached the root node of the source file.
      • This function also deduplicates warnings which differ only by their tracepoints, which we can do since we now know which sets of tracepoints were suppressed or not (see item 3).
      • There are certainly performance improvements to be had here, e.g. maybe by caching/memoizing the search somehow. In principle, however, this is an initial implementation which seems to work.
   6. That sink function is used at compile to take and return the warnings, so warning suppression always occurs after a compilation.

TODO

A few TODOs before undrafting:

• Adjust syntax
• Why are Windows tests broken?!? (Fixed: we weren't handling \r\n correctly. Thanks @LaurenzV!)
• More tests, including testing context { } and tracepoints
• Improve the Trace trait stuff in diag.rs (I feel like there might be a less convoluted way to do this) Keeping as is for now
• Optimize newline counting in nodes
• Improve the tree search code
• Update Decorator::arguments considering DecoratorName
• Other possible minor improvements