Sparkles

Appearance

LSP Semantic Tokens (protocol) ​

The semantic tier of the highlighting stack, standardized: a Language Server Protocol request family (since LSP 3.16, December 2020) that lets a compiler-grade out-of-process server attach symbol-aware labels (parameter, readonly, defaultLibrary) to ranges a syntactic highlighter cannot classify — delivered as integer-encoded, delta-updatable token arrays and, in the reference client (VS Code), rendered as an overlay on a TextMate base. Within the highlighting cluster this is the tier above both the TextMate model and CST queries; it is a wire protocol, not an engine.

FieldValue
LanguageProtocol specification (JSON-RPC; markdown spec sources)
LicenseSpec repository: Creative Commons / MIT mix per Microsoft OSS practice (see repo); protocol itself is an open standard
Repositorymicrosoft/language-server-protocol (_specifications/lsp/3.17/language/semanticTokens.md)
DocumentationLSP specification
Key authorsMicrosoft (VS Code / LSP team); Dirk Bäumer et al.
CategorySyntax highlighting — semantic tier (protocol)
Algorithm / grammar classNone — a transport for server-computed classifications; the server runs a real front-end (name resolution, types)
Lexing modeln/a — tokens arrive pre-classified; positions are UTF-16 line/character pairs, relative-encoded
OutputSemanticTokens { resultId?, data: uinteger[] } — 5 integers per token; full, full/delta, and range request variants + a refresh signal
Highlighting / theme modelA negotiated legend: predefined 23 token types + 10 modifiers (3.17), extensible; clients map legend entries to theme rules
Latest releaseLSP 3.17 (stable; decorator type added); 3.18 draft adds label; pinned spec checkout 25005c8 (2026-07-09)

NOTE

This deep-dive surveys the semanticTokens request family as specified in the pinned LSP spec sources (3.17 file, with 3.16 introduction and 3.18 draft noted) plus the client-side merge semantics that make it a tier rather than a replacement. Server implementations (rust-analyzer, clangd, the TypeScript server) and VS Code's renderer internals are referenced, not catalogued; the in-process IDE alternative to this design is the IntelliJ Platform page, and the syntactic tiers it layers onto are the rest of the cluster.

Overview ​

What it solves ​

Syntactic highlighters — TextMate engines, CST queries — classify what text is structurally; they cannot know what it refers to. Is this identifier a parameter or a global? Is that method static, async, deprecated? Answering requires name resolution and type information, which lives in a language server. The spec states the purpose and its defining engineering constraint in one paragraph (semanticTokens.md:5):

"Semantic tokens are used to add additional color information to a file that depends on language specific symbol information. A semantic token request usually produces a large result. The protocol therefore supports encoding tokens with numbers. In addition optional support for deltas is available."

Design philosophy ​

  1. A tier, not a replacement. The protocol assumes a fast syntactic base already painted the file; semantic tokens refine it. The client capability that encodes this is explicit (semanticTokens.md:261-273): "Whether the client uses semantic tokens to augment existing syntax tokens. If set to true client side created syntax tokens and semantic tokens are both used for colorization. If set to false the client only uses the returned semantic tokens for colorization." (augmentsSyntaxTokens, @since 3.17). VS Code's shipping behavior — TextMate base, semantic overlay, blended per theme rules — is the two-layer architecture in production; it turned semantic highlighting on for TypeScript/JavaScript in v1.43 (February 2020 release).
  2. The wire format is shaped by scale. "A semantic token request usually produces a large result" — so tokens are integers, not objects; positions are relative, not absolute; and updates are array edits, not re-sends. Every choice follows from the size of the payload.
  3. Vocabulary is negotiated, not fixed. Token types/modifiers are strings in a legend declared by the server and matched against client capabilities: "The protocol defines a set of token types and modifiers but clients are allowed to extend these and announce the values they support in the corresponding client capability." (semanticTokens.md:9) — a middle path between TextMate's open scope strings and @lezer/highlight's closed tag set.

How it works ​

The legend: 23 types × 10 modifiers ​

A token is "one token type combined with n token modifiers" (semanticTokens.md:9). The predefined SemanticTokenTypes (3.17, 23 entries — decorator arrived in 3.17, the 3.18 draft adds label for a 24th): namespace, type, class, enum, interface, struct, typeParameter, parameter, variable, property, enumMember, event, function, method, macro, keyword, modifier, comment, string, number, regexp, operator, decorator — with type documented as "a fallback for types which can't be mapped to a specific type like class or enum". The 10 SemanticTokenModifiers: declaration, definition, readonly, static, deprecated, abstract, async, modification, documentation, defaultLibrary. Servers declare which subset they emit in a SemanticTokensLegend; the integers on the wire index into it.

Five integers per token, relative ​

The encoding rationale is stated outright (semanticTokens.md:100):

"The protocol for the token format relative uses relative positions, because most tokens remain stable relative to each other when edits are made in a file. This simplifies the computation of a delta if a server supports it. So each token is represented using 5 integers."

Per token i: deltaLine (line, relative to the previous token), deltaStart (start character, relative to the previous token's start when on the same line), length, tokenType (legend index; "We currently ask that tokenType < 65536"), tokenModifiers (a bitset over the modifier legend — "a tokenModifier value of 3 is first viewed as binary 0b00000011, which means [tokenModifiers[0], tokenModifiers[1]]", semanticTokens.md:97-98). A whole file is one flat uinteger[] — cache-friendly, alloc-light, and language-neutral.

Full, delta, range, refresh ​

Four verbs cover the lifecycle:

  • textDocument/semanticTokens/full — the whole file; the response may carry a resultId: "If provided and clients support delta updating the client will include the result id in the next semantic token request. A server can then instead of computing all semantic tokens again simply send a delta."
  • full/delta — the client sends previousResultId; the server answers with SemanticTokensEdit { start, deleteCount, data? } operations on the raw integer array: "The delta is now expressed on these number arrays without any form of interpretation what these numbers mean" (semanticTokens.md:172) — relative positions make most of the array survive an edit unchanged. Clients "must not assume that they are sorted" and should apply edits back-to-front.
  • range — the viewport tier, with two spec'd use cases (semanticTokens.md:460-463): faster first paint "when a user opens a file", and as the only offering "if computing semantic tokens for a full document is too expensive". The server may answer with a broader range than requested.
  • workspace/semanticTokens/refresh — server→client: re-request everything (e.g. after "a project wide configuration change").

Merge semantics and rendering constraints ​

Two client capabilities bound what servers may emit: multilineTokenSupport ("If multiline tokens are not supported and a tokens length takes it past the end of the line, it should be treated as if the token ends at the end of the line and will not wrap onto the next line") and overlappingTokenSupport — by default tokens are single-line and non-overlapping, i.e. exactly the shape a line-oriented renderer (or Shiki-style token list) already handles. The overlay itself is theme-driven: in VS Code, semantic token rules sit beside TextMate scope rules in the theme, and where a semantic token exists it wins (or blends) over the syntactic base — the practical answer to "how do two highlighting tiers coexist without flicker": the base paints instantly, the semantic pass re-paints asynchronously when the server catches up.

Algorithm & grammar class ​

  • No algorithm — a contract. All classification intelligence lives server-side, behind a real front-end (rust-analyzer's HIR, clangd's AST, tsserver's checker). The protocol only standardizes labels, positions, encoding, and update discipline. This is the survey's cleanest separation of classification from rendering.
  • Precision class: true semantics. Everything below this tier is context-free: TextMate sees a line + stack; tree-sitter queries see the parse tree; locals-tracking approximates scoping textually. Semantic tokens see the program — resolved names, types, modifiers — the only tier that can color parameter vs variable vs property correctly in every case.
  • Latency class: asynchronous by design. The server may be slow; the protocol embraces it (deltas, range-first, refresh) rather than pretending otherwise. Highlighting becomes eventually-consistent — a fundamental UX contrast with the synchronous tiers.

Interface & composition model ​

  • Composition is the whole point: augmentsSyntaxTokens codifies the two-layer stack — syntactic base (any engine) + semantic overlay (any server) — with the theme as the merge point. A tool with its own fast tier keeps it; the semantic layer is additive.
  • Legend negotiation decouples vocabularies: servers can extend types/modifiers, clients declare what they understand, unknown labels degrade to nothing (never an error).
  • Transport-agnostic classifications: because tokens are (range, type, modifiers) triples with no engine coupling, any consumer — an editor, a bat-style CLI talking to a language server, an HTML renderer — can fold them over its own token stream; the encoding is deliberately trivial to decode.
  • The request family is the API surface: full/delta/range/refresh map exactly onto editor lifecycle events (open, edit, scroll, config change) — a ready-made checklist for any client implementation.

Performance ​

  • Payload engineering: flat integer arrays (no JSON object per token), relative positions, < 65536 type indices — the spec's own numbers-first framing. A 10k-line file's tokens fit in one compact array.
  • Deltas amortize edits: relative encoding means an edit shifts only nearby tokens' deltas; the resultId + array-edit protocol re-sends only the changed slice.
  • range bounds first paint — semantic highlighting for the viewport before the whole file is analyzed; the same viewport-first discipline as Helix's windowed queries and Emacs' jit-lock, expressed at the protocol level.
  • The unavoidable cost is server latency: classification quality is bought with an out-of-process round-trip and whatever analysis time the server needs; the base tier exists precisely to hide it.

Highlighting & theme model ​

This is the extra spine dimension for the syntax-highlighting cluster:

  • Label vocabulary — a negotiated legend of type + modifier-bitset tuples: structured like @lezer/highlight's tags (not free-form strings), extensible like TextMate scopes (unlike Lezer's closed set), and deliberately small (23+10) because themes must cover it.
  • Inter-unit state — resultId chains: the client holds the previous array; the server diffs against it. State is per document per server, checkpointed at every response — the semantic tier's analogue of grammar-state checkpointing.
  • Theme resolution — client-side, layered: legend entries resolve through theme rules that coexist with (and outrank) the syntactic base's rules; augmentsSyntaxTokens decides blend-vs-replace. No theme format is specified — the protocol stops at labels.
  • Rendering targets — whatever the client renders: by default single-line, non-overlapping tokens (the capabilities gate anything richer), so the output contract matches every line-oriented backend in this survey, ANSI included.

Error handling & recovery ​

  • Degrades to the base tier. No server, slow server, or an error → the syntactic base simply remains; semantic color is progressive enhancement by construction. This is the strongest never-fail story in the cluster because failure is invisible.
  • Stale-result discipline: deltas are only valid against the exact previousResultId; on mismatch the server returns a full result. refresh handles world-changes (config, dependencies).
  • Spec'd edge behavior (truncate-at-EOL for multiline, no overlap by default) keeps malformed emissions renderable rather than erroneous.

Ecosystem & maturity ​

  • Introduced in LSP 3.16 (December 14, 2020 — "Add semantic token support" in the spec changelog), extended in 3.17 (decorator, server-cancelable, augmentsSyntaxTokens), still growing in the 3.18 draft (label).
  • Server support is broad: rust-analyzer, clangd, tsserver (via VS Code), gopls, jdtls — semantic tokens are now table stakes for serious language servers; client support spans VS Code, Neovim (built-in LSP), Helix, Eclipse, and editors beyond.
  • The de-facto merge reference is VS Code: default-on for TS/JS since v1.43 (with the 1.43.1 walk-back scoping it to themes that opt in) — the practical proof that a TextMate base + semantic overlay is deployable at scale.
  • For a D tool, this tier is the eventual bridge to compiler-grade highlighting (DMD-as-a-library / DCD-style analysis) without abandoning the fast tiers — implement the client fold now, gain semantics whenever a server exists.

Strengths ​

  • The only true-semantics tier — colors what syntax cannot know (parameter vs variable, readonly, deprecated, defaultLibrary), from real name resolution.
  • Engine-agnostic layering: composes over any syntactic base via augmentsSyntaxTokens; failure is invisible degradation.
  • Payload discipline: integer arrays + relative positions + array-level deltas + viewport ranges — a complete, spec'd answer to "semantic data is big and edits are constant".
  • Negotiated vocabulary with sane defaults: small enough to theme, extensible enough for real languages.
  • Protocol, not product: every editor and server interoperates; classifications outlive any single engine.

Weaknesses ​

  • Not self-sufficient: needs a language server per language and a syntactic base underneath — it is the third tier, never the only one.
  • Eventually consistent: colors arrive after analysis; themes/users must tolerate the repaint (and the 1.43.1 walk-back shows even the reference client tread carefully).
  • UTF-16 positions (LSP legacy) — a persistent mismatch tax for byte-oriented implementations.
  • Coarse vocabulary: 23 types + 10 modifiers is far below TextMate's scope granularity; fine distinctions need non-standard legend extensions, which themes won't know.
  • No theme/merge standardization: blend behavior beyond the one boolean is client-defined — cross-editor rendering differs.

Key design decisions and trade-offs ​

DecisionRationaleTrade-off
Overlay tier, not standalone (augmentsSyntaxTokens)Instant syntactic paint + eventual semantic refinement; failure invisibleTwo systems to theme coherently; repaint visible on slow servers
Integer legend encoding"usually produces a large result" — compact, alloc-light, language-neutralOpaque on the wire; debugging needs the legend; type index capped at 65536
Relative positionsToken deltas stay stable under edits → cheap diffsRandom access requires a prefix scan; absolute formats never materialized
Array-level deltas + resultIdRe-send only changed slices, uninterpretedClient must hold previous arrays and apply unsorted edits back-to-front
range requestViewport-first paint; an escape hatch when full analysis is too expensiveServer may over-answer; clients must reconcile range and full results
Negotiated, extensible legendReal languages outgrow 23 types; capability handshake keeps both sides honestExtensions are invisible to themes that don't know them
Single-line, non-overlapping by defaultMatches every line-oriented renderer; richer shapes are opt-in capabilitiesMultiline constructs (raw strings) get clipped on conservative clients

Sources ​