sparkles:syntax — Specification

Abstract

sparkles:syntax highlights source code for terminals, web pages, and GPU text renderers through one pipeline. A parsing engine reduces a buffer to a stream of events that assign each span of text a role, such as keyword or string. A theme maps roles to styles once, and interchangeable backends fold the stream into ANSI escape sequences, HTML, or runs of text with their resolved styles. Engines and backends meet only at the stream, so one theme styles every output alike. Highlighting is total. A query pattern the engine cannot evaluate is disabled alone; a missing grammar or an exhausted size or time limit leaves the whole buffer uncolored; and a backend given a stream renders all of it.

Introduction

Sparkles shows code in many places: the hue viewer in a terminal and in a GPU window, the documentation site's static HTML, test reports, and diffs. Each place wants the same code colored the same way under one theme, but in a different medium. A terminal needs escape sequences, a web page needs markup, and a GPU text engine needs resolved style data with neither.

No existing highlighter covers that matrix. bat prints ANSI but has no HTML and no syntax-tree mode; Shiki produces HTML but no ANSI; the tree-sitter highlighter has a precise engine but no product layer around it; editors keep their highlighters to themselves. Two engine families compete as well. The fast family tokenizes line by line with TextMate-style regular-expression grammars; the precise family parses the whole buffer with tree-sitter and queries the syntax tree with pattern files. Their grammars name roles in overlapping but different dialects, and a highlighter, unlike a parser, may never reject its input.

The library is engine-agnostic in the middle and pluggable at both ends. Every engine emits one highlight event stream: an ordered, balanced sequence of "enter role", "leave role", and "text range" events over byte offsets. Roles are highlight labels, hierarchical dotted names such as keyword.control. A theme resolves against the label vocabulary once, by longest dotted prefix, so a label without a rule of its own inherits its parent's style and every later lookup is an index.

Renderers are folds over the stream, and each output line they write is valid on its own, with every open style closed at the line's end and reopened on the next. The first engine is the precise one. It ports the semantics of tree-sitter's reference highlighter, including injections: languages embedded in another, such as code fences in markdown. Guards bound the size, parse time, and query work of every run. Under the totality law, an engine that fails reports an error instead of a stream, and the caller shows the source uncolored rather than half-colored.

This page covers the event stream, the label vocabulary, themes and colors, the ANSI and HTML backends, the styled-run data a GPU backend consumes, the tree-sitter engine with its guards, and how grammars are supplied: pinned through Nix, never vendored. The C binding itself is sparkles:tree-sitter, and how hue or the documentation site lays highlighted code out is theirs to specify. The package also hosts a structural markdown model that walks the syntax tree directly instead of reading the event stream; this page does not specify it. Out of scope, each for a reason given in §4, are a fast engine (designed as a second engine behind the same stream), combined injections whose separate fragments parse together as one document, scope-aware local-variable coloring, incremental editor-style re-highlighting, language detection beyond file extensions and fence labels, semantic tokens from a language server, and terminal-capability detection beyond classifying color depth.

§1 states why the library exists, §2 the design decisions, §3 the in-tree code it reuses, §4 the non-goals, and §5 maps each decision to its prior art. Delivery order lives in PLAN.md. Label-vocabulary dialects records how capture names from different grammar dialects reconcile with the vocabulary, and the next-milestones handoff guides the deferred engine and renderer work. The evidence base is the syntax-highlighting cluster of the parsing survey.

1. Why

The survey's capstone ends on an empty slot: nobody ships the full matrix. bat has the CLI product shape but no precise mode and no HTML; Shiki owns the HTML doctrine but has no ANSI; the tree-sitter CLI has the precise engine but no product layer; editors keep their highlighters unreusable. Sparkles has concrete uses on both sides of that matrix:

• ANSI: highlighting D snippets in terminal tooling (test-runner reports, core-cli components, an eventual bat-shaped pager/printer).
• HTML: rendering code for docs and web output (the Shiki role, self-hosted).
• Styled runs as data: the terminal app's rendering trajectory (a future Vulkan-based text engine) consumes resolved styled runs directly — neither escape codes nor markup.

The library is engine-agnostic in the middle and pluggable at both ends: token producers (engines) feed one highlight-event stream; rendering backends fold that stream into ANSI, HTML — or GPU draw data. The first engine is the tree-sitter precise mode (whole-buffer CST + highlights.scm queries, the tree-sitter-highlight semantics); the TextMate-style fast mode (syntect) is a later engine behind the same seam.

2. What — the design center

Each decision cites the prior-art page it reifies and the in-tree code it builds on.

• One event stream as the inter-engine seam. Engines emit HighlightEvent { source(start, end) | push(label) | pop } — offset-based (no slices in stream state), ordered, balanced, and infallible: engine failures surface before/around the stream, never inside it, so renderers are total (the cluster's totality law: the worst legal output is uncolored text). This is the tree-sitter-highlight event model; TextMate scope stacks map onto push/pop at stack deltas, so the future fast engine needs no adaptation layer.

• A canonical, scope-compatible label vocabulary, plus an alias layer. Dotted names (keyword.control, string.special.key, ~65 entries) drawn from the convergence the survey documents: tree-sitter capture names broadly track TextMate scope names, so one theme layer drives both engines. The vocabulary is hierarchical, because resolution is longest-dot-prefix and only a hierarchy gives it somewhere to degrade to. LabelSet interns names to LabelIds at configure time.

  The convergence is only broad: upstream tree-sitter and older neovim queries spell many of the same concepts flat (number, property, text.strong), and those chop to nothing. So capture-name and theme-selector resolution deliberately differ — both use longest-dot-prefix (Helix's semantics, chosen over the reference crate's part-subset rule for predictability), but capture names additionally consult standardAliases at each depth. Selectors are written in our own vocabulary; capture names arrive from a supply chain that spells things several ways. Treating the two name sets as interchangeable is what left a third of the bundle's captures unstyled — see label-vocabulary dialects.

• Themes resolve once, then index in O(1). Theme (ordered selector → StyleSpec rules) resolves against a LabelSet into a flat labelId → StyleSpec table at configure time — the Helix load-time model, not per-token selector scoring. Two built-in themes ship as D data (dark: Catppuccin-Mocha-derived; light: Solarized-Light-derived; both MIT-sourced with attribution).

• A real color type instead of encoding tricks. Color is a sum type { unset, default_, palette(index), rgb } — bat's #RRGGBBAA alpha conventions (alpha 0 = palette index, alpha 1 = terminal default) parsed into structure, not propagated. Depth folding (trueColor → ansi256 → ansi16) reifies bat's tiers; detection ($COLORTERM/$TERM) is a pure classifier + a thin env wrapper, local to this package until a second consumer promotes it to core-cli.

• Renderers are folds; backends are additive. renderAnsi (minimal SGR diffs between runs; reset before every newline and re-open after, so every output line is independently valid — the discipline base.text.ansi's SgrState already models) and renderHtml (close + re-open all open spans at \n, the reference HtmlRenderer rule; inline-style and CSS-class modes, dots→dashes class mapping; Shiki-style CSS-variable multi-theme reserved as a later mode). Both write to any output range via base.text writers.

• The GPU backend is a design constraint today, a milestone later. A Vulkan text engine consumes byStyledSpan(events) (the flattening fold to maximal innermost-wins runs) plus ResolvedTheme lookups — data, not markup. Therefore StyledSpan, byStyledSpan, and ResolvedTheme are public, documented, tested API — the third-backend contract — and FontStyle stays backend-neutral. A toRgb(Color, palette) concretizer and a byStyledLine per-line adapter are recorded seams.

• The precise engine ports the reference semantics 1:1. Parse the buffer (tree-sitter C runtime via ImportC, the ghostty pattern), run the highlights.scm query, walk captures with the reference event loop (ends-pop-before-starts, same-node later-pattern-wins, cancellation every 100 iterations). Text predicates (#eq?, #match?, #any-of? families) are evaluated library-side; unknown predicates disable one pattern with a warning rather than failing the language — a deliberate divergence from the reference's hard error, because our query supply chain spans dialects and a batch highlighter has a plain-text fallback.

• Guards are a checklist, not a choice (the cluster's guard taxonomy): size cap (512 MiB default; hard 2 GiB ceiling — tree-sitter's 32-bit indices), parse budget (500 ms via the 0.25 progress-callback API), query-cursor match limit (256, Helix's tuned value), optional query deadline and host cancellation flag. Degradation is visible (warnings, Expected errors) and never sticky-off.

• Grammars are supplied, not vendored. A Nix bundle (ts-grammars) links <lang>/parser + <lang>/queries/ per language from nixpkgs#tree-sitter-grammars (D built from gdamore/tree-sitter-d, Helix's pin), exported to tests and tools via SPARKLES_TS_GRAMMAR_PATH. Two grammars come from outside nixpkgs, both pinned through pkgs.tree-sitter.buildGrammar: D, and SDLang — the latter maintained in-house at PetarKirov/tree-sitter-sdl because none existed, which is what makes the repo's own dub.sdl files highlight. Query files are consumed as shipped (upstream dialect = the reference semantics we implement); per-language packaging quirks are normalized in the bundle derivation, keeping supply-chain mess out of D code.

3. What it builds on (reuse, don't reinvent)

• sparkles.base.text.writers — the output-range writer conventions and @nogc SGR primitives (writeEscapeSeq) the renderers extend.
• sparkles.base.text.ansi — SgrState/writeSgrReset/byAnsiToken: the per-line-reset idiom, and the round-trip harness the ANSI renderer's invariant tests reuse.
• sparkles.base.buffer — the @nogc event/label stacks and the checkWriter golden-test helper.
• The expected idiom — engine and loader failures are Expected!(T, TsError), never exceptions on the highlight path.
• The ImportC + pkg-config + Nix recipe and its seven in-repo precedents (libs/ghostty first among them) — the binding is libs/tree-sitter, a sourceLibrary with a unique shim stem, libs "tree-sitter" resolved from the flake's nixpkgs.
• The test-runner fast path + skipTest — grammar- dependent tests skip cleanly outside the devshell instead of failing or silently passing.

4. Non-goals (and why)

• No TextMate/fast engine yet. It is the second engine, not the first — the survey's fast/precise split stands, but the precise mode has the cleaner v1 supply chain (compiled grammars from nixpkgs vs a .sublime-syntax interpreter plus an Oniguruma binding plus a YAML subset). The event seam is designed for it; nothing else waits for it.
• Injections landed (M7); no combined injections, no locals. Injection layers (markdown's inline grammar, fenced code blocks, front-matter) ship in M7 via highlightInjected + the layer stack. injection.combined and locals (def/use coloring) stay deferred — the latter a quality bonus for batch rendering.
• No incremental/editor loop. v1 is batch (parse once, highlight once) — the Helix machinery (persistent trees, injection recycling) earns its keep only under an editor contract. The seam (highlightTree over a kept-alive tree, viewport-bounded cursors) exists and is documented, not built.
• No detection cascade. v1 maps fence labels/extensions via canonicalLanguage; the Linguist-shaped strategy cascade and content scoring are a product concern for the future bat-clone, not the library core.
• No semantic tier. The LSP semantic-tokens overlay fold is future work; the event stream's composition law (fast base paints, refinement overlays, failure invisible) already matches it.
• Not a terminal-capability framework. The pure color-tier classifier lives in sparkles.base.term_color (shared by syntax and core-cli); core-cli's TermCaps now carries the resolved colorDepth. Higher-level detection cascades stay out of scope.

5. Prior-art map

Where each design decision comes from in the survey:

Decision	Prior art (survey page)	What to borrow
Event stream source/push/pop	tree-sitter-highlight	the event vocabulary; streaming, early-stop rendering
Scope-compatible dotted labels	ts-highlight · syntect	one theme layer across engines (the stated convergence)
Longest-dot-prefix resolution, resolved once	Helix	load-time theme tables; captures add an alias layer, themes don't
Color tiers + palette encodings	bat	ansi256_from_rgb fold; #RRGGBBAA semantics (as a sum type)
Per-line-valid ANSI	bat · in-tree ansi.d	reset/re-open at \n; SGR state discipline
Per-line-valid HTML, class/inline modes	ts-highlight · Shiki	HtmlRenderer newline rule; structured output doctrine
Multi-theme CSS variables (reserved)	Shiki	--syn-* custom properties, light-dark()
Guard checklist	Helix · Shiki · bat	size cap, parse budget, match limit, cancellation
Predicate posture: degrade, don't fail	Helix (dialect drift)	disable pattern + warn; plain-text fallback stays reachable
Grammar supply as packaged artifacts	Helix	fetch/build pipeline shape, moved into Nix
Detection deferred to a cascade later	Linguist	the composite cascade recorded for the product layer
GPU backend as styled-run consumer	sh-fit · in-tree terminal	data-not-markup third backend; public flatten fold

The milestones that build this — bottom-up, each independently useful — are in PLAN.md.