Chroma (Go)
The proof that a highlighting corpus can be ported, not rewritten: Chroma is Pygments' engine re-implemented in Go with the lexer corpus machine-translated to XML data files (282 of them, plus 19 hand-written Go lexers where translation fails), running on a backtracking regex engine with a per-match timeout Pygments never had. It powers Hugo's and Gitea's highlighting, and it is the closest existing analogue to what a sparkles:syntax fast mode would attempt — a native engine consuming someone else's grammar corpus.
| Field | Value |
|---|---|
| Language | Go (~13.2 kLOC — ~10× smaller than Pygments because lexers became data) |
| License | MIT (© 2017 Alec Thomas) |
| Repository | alecthomas/chroma |
| Documentation | README.md (the project's own docs) |
| Key authors | Alec Thomas (creator, 2017) + contributors |
| Category | Syntax highlighting — ported lexer corpus (native engine over translated Pygments lexers) |
| Algorithm / grammar class | Pygments' regex state machine, interpreted from XML rule files; whole-text scan with \G-anchored patterns |
| Lexing model | dlclark/regexp2 (a .NET-compatible backtracking engine) — deliberately not Go's stdlib RE2, which lacks backreferences/lookaround |
| Output | iter.Seq[Token] (v3) → formatters: HTML (classes or inline), ANSI 8/16, 256, truecolor, SVG, JSON |
| Highlighting / theme model | Pygments' token hierarchy re-encoded as integer ranges (Keyword = 1000, NameBuiltin = 2100) with arithmetic parent lookup |
| Latest release | Stable v2.27.0; pinned checkout 01c740b (2026-07-08) = v3.0.0-alpha.5 (v3 = iter.Seq API + module path bump) |
NOTE
This deep-dive surveys the engine (regexp.go, types.go, formatters) and the porting pipeline (_tools/pygments2chroma_xml.py, the XML lexer format) at the pinned checkout. The upstream corpus semantics are the Pygments page; Chroma's interest for this survey is precisely the delta: what surviving translation costs, and what a fresh native implementation added (guards) or simplified (token encoding).
Overview
What it solves
"Chroma takes source code and other structured text and converts it into syntax highlighted HTML, ANSI-coloured text, etc." and — the strategy in one sentence — "Chroma is based heavily on Pygments, and includes translators for Pygments lexers and styles." (README.md:11-15). Go's ecosystem (Hugo above all) needed batch highlighting without a Python runtime; rather than author hundreds of grammars, Chroma imported them.
Design philosophy
- Port the corpus, keep the model. Lexers become XML serializations of Pygments'
tokensdicts, generated by a script that imports the actual Pygments lexer class and renders it through a template (_tools/pygments2chroma_xml.py): "In many cases lexers can be automatically converted directly from Pygments by using the included Python 3 script" (README.md:229-237); "All Pygments styles have been converted to Chroma" likewise. The engine then re-implements Pygments' semantics faithfully enough that the data files just work. - Be honest about the residue. The README's "What's missing compared to Pygments?" section is the survey's best first-hand account of porting cost: "Pygments lexers for complex languages often include custom code to handle certain aspects, such as Raku's ability to nest code inside regular expressions. These require time and effort to convert." — plus "I mostly only converted languages I had heard of, to reduce the porting cost." and "Though the Chroma API supports content detection, very few languages support them." The 19 hand-written Go lexers are exactly the custom-code residue.
- A fresh engine gets fresh guards. Where Pygments trusts its grammars, Chroma bounds them: a 250 ms per-match regex timeout, a zero-width-match force-advance, and a mutator limit — defensive engineering absent upstream.
How it works
Lexers as XML
A grammar file is a direct serialization of the Pygments model (lexers/embedded/*.xml): a <config> block (name, aliases, filename globs, MIME types) and <rules> with named <state>s of <rule pattern="…"> entries emitting <token type="GenericDeleted"/> and stack operations — the tokens dict, one-to-one, as data. 282 XML lexers + 19 Go lexers (Astro, Go itself, HTML, Haxe, Genshi, ERB, … — the ones needing code) + 74 converted styles. The conversion script runs upstream Pygments as the source of truth, so corpus refreshes are re-runs, not rewrites — the porting pipeline a D fast mode would want, demonstrated end to end.
The engine: regexp2, \G, and guards
Go's stdlib regexp is RE2 — linear-time but without backreferences or lookaround, which Pygments patterns use freely. Chroma therefore uses dlclark/regexp2, a backtracking .NET-compatible engine, and pays the backtracking risk deliberately — then bounds it (regexp.go:352-373): every pattern is compiled anchored with \G (match exactly at the current position — the equivalent of Pygments' re.match(text, pos)), and rule.Regexp.MatchTimeout = time.Millisecond * 250 puts a hard ceiling under every match attempt. The tokenizer is the same whole-text stateful-stack loop as upstream (for l.Pos < end && len(l.Stack) > 0, first match wins, advance by the match), plus a guard Pygments lacks: "A zero-width match that did not change state will never advance" — force-advancing one Error char instead of looping.
Tokens as integers
Pygments' tuple-singleton hierarchy becomes flat integer constants with range-encoded parentage (types.go): Keyword = 1000, Name = 2000, NameBuiltin = 2100, Literal = 3000; TokenType.Parent() is arithmetic (t % 100 != 0 → t/100*100, else t/1000*1000, else 0). Style resolution inherits accordingly — "when CommentSpecial is not defined, Chroma uses the token style from Comment" (README.md:267) — the same parent-walking theme completeness as upstream, at integer-compare cost. A Coalesce filter merges adjacent same-type runs (capped at 8192 chars).
Formatters and detection
Formatters: HTML (classes via WithClasses + WriteCSS, or inline styles; light/dark scoping via .chroma.dark mode classes), terminal at three tiers (tty_indexed.go for 8/16/256, tty_truecolour.go), SVG, JSON, and debug emitters — "Chroma supports HTML output, as well as terminal output in 8 colour, 256 colour, and true-colour." (README.md:243). Palette downsampling uses a redmean weighted-Euclidean distance (colour.go:62-69) — perceptually better than Pygments' plain Euclidean (though doc comments claiming "the Lab colour space" oversell what the code does — a doc-vs-code discrepancy worth noting).
Detection mirrors the upstream API with less muscle: lexers.Match(filename) glob-matches with an honest doc ("Note that this iterates over all file patterns in all lexers, so is not fast."), and Analyse(text) exists but — per the README — "very few languages support them. I have plans to implement a statistical analyser at some point, but not enough time." In practice Chroma detects by filename + priority; content analysis is the corpus feature that didn't survive the port (Pygments' analyse_text functions are Python code — exactly the untranslatable part).
Error handling, with attribution
The no-match fallback is copied from upstream, comment and all (regexp.go:238-255): "// From Pygments :\ // If the RegexLexer encounters a newline that is flagged as an error token, the stack is emptied and the lexer continues scanning in the 'root' state. This can help producing error-tolerant highlighting for erroneous input…" — newline resets the stack to root; otherwise one-char Error advance. Cross-implementation recovery semantics, preserved verbatim.
Algorithm & grammar class
- Pygments' machine, interpreted from data: ordered-choice regex rules over a state stack, whole-text position-anchored scan — semantics preserved so the translated corpus behaves; see Pygments for the model itself.
- The engine choice is forced by the corpus: backrefs/lookaround in Pygments patterns rule out RE2;
regexp2's backtracking is the price of compatibility, and the 250 ms timeout is the mitigation. The same corpus-dictates-engine dynamic as Shiki's Oniguruma dependency, resolved with a timeout instead of WASM/transpilation. - The residue defines the boundary: whatever needed Python callbacks becomes a hand-written Go lexer or is dropped — quantifying (282 vs 19 + omissions) how much of a lexers-as-code corpus is actually data in disguise: most of it.
Interface & composition model
- Four axes, like upstream: lexers (registry + XML loading), styles, formatters, and a
Coalesce/filter stage; v3'sTokenise(...) (iter.Seq[Token], error)makes the stream a native Go iterator. - The conversion pipeline is part of the product:
_tools/pygments2chroma_xml.py+style.pykeep the corpus refreshable from upstream — the maintained seam between ecosystems. - Embeds where Go embeds: Hugo ("a static site generator that uses Chroma for syntax highlighting code examples"), Gitea,
moor(a pager), and a documentedless(1)LESSOPENintegration with--failpassthrough — the CLI/pager composition patterns a bat-shaped tool recognizes.
Performance
- Guarded where upstream is not: 250 ms per-match timeout, zero-width force-advance, mutator limit (10 000) — the complete pathological-input checklist Pygments lacks, in ~13 kLOC.
- No comparative benchmarks published; compiled-language interpretation of the same rule sets, with regexp2's backtracking as the variable cost. The corpus's regex quality — inherited from upstream — remains the true bound.
- Batch, whole-text, no incrementality — same posture as upstream;
ensure_nl, coalescing, and iterator output are stream conveniences, not architecture changes.
Highlighting & theme model
This is the extra spine dimension for the syntax-highlighting cluster:
- Label vocabulary — Pygments' taxonomy, integer-encoded: the same hierarchy and short-name semantics, with parentage as arithmetic on range-coded constants — a compact re-encoding a
@nogcD implementation would recognize as the right shape (compare syntect's 16-byte packed scopes: same instinct, different structure). - Inter-unit state — none exposed (whole-text scan, as upstream);
LexerStateis internal to aTokenisecall. - Theme resolution — ported styles + parent-walk inheritance, with a
Backgroundtoken for defaults; 74 Pygments styles translated wholesale — themes survived the port even better than lexers. - Rendering targets — HTML (classes/inline, light/dark mode classes), three ANSI tiers, SVG, JSON; redmean-weighted palette downsampling. Both backends a dual-output tool needs, in one small codebase.
Error handling & recovery
- Upstream's recovery, verbatim (newline→root reset, one-char
Errorskip) — with the source comment crediting Pygments. - Plus real guards: the match timeout converts catastrophic backtracking from a hang into a degraded region; the zero-width guard converts a grammar bug from an infinite loop into one
Errorchar. - Detection degrades to filename: absent
analyse_textports, ambiguous content falls back to glob/priority — wrong-language highlighting is the failure mode, never an error.
Ecosystem & maturity
- Created June 2017 (first tag September 2017), stable on v2 (
v2.27.0), with v3 (pinned here as alpha) modernizing the API to Go 1.23 iterators — steady single-maintainer-plus-community cadence. - Hugo made it infrastructure: every Hugo site's code blocks are Chroma output; Gitea and Go tooling follow — the Go ecosystem's default, as Pygments is Python's.
- Corpus freshness is a process, not a property: lexers advance when someone re-runs the converter against upstream — the structural cost of a ported corpus, worth pricing into any similar plan.
Strengths
- The porting playbook, demonstrated: corpus-as-data extraction from a lexers-as-code ecosystem, with the converter maintained in-tree and the residue (hand-written lexers) explicitly bounded.
- Guards upstream never had: per-match timeout + zero-width guard + mutator limit — the adversarial-input posture done right in a small engine.
- Compact token encoding (integer ranges, arithmetic parents) — the taxonomy's semantics at native cost.
- Both output families (HTML + three ANSI tiers) with ported themes and light/dark support.
- Honest documentation of fidelity gaps and slow paths — rare and valuable for downstream planners.
Weaknesses
- Fidelity is bounded by translation: complex Pygments lexers (custom code) are hand-ported, simplified, or absent; corpus coverage is "languages I had heard of".
- Content detection effectively unimplemented — the
analyse_textintelligence didn't survive the port; detection is filenames. - Backtracking engine by necessity: regexp2 + timeout is mitigation, not linearity; a hostile pattern still costs 250 ms per rule per position in the worst case.
- Corpus staleness risk: upstream moves; the XML snapshot moves only on converter re-runs.
- Batch-only, like its parent — no incrementality, no checkpointing, no editor story.
Key design decisions and trade-offs
| Decision | Rationale | Trade-off |
|---|---|---|
| Machine-translate the Pygments corpus to XML | Hundreds of languages for the cost of a converter script; refreshable from upstream | Custom-code lexers don't translate; coverage and freshness are process-bound |
regexp2 (backtracking) over stdlib RE2 | The corpus's backrefs/lookaround are non-negotiable | Backtracking blowup risk — hence the timeout; two regex engines in the binary |
250 ms MatchTimeout + zero-width guard | Convert hangs into bounded degradation; survive grammar bugs | A slow rule silently loses its match; timeout tuning is one-size-fits-all |
| Integer range-encoded token types | Parent lookup as arithmetic; compact, allocation-free taxonomy | Fixed capacity per level (100/1000 slots); less self-describing than tuples |
| Keep upstream's recovery verbatim | Corpus behaves identically on broken input; cross-port semantics documented in a comment | Inherits the mis-scope-until-newline failure mode unchanged |
Ship Analyse API without a real corpus of scorers | API parity with upstream; leaves the door open | Detection is filename-only in practice — the honest README says so |
Sources
README.md— positioning ("based heavily on Pygments"), formatter list, style inheritance, "What's missing compared to Pygments?", adoption (Hugo,lessintegration), v3 API noteregexp.go—\Ganchoring +MatchTimeout(compile path), the tokenizer loop, zero-width guard, the attributed Pygments recovery fallback;types.go— integer token ranges +Parent();colour.go+formatters/tty_indexed.go— redmean distance downsampling_tools/pygments2chroma_xml.py— the conversion pipeline;lexers/embedded/*.xml— the ported corpus (282 files);lexers/*.go— the 19 hand-written residue lexers- Related deep-dives: Pygments (the upstream model and corpus) · syntect + Shiki (corpus-reuse via grammar-as-data instead) · bat (the pager shape Chroma's
lessintegration approximates) · the synthesis