Sparkles

Appearance

Unicode algorithms (Rust, C, and C++) ​

A source-grounded survey of SIMD UTF validation/transcoding, terminal width and segmentation, and Unicode normalization relevant to sparkles.base.text.

Last reviewed: September 30, 2026.

MetadataValue
LanguagesRust; scalar C and C++ baselines
CategoryUTF codecs, Unicode properties, segmentation, width, normalization
Primary repositoriesxutf, simdutf8, encoding_rs, simd-normalizer, unicode-rs, utf8proc, uni-algo
LicensesSee the local inventory and pinned manifests/licenses
EvidenceLocal source inspection at the listed commits; no performance measurements on this page
Related Sparkles contractsUTF validation, terminal width, grapheme iteration, bounded analysis

IMPORTANT

SIMD byte classification is not SIMD normalization, SIMD segmentation, or SIMD property lookup. The implementation boundaries below distinguish vectorized scans from scalar Unicode state machines. Upstream benchmark numbers and “fastest” statements are not local evidence and are not reproduced as results.

Overview ​

What it solves ​

The relevant questions are whether another implementation does the same operation, where it actually uses vectors, and which optimization can transfer without changing Sparkles' observable semantics. UTF well-formedness, terminal-cell width, grapheme boundaries, and normalized search units are separate contracts.

xutf describes its codec policy explicitly:

“Decoding never fails: truncated sequences, lone surrogates, garbage in → defined output out.”

That is useful prior art, but not a substitute for Sparkles' strict validator or its malformed-byte-preserving analyzer. simdutf8 states the distinction between its own interfaces:

“The basic API flavor is fastest on valid UTF-8, but only checks for errors after processing the whole byte sequence and does not provide detailed information if the data is not valid UTF-8.”

Design philosophy ​

The most transferable pattern is scan cheap cases in bulk; preserve exact state for the exceptional case. simd-normalizer is precise about the boundary:

“Non-passthrough bytes are handled with scalar decode, decompose, CCC sort, and optional recomposition.”

The scalar baselines remain useful for correctness and algorithm comparisons, not as evidence of SIMD. utf8proc positions itself as a “small, clean C library that provides Unicode normalization, case-folding, and other operations”; uni-algo emphasizes that it “handles such problems … properly and always according to The Unicode Standard.” Neither quote establishes speed or interchangeability.

Local clone inventory ​

All paths below are relative to REPOS=/home/petar/code/repos. Existing rust/xutf was inspected without updating it. All eight other paths were absent and were cloned with git clone --depth 1; no existing checkout was replaced. Dates are commit dates, not release dates. These revisions are the citation and future benchmark inputs, not assertions that their package version strings identify published releases.

ProjectLocal pathInspected commitCommit dateLicense evidenceUnicode data
xutfrust/xutf9bb347af041369a68a4effd1ca85c6d2f9b4e17bSeptember 3, 2026MITWidth/graphemes/normalization 17; category/script 16 by default, ucd-17 opt-in
simdutf8rust/simdutf8641d57f313df57354246d2b68d4778c092e076c3June 15, 2026MIT OR Apache-2.0UTF encoding validity; no versioned category database
encoding_rsrust/encoding_rsa155adc7271c9e507556d053c95b459186671d2aSeptember 24, 2026(Apache-2.0 OR MIT) AND BSD-3-ClauseWHATWG encodings; UTF transcoding does not require a width/category database
simd-normalizerrust/simd-normalizerd2352acda3e1709b07d649dcbdd542bd9c1d2540July 13, 2026Apache-2.017
unicode-segmentationrust/unicode-segmentation346f36a42592047c3f3f324141554f05ac0b5908September 17, 2026MIT OR Apache-2.018
unicode-normalizationrust/unicode-normalizationb77212b847ead642538c9c63060ac59f7a8563c5September 18, 2026(MIT OR Apache-2.0) AND Unicode-3.018
unicode-widthrust/unicode-width5180748ec0cacabfad0478996e9a89a612274e8fSeptember 16, 2026MIT OR Apache-2.018
utf8procc/utf8proc4bfe012cb879a58a70715526e9db6a49486df571September 28, 2026MIT plus Unicode data notice18
uni-algocpp/uni-algo6c091fa266ac03a852128429af3b7481cf50a0abJanuary 5, 2024Unlicense OR MIT; Unicode data notice15.1

WARNING

A fresh clone is not a Unicode-version match. All three inspected unicode-rs heads and utf8proc already use Unicode 18; Sparkles' generated tables use 17. Restrict a comparison to a verified shared repertoire/properties or select an explicitly pinned Unicode-17 revision. A Cargo version constraint alone does not establish equivalent tables.

How it works ​

SIMD coverage and dispatch ​

ImplementationActual vectorized workScalar work / limitationsExact implementation files
xutfPortable core::simd ASCII scans; mixed-width transcoding; focused AVX-512 and NEON kernels; ANSI introducer discovery; normalization ASCII quick-checkProperty lookup, cluster join state, decomposition, ordering, composition; permissive raw codecssrc/simd.rs, src/native.rs, src/x86.rs, src/neon.rs, src/strip.rs, src/normalize.rs
simdutf8Lookup/shuffle UTF-8 validity checks, ASCII fast path; x86 AVX-512/AVX2/SSE4.2, ARM NEON, Wasm SIMDShort-input scalar fallback; exact error localization in compat uses scalar validation around the bad blocksrc/implementation/algorithm.rs, src/implementation/x86/mod.rs, README technical details
encoding_rscore::simd acceleration behind simd-accel; UTF-8 validation delegates to simdutf8 on supported architectures even without that featureScalar paths remain; legacy encodings are a separate contractsrc/mem.rs, src/simd_funcs.rs, feature/dispatch documentation
simd-normalizer64-byte bound classification, masks identifying exceptional bytes; form-specific quick-check safe ranges; x86 SSE4.2/AVX2/AVX-512, ARM NEON/SVE2, Wasm SIMD128Scalar decode/decomposition/CCC buffering/composition and scalar property trie lookupssrc/simd/mod.rs, src/simd/x86_64/avx2.rs, src/normalizer.rs, src/quick_check.rs
unicode-segmentationNo explicit SIMD in inspected boundary algorithmchar iteration and grapheme category/state-machine transitionssrc/grapheme.rs
unicode-normalizationNo explicit SIMD in inspected normalization iteratorsScalar decomposition/recomposition over Iterator<Item = char>src/decompose.rs, src/recompose.rs
unicode-widthNo explicit SIMD in inspected width algorithmReverse scalar chars() fold carrying width statesrc/lib.rs, src/lookup.rs
utf8procScalar baseline, not a SIMD competitorStrict scalar decoder; staged property lookup; stateful grapheme rules; normalizationutf8proc.c
uni-algoScalar/SWAR baseline, not a SIMD competitorScalar staged lookups and Unicode state; four-byte ASCII conversion optimizationimpl_conv.h, impl_norm.h, impl_segment_grapheme.h

xutf requires nightly features including portable_simd, as visible in src/lib.rs. Some specialized codec instructions are compiled conditionally; PEXT/PDEP are scalar BMI2 instructions, not SIMD. Its specialized x86 availability check is in src/x86.rs. A benchmark must report target features and selected path rather than equating the presence of an AVX-512 source file with execution of it.

simdutf8 uses runtime x86 selection unless target features fix the implementation at compile time. Its documentation says inputs shorter than 64 bytes use core::str::from_utf8() except direct implementation APIs. encoding_rs separately distinguishes simd-accel plus std multiversioning from its UTF-8 validation dispatch; record both features when benchmarking. simd-normalizer caches the x86 backend through OnceLock; its ARM implementation instead uses a cached NEON/SVE2 decision and direct branches. These are source facts, not observed backend execution here.

Unicode property lookup and state ​

The xutf property table is not a vector gather. props(cp) does one direct indexed byte load for the 65,536 BMP entries; astral scalars use a first-stage index and a 128-codepoint leaf. The packed byte contains grapheme-break class, two width bits, Extended_Pictographic, and Indic-conjunct extension state. Out-of-range raw values receive the documented DEFAULT_PROPS (width one, break class Other).

The separate category/script word packs both properties together. src/ucd.rs documents why its default is one release behind width/grapheme data; Cargo.toml enables Unicode 17 category/script data only with ucd-17. Do not copy the README's broad “Unicode 17” description into a category-lookup claim without this qualification.

ClusterState and next_cluster combine segmentation and terminal width without materializing decoded clusters. measure_width retains the rejected boundary scalar and its property byte for the next cluster. Its simple(p) shortcut admits Other/LV/LVT only when extended pictographic and pending emoji-text promotion are excluded. This is a restricted state optimization, not “all non-ASCII codepoints are additive.”

Normalization and analysis ​

xutf offers NFC/NFD/NFKC/NFKD transformations with scalar generated decomposition/composition data and a SIMD ASCII scan. Its make_nfc/make_nfkc interfaces reuse an owned string's allocation and report required capacity when expansion does not fit. Its to_* interfaces copy the input; into_* may grow capacity. Capacity policy is part of any fair comparison.

xutf::is_nfc and is_nfkc are conservative quick-check predicates: false can mean Maybe, not that output must change. The pinned source explicitly says “false means normalization may change the text.” They must not be benchmarked as interchangeable exact normalizedness predicates.

simd-normalizer exposes nfc().normalize(&str) and corresponding forms, returning Cow<str>. Already-normalized inputs may remain borrowed; changing inputs produce output storage. Its SIMD bound masks bypass safe regions; its scalar engine still orders combining classes and composes. Its casefold uses Unicode simple C+S folding and optional Turkish overrides; ASCII classification is vectorized but ASCII lowercasing in that path is scalar. The default matching/confusable pipeline is not Sparkles' analyzer.

The scalar Rust normalization baseline describes its actual output surface as “Methods for iterating over strings while applying Unicode normalizations” in src/lib.rs. Consuming an iterator to obtain normalized values and collecting it into a UTF-8 String are different measured jobs.

Analysis ​

Contracts and error offsets ​

SurfaceSparkles contract in inspected sourcesCompetitor compatibility
Strict UTF-8 validationindexOfInvalidUtf8 returns input length or the start byte of the first invalid sequence; overlongs, surrogates, truncation and out-of-range encodings failsimdutf8::compat::from_utf8(...).err().valid_up_to() supplies the compatible first-error surface. basic is only a boolean/full-scan comparison. xutf raw codecs do not validate.
Malformed analysisEach malformed byte becomes a distinct opaque unit above the scalar range, retaining byte provenanceValid-str normalization interfaces cannot accept the same input. utf8proc returns a UTF-8 error; uni-algo lenient operations replace malformed sequences. None is a full analyzer substitute.
Scalar widthControls, noncharacters, separators, all Mn/Mc/Me/Cf, and conjoining ranges zero; regional indicators two; ambiguous narrowunicode-width::UnicodeWidthChar::width returns None for controls, requiring an explicit zero policy. xutf follows a different zero-width set, not “all Mc zero.” utf8proc_charwidth is scalar width, not cluster width.
Cluster widthLeading-codepoint width adjusted by Sparkles' VS15/VS16 policy; cluster members do not add widthxutf has join-state/promotion rules and documents differences from unicode-width, including VS15 and bare keycaps. unicode-width includes script-specific ligatures and U+17D8 width three. Output agreement must gate each fixture.
ANSI widthvisibleWidth recognizes ESC-prefixed sequences using escapeLength; unterminated sequences consume remaining inputxutf::width does not remove ANSI; width_ansi does. The latter also accepts C1 introducers, unlike Sparkles' ESC-only discovery. ESC-based SGR/OSC fixtures are candidates, not proof of parser identity.
GraphemesPhobos graphemeStride; decoded window capped at 32 codepoints; malformed decoding uses replacement; escapes are separate zero-width itemsxutf supplies Unicode-17 extended graphemes without that cap; unicode-segmentation/utf8proc heads supply Unicode 18. Exclude malformed text, escapes and over-cap clusters for a common boundary-only comparison.
Analysis outputBounded TextUnit values plus byte spans/flags; NFC or NFKC, sensitive/simple/full fold, optional mark stripping/word flags/stopwords; explicit capacity failureA normalizer-only row lacks provenance, flags, folding and capacity semantics. Label it a component comparison, never end-to-end analyzeText speed.

Sources: Sparkles validation, width, clusters, ANSI grammar, analysis; xutf width policy, property generator, ANSI parser; unicode-width policy; utf8proc API/errors; uni-algo error policy.

Sparkles' generated width/normalization tables declare Unicode 17, but the inspected width and grapheme modules also use Phobos categories/segmentation. The generated version alone therefore does not establish that every property in the entire width/segmentation pipeline is Unicode 17. Record the D runtime version or compare outputs directly.

Bounds, tails and dispatch ​

The dispatch inventory establishes available implementations, not the backend a local binary executed. xutf::plain_prefix uses full in-bounds SIMD windows plus a scalar tail. Its other ASCII kernels can overlap a final window; that does not grant a caller permission to read beyond a slice. The normalizer's AVX2 scanner requires 64 readable bytes, with shorter input handled elsewhere. For any D adaptation, retain a bounded scalar tail, explicit target-feature selection, and the existing fallback; report actual build features with results.

Sparkles' 32-codepoint grapheme cap and analyzer segment/output capacities are independent bounds. An unlimited competitor cannot establish equivalence at those failure or split boundaries. Width-only optimizations must also preserve pending presentation context at the ASCII/non-ASCII transition.

Benchmark fit and evidence ​

These are proposed rows for sparkles:test-runner integration, not executed results. Verify exact outputs before entering timings into a shared chart.

Measured operationShortlisted APICandidate shared fixture / excluded cases
Validity plus first invalid bytesimdutf8::compat::from_utf8; Sparkles indexOfInvalidUtf8Valid ASCII/mixed UTF-8 plus malformed leads/second bytes and block-crossing truncation. Normalize success to input length and error to valid_up_to. Include early/late errors separately.
Boolean full-input validitysimdutf8::basic::from_utf8Separate chart from first-error semantics; never supply fabricated offsets.
UTF-8 → UTF-16 into preallocated outputencoding_rs::mem::convert_utf8_to_utf16_without_replacement; xutf::transcode_into::<Utf8, Utf16>(src, dst, AsciiCase::Preserve)Well-formed inputs only; destination sized for the largest documented requirement; compare produced code units, not lengths alone. Replacement/first-error behavior is not common.
UTF-16 → UTF-8 into preallocated outputencoding_rs::mem::convert_utf16_to_utf8; xutf::transcode_into::<Utf16, Utf8>(src, dst, AsciiCase::Preserve)Valid paired UTF-16 only. encoding_rs replaces unpaired surrogates; xutf raw codecs preserve permissive values.
Plain terminal-cell widthxutf::width_str; unicode_width::UnicodeWidthStr::width; Sparkles visibleWidthPrintable ASCII, CJK prose, precomposed Latin, ordinary combining accents and common emoji/RI, after output agreement. No ANSI, control/ligature/presentation edge cases or pathological clusters in a supposedly common row.
ESC-styled widthxutf::width_ansi_str; Sparkles visibleWidthValid ESC-based SGR and OSC 8 around compatible text; classify C1 and unterminated/malformed grammar cases separately. No allocating strip-then-width adapter silently treated as a direct API.
Extended-grapheme boundariesxutf::grapheme_indices::<Utf8>; UnicodeSegmentation::grapheme_indices(text, true); utf8proc_grapheme_break_stateful with strict decodeValid escape-free text, shared Unicode repertoire, clusters within Sparkles' 32-codepoint window. Hash/compare boundary offsets, not just cluster count. xutf also computes width; disclose that extra work.
NFC/NFKC transformed scalar valuessimd_normalizer::nfc()/nfkc().normalize; xutf normalization traits; unicode_normalization::UnicodeNormalization::nfc()/nfkc(); utf8proc_map; una::norm::to_nfc_utf8/to_nfkc_utf8Valid common-repertoire inputs and forms, case-sensitive/no mark stripping. Separate iterator consumption, borrowed-result, in-place and allocated UTF-8-output rows. Sparkles comparison is the normalization component, not complete provenance analysis.

Upstream benchmark entry points useful for adapter work are xutf/benches/width.rs, benches/width_ansi.rs, benches/graphemes.rs, benches/normalize.rs, and benches/throughput.rs; their published numbers are not substituted for a same-host runner.

D/CTFE and license fit ​

The Rust vector intrinsics are not directly executable D CTFE code. A port should keep the established scalar compile-time path and specialize runtime byte scanning only. Preserve the existing Unicode tables, caps, ANSI grammar, and leading-codepoint width policy: the proposed ASCII-prefix optimization leaves the last ASCII starter before non-ASCII to the scalar cluster path rather than importing xutf's width semantics.

The inventory records software and derived-data licenses separately where manifests/notices require it. MIT/Apache code is not unlicensed pseudocode; keep required notices if code is copied. Likewise, generating new tables from upstream Unicode data requires preserving its applicable data notice. Algorithm inspiration does not justify copying a competitor's Unicode generation or width overrides silently.

Implementation ideas for Sparkles ​

Each item is a design inference from cited source, not a measured speedup or an instruction to change Unicode policy.

  1. Printable ASCII width in bulk. Follow xutf::plain_prefix: reject a non-plain head cheaply, vector-scan 0x20..=0x7E, and return to the established scalar/cluster path at an exceptional byte. Preserve ANSI boundaries and the last possible presentation-sensitive base. xutf holds #, *, and digits before VS16/keycap continuation; Sparkles has its own graphemeClusterWidth policy, so derive its boundary condition from that policy instead of copying the whitelist unexamined.
  2. Avoid repeated decoding/property work. measure_width carries a lookahead scalar/property into the next cluster and admits provably simple runs. This suggests reducing scanCluster's repeated prefix graphemeStride and width rescans, while retaining its cap, output fields and malformed behavior. Use a state-machine-compatible shortcut; do not treat all CJK, all marks or all non-ASCII as interchangeable.
  3. Fused generated scalar properties. The xutf packed table can replace several separate tests with one scalar lookup. A direct BMP array trades about 64 KiB for one-load indexing; an all-plane deduplicated trie trades additional dependent loads for smaller tables. Generate from Sparkles' own source/version/policy and benchmark short terminal strings, mixed text, and cache pressure before selecting a layout. This is a property-layout idea, not evidence that gathers would help.
  4. Analysis ASCII specialization without losing materialization. analyzeText still must emit every unit, source span and uppercase flag even when normalization is a no-op. A vector ASCII/uppercase probe can bypass Unicode lookup/decomposition for known-safe runs, but must flush any pending combining segment and maintain word/fold/stopword policy and exact capacity errors. A borrowed Cow return is not a valid fast-path replacement for this API.
  5. Safe-region normalization checks. simd-normalizer quick-check classifies byte ranges and known-safe scripts before scalar trie lookup. These bounds depend on form and valid UTF-8: NFC-safe is not NFKC-safe, and Sparkles accepts opaque malformed bytes. Reuse only an explicitly proven byte/scalar subset; never skip provenance, folding, or pending composition across its boundary.
  6. Do not pay normalization allocation to benchmark analysis. xutf in-place capacity handling and simd-normalizer borrowed/owned results illustrate separate storage models. Make storage policy and produced output part of the row name rather than crediting different work as algorithm speed.

Strengths ​

  • Real SIMD implementations are inspectable in local, pinned source trees; no reliance on crate descriptions alone.
  • xutf combines width and segmentation with compact scalar properties and allocation-free borrowed outputs, closely matching terminal workloads.
  • simdutf8::compat offers a strict first-error surface; encoding_rs offers slice-output transcoding APIs.
  • simd-normalizer is an actual SIMD-guided normalization competitor with explicit scalar boundaries and dispatch.
  • Scalar baselines expose independent property layouts and Unicode algorithms, making them useful correctness/component references.

Weaknesses ​

  • There is no drop-in full competitor for Sparkles' bounded, malformed-byte-preserving, provenance-bearing analyzer in this shortlist.
  • Unicode versions and width policies differ substantially; matching an operation name is insufficient.
  • Nightly features, backend dispatch, allocation, cluster caps and ANSI grammar can dominate or invalidate an ostensibly common comparison.
  • No local runtime result is established here; source structure alone cannot rank implementations on a host.

Key design decisions and trade-offs ​

DecisionRationaleTrade-off
Separate SIMD scans from scalar Unicode algorithmsVector intrinsics prove actual acceleration coverage, not acceleration of every stageMore precise reporting; fewer broad “SIMD Unicode” claims
Keep first-error and boolean validation distinctExact failure location/early exit is consumer-visible workTwo benchmark contracts instead of a misleading single ranking
Retain scalar competitorsIndependent algorithms and data layouts inform correctness and implementationThey cannot be presented as SIMD implementations
Gate fixtures on output agreementUnicode versions, terminal policies and grapheme caps differA compatible subset supports a scoped result, not universal equivalence
Compare normalization as a componentSearch analysis additionally emits provenance, flags and bounded errorsNo claim that standalone normalization replaces or outruns complete analysis
Pin local heads without modifying existing clonesReproducible source and respect for user checkoutsUnicode-18 heads need deliberate matching inputs or a separately selected revision

Sources ​