hue DSV preview — Feature Requirements (the data browser)
Status: design · Date: 2026-08-18 · Scope: Delimiter-Separated Values (CSV, TSV, PSV, semicolon-CSV, …) as a hue content kind: the grid preview across all four sinks, the dialect sniffer, the interactive data-browser tier (multi-key sort, filtering, column hide/reorder), the selection/copy deltas, and the sparkles:dsv engine underneath.
NOTE
Phase 1 shipped: the D0 engine (95de8bb2 — libs/dsv) and the D1 basic preview (84f56c0b — the grid in every sink through the existing md table path, layout pinned by golden glyph-grid fixtures). CHK executed (2026-08-18): the table unification merged, the audit is recorded in the Milestones section, and D2–D5 are re-planned and normative. D2 shipped (a5b4bfb8): the record-number gutter, per-column caps, decimal alignment, and the pinned header band (a sparkles.ui.components.table viewport extension, af48be56, since generalized to freeze panes). D3 shipped (799266ca): the source copy format, chrome exclusion, and byte-exact whole-grid reproduction. D4's core shipped (546107c6–8fcf9290): the projection engine, the browser machine, and the TUI's filter bar, header-click sort and reset; the GUI browser wiring followed (95058b9d), then the columns palette (f8cfd284) — residue per row (header menus, fuzzy highlight positions). Everything post-CHK is forward-looking design. Status legend and ID conventions: see the overview.
Design & scope
What it is
A .csv (or .tsv, .psv, sniffed .txt/stdin) target renders as a decorated data grid by default in every sink — the MOD8 doctrine applied to a third document family: GUI, TUI, non-interactive ANSI, and HTML all paint the same grid from the same model, and --raw (CLI9) reverts to highlighted source. Interactively the grid is a data browser: a pinned header, a row-number gutter, multi-key sorting, a filter bar plus header menus, and column hide/reorder — a viewer in the VisiData/csvlens tradition, not an editor.
Almost everything visual already exists. The box-drawn grid is MDP10's table (resolveTracks, bold header + heavy ┝━┿━┥ rule, per-column alignment); the selection regime is TBL1–TBL6 unchanged; the wide-content scroll idiom is COD6's border scrollbars; the filter bar follows the picker's query-language doctrine (PKQ). What is genuinely new is the engine (dialect sniffing, an RFC 4180 parser with an identity channel, a record index, projection compute) and the browser state machine over it.
Why the engine is a library (sparkles:dsv)
The same reasoning as sparkles:diff and sparkles:fuzzy: hue's engines are things this repository owns, unit-tests, and benchmarks. sparkles:dsv (libs/dsv) is the allocation-conscious compute core — dialect detection, parsing, the record index, typed columns, and sort/filter execution — @safe pure nothrow @nogc throughout, texts borrowed as spans, storage in caller-owned SharedBuffer arenas, errors as Expected. sparkles:base is its only dependency: the fuzzy full-text remainder (DSF3) is matched by sparkles:fuzzy and combined by the host, so neither engine depends on the other. hue owns the state machine, chrome, keymap, and job scheduling.
Reference points
| Borrowed | From |
|---|---|
| the viewer-not-editor data-browser shape | VisiData, csvlens |
| dialect sniffing by field-count consistency | Python csv.Sniffer (design, not code) |
| quoting/parsing baseline | RFC 4180 (+ the real-world deviations) |
| typed columns driving alignment & comparison | xsv/qsv, VisiData |
| per-column rainbow tint (raw view, deferred) | Rainbow CSV |
| cell-level CSV diff (deferred) | daff |
Sequencing constraint: the table-rendering unification
Resolved. The table-rendering unification merged (2026-08-18: the two-view architecture — layout.d content-agnostic core / render.d string view / widgets.d widget view, with the md table case an adapter over buildTableWidgets; see docs/specs/core-cli/table.md §3.0). Phase 1 (D0/D1) shipped before the merge without touching sparkles.ui.components.table, as planned, and rebased over it cleanly — the DSV golden grids are byte-identical under the unified renderer. The CHK re-orientation executed (the audit lives in the Milestones section); the phase-1 no-table-edits invariant is retired: post-CHK work extends the unified component where a row calls for it.
Non-goals
- Editing. hue is a viewer; the bounded write surfaces belong to the diff spec (
DST5). A projection (sort/filter/hide) never writes back. - Multi-sheet workbooks (
.xlsx, ODS) — a different acquisition problem. - A query engine. The filter bar filters rows; it is not SQL, joins, or aggregation.
Content kind & dispatch (DSK)
| ID | Requirement | Status | Traces to |
|---|---|---|---|
| DSK1 | DSV must be a content kind like code/markdown/twoslash/diff (the dispatch-collapse doctrine, MOD8/MOD9): detected once by DocumentPipeline.load, dispatched through the same backend pick, rendered by every sink — no new modes. | full (84f56c0b) | document.d ContentKind.dsv; fromDsvSource |
| DSK2 | Detection: the extensions .csv, .tsv, .psv (and .ssv) must select the kind and seed the dialect (DSD1); a .txt/extensionless/stdin source must be content-sniffed (DSD5); --dsv must force the kind (a detection input, like --markdown), and --raw must force highlighted source in every sink (CLI9). | full (84f56c0b) | detect extension family; CliParams --dsv; fromSource DSD5 gate |
| DSK3 | A DSV document must render the grid preview by default in every sink — GUI, TUI, ANSI, HTML — from one shared model and one widget view (viewDsv), the exact shape of MOD8's markdown rule. | full (84f56c0b) — golden glyph-grid fixtures pin the shared layout (dsv_view.golden.*); ANSI/HTML verified by smoke | the ANSI/HTML case dsv: arms; PreviewModel.present routing; apps/hue/test/fixtures/dsv/ |
| DSK4 | --raw on a DSV file renders highlighted source like any text file; without a bundled DSV grammar it degrades to plain text per DEG2. Rainbow per-column raw styling is deferred (DSZ2). | full (84f56c0b) | the existing raw path (DEG2 plain-text degrade verified) |
| DSK5 | The status chrome must name the resolved dialect (delimiter · quote · header on/off), the row counts (visible/total when projected), the projection state (sort keys, active filter, hidden-column count), and the copy mode (SEL7 doctrine). | full (8fcf9290) — dialect · rows · projection segments in the TUI header; the GUI carries the same readout in its chrome bar's mode segment (95058b9d; its copy-mode flash covers SEL7) | dsvStatusNote + DsvBrowser.chromeNote in docNote |
Dialect detection (DSD)
Precedence: flags > sniff > extension seed. All sniffing reads a bounded sample — the first 100 records or 256 KiB, whichever ends first (provisional; one constant, shared with width measurement DSN3).
| ID | Requirement | Status | Traces to |
|---|---|---|---|
| DSD1 | The extension must seed the delimiter (.csv → , · .tsv → tab · .psv → | · .ssv → ;), and a consistency sniff over the sample may override it within the candidate set {, ; \t |}: the winning delimiter maximizes field-count consistency across sample records (quote-aware), ties broken by the seed, then by set order. This is what catches the semicolon CSVs European Excel writes into .csv. | full (95de8bb2) — hue wiring 84f56c0b | seedForExtension; sparkles.dsv.dialect.sniff; adaptDsv |
| DSD2 | The quote character must be sniffed from {" '} — " unless ' pairs consistently and " does not; RFC 4180 doubled-quote escaping ("") applies to whichever wins. --dsv-quote forces it. | full (95de8bb2) — --dsv-quote wiring 84f56c0b | sniff quote evidence; DsvFlags |
| DSD3 | Header presence must be decided by heuristic under --dsv-header=auto (default): the first record is a header when its type profile differs from the body (body columns typed numeric/date while the first row is not) or when it is all-unique, non-numeric names; yes/no force it. A headerless file gets synthetic A B C… column names (display-only, never serialized into a copy). | full (95de8bb2) — flag + synthetic names 84f56c0b | detectHeader; columnName (A B C…) |
| DSD4 | --dsv-delimiter=<char>, --dsv-quote=<char>, --dsv-header=auto|yes|no must force any sniffed decision; a forced delimiter outside the candidate set is accepted verbatim (sniffing then only decides quote/header). All three are runtime-visible in the status chrome (DSK5). | full (84f56c0b) | CliParams.dsv* → DocumentPipeline → adaptDsv overrides; re-run header heuristic |
| DSD5 | Content detection for .txt/extensionless/stdin: the source is DSV when the sniff finds ≥ 2 columns at ≥ 90% field-count consistency over ≥ 3 sample records (provisional thresholds); otherwise it stays plain text. This is a bounded first step toward DEF6 content-based detection, and the seam it should later flow through. | full (95de8bb2) — stdin/.txt wiring 84f56c0b | SniffResult.looksDsv; contentLooksDsv; the fromSource gate |
| DSD6 | Encoding: UTF-8 with an optional BOM (stripped before parsing, preserved by whole-document copy DSC4); records end at CRLF or LF (mixed accepted, each record remembers its own terminator for DSC4); malformed UTF-8 bytes display as replacement glyphs but round-trip exactly through copy (the raw span is authoritative). | partial (95de8bb2) — the parse side ships; the copy consumers are DSC4 | parseDsv (BOM, per-record Terminator, lone-CR literal) |
Model & parser — sparkles:dsv (DSM)
| ID | Requirement | Status | Traces to |
|---|---|---|---|
| DSM1 | Parsing must implement RFC 4180 quoting — quoted fields, embedded delimiters, embedded newlines, doubled-quote escapes — plus the tolerated deviations: unquoted quotes mid-field are literal, a quote after field content re-enters quoted mode (Excel behavior), a final record without a terminator is a record. | full (95de8bb2) | sparkles.dsv.parse.parseDsv |
| DSM2 | Every cell must carry an identity channel: its half-open raw byte span in the borrowed source (quotes and escapes included) alongside its decoded text — the SEL1 discipline — so sub-cell selection, char-precise crossing (DSC3), and whole-document reproduction (DSC4) stay honest. Decoding is lazy/windowed; the model never copies the file. | full (95de8bb2) | DsvCell.raw; decodeCell (simple cells borrow) |
| DSM3 | Ragged rows must degrade, never error: a short record renders padded missing cells (distinct missing styling), a long record grows the grid to the maximum field count (overflow columns named …+1), and the row is marked ragged; the status chrome counts ragged rows. An empty file or header-only file renders an empty grid plus a note (DEG doctrine). | partial (95de8bb2) — accounting, padding, …+N names and the ragged readout ship (84f56c0b); distinct missing styling and the empty-grid note pend (empty degrades to the plain view) | DsvDoc.raggedCount; buildTable padding/overflow |
| DSM4 | Each column must carry an inferred type from the sample: int · float · date (ISO 8601) · bool · string, with empty cells excluded from inference; a column types as the most specific type ≥ 95% of sampled non-empty cells satisfy (provisional). Types drive alignment (DSG3), sort comparison (DSS2), and filter operators (DSF2) — never rendering of the value itself. | full (95de8bb2) | classifyValue; inferColumnTypes (95% most-specific-first) |
| DSM5 | The library surface is @safe pure nothrow @nogc: borrowed input spans, SharedBuffer-arena storage of plain-data offsets (no string fields), Expected errors, and chunk-bounded entry points (DSN5) that read no clock and touch no I/O — the sparkles:diff/sparkles:fuzzy doctrine. Benchmarked from the first commit. | partial (95de8bb2) — attributes/borrowing/Expected/bench ship; chunked entry points await DSN5 | libs/dsv package contract; bench.d |
| DSM6 | Projection compute lives in the library: stable multi-key sort over typed comparators (DSS), typed constraint evaluation (DSF2), and the composed projection (a row-index permutation + column visibility/order) as a pure function of (model, spec) — deterministic, independent of chunking and enumeration order. | full (546107c6) | sparkles.dsv.project — applyProjection/compareTyped; determinism by index-tiebreak total order |
The grid view (DSG)
DSG1/DSG3/DSG6 are phase-1 (they render through the existing md table path); DSG2, DSG4, and DSG5 need capabilities that path does not have (pinned bands, per-column caps + document-scale horizontal scroll, a row gutter) and are post-CHK — designed here, built against the unified table after the checkpoint.
| ID | Requirement | Status | Traces to |
|---|---|---|---|
| DSG1 | The grid must render as the MDP10 box-drawn table: rounded outer corners, │ separators, bold header + heavy ┝━┿━┥ rule, light inner row rules by default, one padding cell per separator — the same widget vocabulary, so GUI/TUI/HTML parity is by construction (MDP-T2 native glyphs in cells). | full (84f56c0b) | adaptDsv → the md table path (viewMarkdown) |
| DSG2 | The header row and the column-rule must stay pinned while the document scrolls vertically (GUI/TUI: a non-scrolling band above the viewport; HTML: position: sticky), and the row-number gutter stays pinned horizontally — the COD6 pinned-gutter idiom at document scale. | full (af48be56) — the component's pinned band (TableViewportSpec.pinHeader); wired for DSV in a5b4bfb8 (interactively on by default via the auto table clamp); HTML: position: sticky ships with the semantic table (DSG6) | buildTableWidgets freeze-pane emission (pinHeader = top-rows sugar); MdTableExtras.pinHeader; golden tall.txt |
| DSG3 | Per-column alignment from type: numeric and date columns right-aligned, booleans centered, strings left — plus missing cells dimmed and ragged overflow tinted; the slots come from the theme like every widget (THM). | partial (84f56c0b) — typed alignment (+ decimal, a5b4bfb8) ships; the missing/ragged styling slots need a per-cell decoration channel the md funnel lacks — moved into D3 | buildTable aligns; the pending cell-decoration seam |
| DSG4 | A grid wider than the pane must scroll horizontally behind the frame with the border scrollbar (the COD6 bottom-border bar, ScrollView machine, wheel-sideways/Shift+wheel), the row-number gutter pinned; per-column width is capped (provisional: 64 cells) with overlong cell text ellipsized — the full value remains reachable by copy and (deferred) the cell peek DSZ3. | full (b6913019) — TBL7/TBL8 inherited through the shared table path; the 64-cell per-column cap wired in a5b4bfb8 | TableViewportSpec; MdTableExtras.columnMaxWidth → TableProps.columnMaxWidths; golden wide.txt |
| DSG5 | The row-number gutter numbers data records 1-based in source order (the header row unnumbered), and keeps showing source record numbers under any projection — a sorted/filtered view reveals provenance instead of renumbering (the VisiData reading). Gutter content is excluded from selection (SEL2). | full (a5b4bfb8) — the stub column ships; its copy exclusion is DSC (D3) | adaptDsv gutter column; TableProps.headerCols heavy stub rule; frozen against h-scroll (freezeLeftColumns, golden wide-scrolled.txt); copy-excluded + selection-inert (799266ca) |
| DSG6 | Sink parity: non-interactive ANSI emits the whole grid once — the pager case: projection/viewport controls do not apply and a --table-max-lines clamp is ignored (a pipe can never scroll; the clamp stays honored for tables embedded in markdown documents). A grid wider than the pane clips behind the frame at offset 0 by default; --table-overflow wrap wraps cells to fit instead. HTML emits the semantic <table> (DSG6's other half) — static, so browser-side sort/filter is out of scope for the v1 emit. | full (84f56c0b) — the semantic <table> (+ sticky thead over the theme bg) ships via the shared MdDoc → HTML emitter (the --html arm was rerouted off the widget-HTML interpreter, which cannot lay out the table view's positioned stack — see open-issues); scope attributes pend in the shared emitter; the pager clamp-ignore landed with the UAT fixes on the D4 branch | the ANSI sink arm; the HTML dsv arm → renderMarkdownHtml |
| DSG7 | Numeric and date columns should decimal-align (values aligned on the decimal point / a common tail), using the table core's decimal-pad solver — a capability the unification brought that a data grid should not leave unused. | full (a5b4bfb8) | MdTableExtras.columnAligns Align.decimal → decimalPadsFor |
Browser state (DSB)
The interactive tier is one projection value — sort keys + filter + column visibility/order — owned by a presentation-free state machine, applied by DSM6, painted by DSG.
| ID | Requirement | Status | Traces to |
|---|---|---|---|
| DSB1 | The projection must be a single regular value (ProjectionSpec): ordered sort keys with per-key direction, the filter AST, and the column order/visibility list. Every mutation route — header click, keymap, filter bar, header menu, columns palette — edits this one value; the machine is presentation-free and unit-tested (the TBL5 doctrine). | full (b2530077) — DsvBrowser + DsvProjection, applied by 8fcf9290 | dsv_browser.d (pure machine); workspace.applyDsvBrowser; gui.applyDsvBrowser |
| DSB2 | Pristine is a state: a one-key reset returns to source order, no filter, all columns, and the chrome must make non-pristine visually unmissable (DSK5) — because copy semantics differ there (DSC4/DSC5). | full (8fcf9290) — Shift-R; the chrome carries the projection segments | DsvBrowser.reset; Command.dsvReset |
| DSB3 | Column hide/reorder must be reachable from the keymap and from a columns palette — a picker-style list of columns with visibility toggles and move-up/down (DEF23 look, TRV list machinery); pointer drag-reorder of header cells is deferred (DSZ4). | full (f8cfd284) — Shift-C opens the palette in both hosts (the settings pane's modal pattern: shared widget tree, dsvPalette keymap scope): visibility toggles + move-up/down over colOrder, the ordered visible list flowing through rendering and copies; pointer drag-reorder stays DSZ4 | DsvBrowser.toggleColumn/moveColumn/colOrder; dsv_palette.d |
| DSB4 | Projection changes on large documents run as background jobs (DSN5) with progress in the status chrome and cancellation on any newer projection edit; the visible grid stays interactive on the previous projection until the new one lands (no half-applied views). | not started — reprojection is synchronous by design until DSN5's chunked jobs (D5); fine at sample scales | the D5 job runner |
| DSB5 | All browser keys route through hue's one binding table (KEY) so lantern lists them and CFG6 can rebind them. | partial (8fcf9290) — /, Shift-R, Shift-C and the palette's own modal scope live in the one binding table (f8cfd284; lantern lists them); sort is still pointer-only | keymap.d dsvFilter/dsvReset + CtxFlag.hasDsvGrid |
Sorting (DSS)
| ID | Requirement | Status | Traces to |
|---|---|---|---|
| DSS1 | Multi-key sort with explicit UI: header click (or the sort key on the focused column) cycles the column asc → desc → removed as the primary key, demoting existing keys; Shift+click (and a keymap chord) appends the column as the next key instead. The header cell shows each key's rank and direction (1▲, 2▼). | full (8fcf9290) — click-cycling, Shift-append, and the rank chrome ( ▲/ 1▲ 2▼ badges, chrome-clean: no source identity, never copied) ship in the TUI (2fca600b) and the GUI (95058b9d — header press cycles, Shift-press appends, badges shared) | DsvBrowser.cycleSort; tui.dsvHeaderSortAt (+mods); MdTableExtras.headerBadges |
| DSS2 | Comparison is typed per DSM4: numeric/date/bool keys compare by value, strings by Unicode-aware folded comparison; cells that fail the column's type (and missing cells) group after all typed values, comparing as strings among themselves — so a dirty column still sorts usefully. | full (546107c6) | compareTyped (typed + non-conforming grouping) |
| DSS3 | The sort must be stable with the source record index as the final tiebreak — a total, deterministic order (the sparkles:fuzzy invariant-6 doctrine), independent of chunking and identical across sinks and platforms. | full (546107c6) — stability by construction: the data-index tiebreak makes the order total | applyProjection |
Filtering (DSF)
Two surfaces, one truth: header menus compile into the same query AST the filter bar edits, so the bar always displays the whole active filter.
| ID | Requirement | Status | Traces to |
|---|---|---|---|
| DSF1 | The filter bar follows the picker's query doctrine (PKQ; the sparkles:fuzzy SPEC's lexing — whitespace tokens, quoting, \-escapes, ! negation): a query splits into column constraints plus a fuzzy remainder, parsing borrows the input and allocates nothing. | partial (8fcf9290) — the bar ships in the TUI and the GUI (95058b9d), apply-on-Enter, not incremental | the / prompt; parseFilterQuery on the PKQ lexing rules |
| DSF2 | Column constraints are name:value forms with typed operators: name:text (case-folded contains), name:=text (exact), name:>n name:>=n name:<n name:<=n (typed numeric/date comparison), name: (empty cell), each negatable (!region:EU). name resolves case-insensitively against headers (quoted when it contains spaces); #3: addresses a column by 1-based index. Constraints AND; evaluation is sparkles:dsv's (DSM6). | full (b2530077) | parseFilterQuery (quoted names, #N, all operators, negation) → sparkles.dsv.project eval |
| DSF3 | The fuzzy remainder must match rows by cell through sparkles:fuzzy (generalLanguage profile; a row admits when any visible cell admits the part), with matched positions highlighted in the grid (PKQ6 analog). The host ANDs constraint results with remainder admission — neither engine depends on the other. | partial (b2530077) — typo-tolerant per-part any-cell admission through sparkles:fuzzy ships; match-position highlighting pends on the cell-decoration seam | fuzzyRowMask; DsvProjection.rowMask |
| DSF4 | Header menus offer the per-column quick filters — contains / equals / range / empty — as a popup on the header cell (popup machinery); applying one splices the corresponding constraint into the AST (visible in the bar, removable from either surface). Value-set enumeration (a distinct-values checklist) is deferred (DSZ5). | not started | header popup → AST splice |
| DSF5 | Filtering must report: matched/total row counts in the status chrome, live-updating as background evaluation progresses (DSB4); an invalid query keeps the previous projection and surfaces the parse error inline in the bar — it never blanks the grid. | full (8fcf9290) — an invalid query keeps the previous projection and surfaces as a notice; the chrome counts visible/total rows | DsvBrowser.setFilter; dsvStatusNote rows segment |
Selection & copy (DSC)
The grid is the TBL regime; these rows bind the DSV deltas. DSC1 is phase-1 by construction (inherited through the existing table path); DSC2–DSC5 are post-CHK — the serializer and crossing seams are exactly what the table unification may reshape.
| ID | Requirement | Status | Traces to |
|---|---|---|---|
| DSC1 | Smart drag, sub-cell/rect/row/column selection, Shift/Alt snapping, the whole-table copy button, and the TUI's OSC 52 route apply unchanged (TBL1–TBL6, TSL); the serializers stay presentation-free and testable (TBL5). | full (84f56c0b) — by construction through the md table path | table_select.d reuse (tsv/markdown formats) |
| DSC2 | --table-copy grows a source mode: the selection re-emitted in the document's own dialect — the resolved delimiter/quote, minimal RFC 4180 quoting (quote only cells containing delimiter/quote/newline), records joined by the document's dominant terminator. source is the default for DSV documents (tsv stays the default for markdown tables); runtime-toggleable with the SEL7 indicator. | full (799266ca) | TableCopyFormat.source; resolveTableCopy (--table-copy=auto); the GUI 3-way toggle; the TUI TSL5 wiring |
| DSC3 | A text-regime drag that crosses the grid maps char-precise to raw source bytes via the identity channel (DSM2) — the TBL4 crossing rule with quoted cells resolving to their raw spans. | partial (799266ca) — grid-regime copies resolve raw cell bytes via the re-parsed model, and the gutter is selection-inert (SEL2); the char-precise text-regime crossing still slices the decoded buffer | DsvCopy.rawCell; unkeyed/srcStart-stripped stub cells |
| DSC4 | Whole-document reproduction (SEL8): in the pristine projection, selecting the whole document reproduces the input file byte-for-byte — BOM, quoting, mixed terminators, ragged rows and all. | full (799266ca) — through the grid regime: a whole-grid source copy IS the input file (BOM/CRLF/ragged/unterminated-quote/no-trailing-newline pinned by tests); the text-regime SEL8 reproduction rides DSC3's remaining half | serializeGridCopy whole-grid shortcut; dsv_view.copy.byteExactHardCases |
| DSC5 | Under a non-pristine projection, copying is WYSIWYG over the projection: grid-regime copies serialize the visible rows/columns in view order (all formats); DSC4's byte-reproduction guarantee is explicitly scoped to pristine, and the chrome's projection indicator (DSB2) is what tells the user which contract is live. | not started | serializers over the projection permutation |
Scale (DSN)
The normative target is ~100 MB / 1M rows interactive — a data browser that dies at 1 MB isn't one. Correctness ships first; the machinery is staged post-CHK (provisional D5). DSN1 alone is phase-1: it is a property of the D0 model, not of the renderer.
| ID | Requirement | Status | Traces to |
|---|---|---|---|
| DSN1 | The model must hold offsets, not copies: the source stays one borrowed buffer; the record index and cell spans are arena-stored integers, so resident overhead is proportional to record count, not content size. | full (95de8bb2) | DsvDoc span arenas over the borrowed source |
| DSN2 | The record index (quote-aware record boundaries — the one inherently sequential pass) must build lazily/in the background; the sample-parsed head renders immediately and the scrollbar range grows as indexing progresses (the status chrome shows progress). | not started | background index job; DSB4 runner |
| DSN3 | Column widths, dialect, types, and the header heuristic come from the bounded sample (DSD preamble) — never a whole-file scan; widths are stable thereafter (no reflow as later rows appear). | full (windowed views) — dsv_view.sampledColumnWidths measures the columns over sniffMaxRecords of the projected order and pins them through MdTableExtras.columnMinWidths → TableProps.columnMinWidths, so the geometry is a property of the view rather than of whichever rows are materialized; over-width content wraps under DSG4's cap instead of widening the column. An UNwindowed adaptation still fits its content exactly (--html, the goldens, copy) | sniff/width measurement over sample |
| DSN4 | Rendering must be viewport-culled (RND1): only visible rows are decoded and laid out per frame; scroll cost must not grow with row count. | full — adaptDsv takes a DsvWindow and materializes that slice only (the projection still runs over every record: sort and filter are properties of the view, not of the screen). The grid's viewport IS the window, so a vertical scroll re-materializes rather than re-laying-out; TableViewportSpec.virtualLines/virtualOffset let the bar, the drag mapping and the host clamps speak view coordinates. Measured on a 3012-row file: first paint 756 ms → 31 ms and flat in row count (34/28/31 ms at 100/1000/3012), a rebuild-triggering keystroke 210 ms → 3.1 ms | adaptDsv over a DsvWindow (windowed materialization) |
| DSN5 | Sort/filter/index execution must be chunk-bounded pure calls (the sparkles:fuzzy searchChunk shape): the library exposes resumable cursors, hue schedules chunks on the event-horizon pool with cancellation between chunks; results are deterministic regardless of chunking (DSS3). | not started | sparkles.dsv.project cursors; DSB4 |
| DSN6 | Provisional budgets, to be calibrated with benchmarks: first grid paint of a 100 MB file ≤ 300 ms (sample-parsed head); full index of 1M rows in the background ≤ 2 s; sort of 1M indexed rows ≤ 1 s; filter keystroke-to-first-results ≤ 100 ms. A libs/dsv/bench/ harness pins them. | partial — the harness exists (sparkles.dsv.bench, dsv.scale.* @workloads over a 73 MB / 1M-row corpus; dub test :dsv -b bench -- --bench -i dsv.scale). First grid paint of a 100 MB file ≤ 300 ms: 261 ms for 102.7 MB / 1.4M rows ✅ — and reached with the whole file parsed eagerly, which the requirement's own "(sample-parsed head)" parenthetical assumed would be necessary. It took removing waste rather than adding DSN2: a duplicate whole-file parse on the open path, readText's Phobos UTF-8 validation (131 ms → 46 ms via indexOfInvalidUtf8), and three explorer tree rebuilds. Sort of 1M rows ≤ 1 s: 172 ms numeric ✅ and 686 ms text ✅ — DSN8 took these from 2.83 s / 3.71 s to 433 ms / 1.6 s by hoisting per-cell work out of the comparator, and sorting the decorated rows rather than an index into them closed the rest. No radix pass was needed. Filter keystroke ≤ 100 ms: 5 ms ✅. Full index of 1M rows ≤ 2 s awaits DSN2, which remains the answer for the multi-GB target rather than for this budget | sparkles.dsv.bench dsv.scale.* |
| DSN7 | The document's model is retained, and a window is a query against it: the dialect, the parse, the sampled types and the header names are functions of the bytes, and the row permutation is a function of the projection — so a scroll, which changes neither, must re-derive none of them. Re-resolution happens only when the document or the flags change. | full — DsvModel (dsv_view.d) owns the resolved document and memoizes the projection and the fuzzy row mask; modelFor is the reuse test (source compared by slice identity, so a reload re-resolves); adaptDsv, DsvCopy.of and fuzzyRowMask all take it, which also removes the second and third whole-file parses a notch used to pay. Measured on a 3012-row file (dub test :hue -b bench -- --bench -i dsv.bench --perf --metrics=instr --perf-iters=8 --bench-min-time 200): a scroll notch 5.0 ms → 2.7 ms unsorted and 10 ms → 2.8 ms with a sort engaged. The two retained figures are 26.94M vs 26.98M instructions — 0.15% apart, so the sort demonstrably no longer participates in scrolling at all. Prior art and the reasoning: UI virtualization catalog | DsvModel; dsv_bench.d |
| DSN8 | Sort and filter must not repeat per-cell work per comparison: a comparison sort makes O(n log n) comparisons, so decoding, classifying or parsing a cell inside the comparator is paid ~log n times per row. Keys are extracted once per row per sort column and the comparator reads only the extracted form. | full — sparkles.dsv.project.sortByKeys is decorate-sort-undecorate over a SortEntry per row per key; text is borrowed from the source for cells needing no decode, so the arena holds only quoted cells, and keys are sized by the filtered row count rather than the document's. The single-key case — nearly every sort a reader performs — then sorts the decorated rows rather than an index over them. An index array looks cheaper, swapping four bytes instead of twenty-four, but the swaps were never the cost: it makes every comparison a random access into a 24 MB key array, and a comparison sort makes ~20M of them over a million rows. 1M rows: numeric key 2.83 s → 433 ms → 172 ms, text key 3.71 s → 1.6 s → 686 ms. A differential test pins the extracted-key order to the direct comparator's across every column type, both directions, and a mixed-direction multi-key sort; a separate 600-row stability test pins the position tiebreak (DSS3), which the differential test's eight-row fixture had too few ties to exercise | sparkles.dsv.project; dsv.scale.sort-1M |
| DSN9 | A scroll costs the steady state on every notch, including the first: no work the open already resolved may be deferred to first interaction, and nothing a rebuild owns may be re-acquired per frame. The requirement is on the worst frame rather than the median — a 240 Hz reader has 4.16 ms, and a tail figure is the only one a reader feels. | full — measured end to end over a pty (apps/hue/tools/tui-scroll-bench.d --samples, which prints every frame in send order: a summary says a spike exists, only the sequence says which keystroke pays it). Opening 76 MB / 1M rows from a 2662-entry directory, the worst of 240 keystrokes went 47–60 ms → 5.5–6.9 ms, and all of it was first-interaction waste, none of it in the DSV path: (a) modelFor compared flag spellings, and a scrolling host replays flagsOf, which cannot spell them the way the document was opened — so the first grid touch re-parsed the whole file (DSN7 now compares what the flags decide); (b) TreeData.add walked the sibling chain to find its tail, quadratic in a directory's width (45 ms at 2662 entries, now O(1) via a lastChild link); (c) a git-status snapshot landing re-walked the filesystem to repaint a letter (ExplorerTui.restyle is now separate from rebuild). Per-frame storage: the widget arena is retained across rebuilds (Builder.reset), which halved the view stage's garbage (1.85 MB → 904 KB per 48-row window) and its collection rate, taking the worst GC pause 5 ms → 3 ms and the scroll notch 2.5 → 2.2 ms | tui-scroll-bench.d; dsv.bench.stage-*, dsv.bench.scroll-notch-* |
Deferred (DSZ)
| ID | Requirement | Status | Traces to |
|---|---|---|---|
| DSZ1 | DSV diff — hue diff a.csv b.csv as a cell-level diff (daff-style): row alignment via sparkles:diff, changed cells tinted in the grid through the preview-diff decoration channel (MDP21, DVN6). | not started | sparkles:diff + grid decorations |
| DSZ2 | Rainbow raw view — per-column color in --raw (Rainbow CSV): either a bundled tree-sitter CSV grammar or a sparkles:dsv-driven highlight-event stream into the existing ANSI/HTML renderers. | not started | DSK4; highlight-event synthesis |
| DSZ3 | Cell peek — a popup showing a truncated cell's full value (popup), with copy. | not started | DSG4 ellipsis sites |
| DSZ4 | Pointer drag-reorder of header cells (the keymap/palette route DSB3 ships first). | not started | header drag machine |
| DSZ5 | Distinct-values menu — a header-menu checklist of a column's value set with counts (needs a full-column scan; rides the DSN5 job runner). | not started | DSF4 |
| DSZ6 | Frozen data columns — pinning the first N data columns against horizontal scroll (the gutter idiom generalized). | not started | DSG4 |
| DSZ7 | Number/date display formatting (thousands separators, locale decimal comma detection for semicolon dialects) — display-only, never mutating copy fidelity. | not started | DSM4 inference notes |
Milestones
Phase 1 runs in parallel with the table-rendering unification (separate worktree) and must not conflict with it: D0 is pure library work, and D1 consumes the existing md table rendering path as-is.
| Milestone | Delivers | Gate |
|---|---|---|
D0 ✅ (95de8bb2) | Basic sparkles:dsv — dialect seed + sniff (DSD1–DSD4, DSD6), RFC parser + identity channel (DSM1–DSM3), typed columns (DSM4), offsets-not-copies storage (DSN1). No hue or sparkles:ui code touched. | unit-tested pure lib; adversarial fixtures (quotes-in-quotes, ragged, CRLF/BOM, huge cells); bench baseline |
D1 ✅ (84f56c0b) | Basic hue preview — the content kind (DSK1–DSK4, DSK5's dialect readout) rendered in all four sinks through the existing md table rendering path: DsvDoc adapted onto the same table model viewMarkdown's table case consumes (DSG1/DSG3/DSG6), inheriting that path's current limits — no pinned header, no width caps / document h-scroll, no row gutter (DSG2/DSG4/DSG5 wait for CHK). The TBL selection/copy regime is inherited by construction (DSC1, tsv/markdown formats only). | golden frames GUI/TUI/ANSI/HTML; DSD5 stdin/txt sniff; zero edits under sparkles.ui.components.table (the parallel-work invariant) |
CHK — re-orientation checkpoint: ✅ executed 2026-08-18. Both entry conditions were met (D0 + D1 shipped; the unification merged) and the branch rebased over the unified pipeline with no adapter changes needed — the D1 MdDoc synthesis feeds the md table case, which is now itself an adapter over buildTableWidgets, and the DSV goldens are byte-identical. The row audit:
| Row | Verdict at CHK |
|---|---|
DSG4 | Largely satisfied by TBL7/TBL8: the framed viewport (pinned frame, border scrollbars, wheel/bar routing, --table-overflow/--table-max-lines) is inherited through the shared table path. Residual: the DSV per-column cap default and its columnMaxWidths wiring. |
DSG2 | Changed shape: the viewport pins the frame while the whole interior (header included) scrolls as one offset — the pinned header becomes a TableViewportSpec pinned-header-rows extension of the component (now permitted), not a host-painted band. |
DSG5 | Changed shape: the core models row headers (TableProps.headerCols stub columns) — the record-number gutter becomes a stub column with a copy-exclusion delta, not host chrome. |
DSG3 | Unchanged owner; easier now (TableWidgetStyle semantic slots are the styling seam for missing/ragged). |
DSG7 (new) | The core's decimal-pad solver makes numeric decimal alignment cheap — added to the D2 scope. |
DSC2–DSC5 | Unchanged shape: the srcStart/srcEnd identity channel survives wrapping in the widget view, the serializers (table_select.d) and the TBL regime are intact; the decoded-buffer vs raw-bytes reconciliation (Document.dsvText) is still the D3 substance. |
DSB/DSS/DSF/DSN | Untouched by the unification; sketches carried forward as-is. |
The milestones below are normative as re-planned at CHK.
| Milestone | Delivers (re-planned at CHK) | Gate |
|---|---|---|
D2 ✅ (a5b4bfb8) | The full grid on the unified table: the pinned-header viewport extension (DSG2 — TableViewportSpec grows pinned header rows), the headerCols record-number stub column (DSG5), the DSV per-column cap default via columnMaxWidths (DSG4 residual), decimal alignment (DSG7), and DSV fixtures for the TBL7/TBL8 viewport goldens | goldens for wide/tall DSV grids incl. the pinned header; component changes land with their own table.md §3.0 updates |
D3 ✅ (799266ca) | DSV copy deltas (DSC2–DSC5): the source dialect format (default for DSV) + stub-column exclusion, char-precise crossing via the raw-span reconciliation, pristine byte reproduction; the missing/ragged styling slots (DSG3/DSM3 completion — the per-cell decoration channel composes with the selection work) | round-trip: whole-doc copy == input bytes; gui.md TBL2/TBL6 + tui.md TSL5 gain source |
D4 ◐ core shipped (546107c6–8fcf9290) | The browser (DSB/DSS/DSF): projection machine, multi-key sort, filter bar + header menus, columns palette; carried forward from D3: the char-precise text-regime raw crossing (DSC3) and the missing/ragged styling slots (DSG3/DSM3) | machine unit tests; determinism (DSS3); keymap in lantern |
| D5 | Scale (DSN2–DSN6): background index, chunked jobs, the 100 MB / 1M-row budgets pinned by libs/dsv/bench/ | DSN6 budgets pinned; a 100 MB corpus scenario |
Cross-references (threading status)
Threaded with this spec: feature-requirements.mdCLI26 (the --dsv family), MOD10 (DSV as a content kind), DEF28 (the roadmap row), and CLI11's planned-source note; index.md's Documentation-map row, ID-scheme mnemonics, and mode-map note; the viewer.md TBL preamble pointer.
Threaded with D0 (95de8bb2): the AGENTS.md sparkles:dsv sub-package row; the module-coverage table below. Threaded with D3 (799266ca): viewer.md TBL2/TBL6 and tui.md TSL5 gained source.
Landing with later code, not before:
- The
apps/huemodule-coverage rows here and in feature-requirements.md — no hue DSV files exist yet (D1). - tui.md: a
DSV-Tparity area (by construction viaTSF3, likeMDP-T/DIF-T) once rows exist to bind.
Module coverage (sparkles:dsv)
| Source | Key symbols | Requirements |
|---|---|---|
libs/dsv/src/sparkles/dsv/model.d | Span, Dialect, DsvCell/DsvRecord/DsvDoc, decodeCell, classifyValue, inferColumnTypes | DSM2–DSM4, DSN1 |
libs/dsv/src/sparkles/dsv/parse.d | parseDsv (tolerant RFC 4180, BOM/terminators, modal/ragged accounting) | DSM1, DSM3, DSD6 |
libs/dsv/src/sparkles/dsv/dialect.d | seedForExtension, sniff, detectHeader, SniffResult.looksDsv | DSD1–DSD3, DSD5 |
libs/dsv/src/sparkles/dsv/bench.d | parse (1k/10k rows) + sniff --bench baselines | DSM5 (the benchmarked-from-D0 half) |
libs/dsv/dub.sdl | library/unittest configurations (base-only dependency) | DSM5 |
→ Feature requirements · GUI · TUI · Picker · Overview