Sparkles

Appearance

sparkles:event-horizon — Delivery plan ​

Companion to SPEC.md: the milestones that build the library to the specification. Each milestone is independently green (builds + tests + lints). Benchmarks start as a baseline in M3 and grow per milestone; M14 is where the cross-runtime matrix and the results doc land.

M1 — Specification ​

SPEC.md — the normative surface: the backend seam (§3), the op and completion vocabulary (§4), the callback tier (§5), buffers (§6), the fiber tier (§7), structured concurrency (§8), errors (§9), capabilities and effect rows (§10), scheduler topologies (§11), the Effect!T veneer sketch (§12), and the public API (§16). Spec examples that need not-yet-implemented modules are parked with <!-- md-example-skip --> directives; the implementation milestones unpark them and make them pass under nix run .#ci -- --verify.

Gate: nix run .#ci -- --verify --files docs/specs/event-horizon/SPEC.md and yarn docs:build are green.

M2 — Scaffold + ring substrate ​

Preparation commits first: add during ~>0.5.0 to the relevant dub.sdl / dub.selections.json / nix/dub-lock.json; then the libs/event-horizon package (dub template from libs/wired/dub.sdl) with sparkles.event_horizon.backend.uring wrapping during's ring setup, and the three-tier probe (backend selection → opcode probe → feature flags) surfaced as a BackendCaps value (SPEC §3). On Linux without a working io_uring, setup returns the hard probe error of SPEC §3 — there is no epoll path.

Gate: dub test :event-horizon green on Linux (probe + ring round-trip unittests, SKIP-degrading on unsupported kernels).

M3 — Callback completion core (tier A) ​

loop.d + op.d + buffer.d: the single-threaded EventLoop with the public callback API — submit op + completion callback/context, in-ring timers (TIMEOUT), per-op deadlines (LINK_TIMEOUT), op-level cancellation handles, run()/runOnce()/stop(); accept/connect/read/write/recv/send plus the datagram pair sendTo/recvFrom in callback style with owned-buffer transfer (SPEC §5–§6). Callback-style echo-server example.

Gate: unittests + the callback echo example verified by nix run .#ci -- --verify; first benchmark baseline recorded under libs/event-horizon/bench/.

M4 — Fiber scheduler + direct-style I/O (tier B) ​

sched.d + io.d: the fiber run queue over the callback core — suspend/resume keyed by op slots, spawn, fiber pooling (Fiber.reset recycling) — and the same I/O surface in direct style (SPEC §7). Direct-style echo-server example (README-able). Base prerequisite: a length setter (grow-with-default) on Buffer in sparkles:base, used by the spec overview example. (handle.d — the erased LoopHandle — is deferred to its first consumer, the M12 veneer; tiers B/C hold the concrete loop.)

Gate: both echo styles green; fiber-vs-callback overhead measured in the bench.

M5 — Structured concurrency + cancellation + errors ​

scope_.d + cause.d (no ring/loop imports): Scope spawn/join/fail semantics, the cancellation tree + protect, deadlines as cancel scopes, the ASYNC_CANCEL slot discipline (slot + buffer stay alive until the terminal CQE), graceful shutdown (cancel accept → drain → unregister), and the Cause/IoResult error model finalized (SPEC §8–§9). The callback tier keeps its op-level cancellation from M3.

Gate: the cancellation/timeout/shutdown test matrix is green — axes: park type (I/O / join / timer) × cancel source (explicit / deadline / sibling failure / shutdown) × race outcome (cancel wins / completion wins, SPEC §8.5 provenance) × protect (inside / outside).

M6 — Effect capabilities + DI ​

capability.d + clock.d + net.d + schedule.d + testing.d: capability structs and the effect-row convention (template introspection helpers, subset projection), handler swapping, TestClock, isNet/isByteStream + SimNet, the deterministic TestSched executor with advanceAndSettle, Schedule (retry/backoff) values, and retry/timeout/race as ordinary functions over fibers (SPEC §10).

Gate: a unit test exercising a nontrivial workflow entirely under TestClock/SimNet — no real I/O.

M7 — Files, subprocess, signals, watch ​

fs.d + proc.d + signals.d + watch.d: OPENAT/READ/WRITE/STATX/ FSYNC file capability; Proc (spawn, piped stdio, WAITID reap); signals (signalfd driven through the ring); Watch (inotify fd driven through the ring). Exposed at both tiers (callback + capability).

Gate: the agent-tooling demo (run a subprocess, stream its output, watch a directory) as a verified example.

M8 — Tier-3 single-ring throughput ​

Multishot accept/recv, provided buffer rings, registered files/buffers, and the DEFER_TASKRUN + SINGLE_ISSUER operating mode (SPEC §3, §6). All of these are guaranteed by the kernel ≥ 6.1 floor (SPEC §3.3) — there are no per-op runtime fallbacks on Linux; instead, fault-injection tests fake probe/caps results to exercise the gating plumbing itself (open-issues O5).

Gate: the benchmark suite shows the expected step-change; the probe/caps fault-injection tests are green.

M9 — Multi-thread topologies ​

group.d + live.d: (a) thread-per-core sharding (one ring per core, MSG_RING handoff); (b) the work-stealing scheduler — per-worker rings, stealing of never-started tasks only, idle workers parked via in-ring FUTEX_WAIT so one wait observes CQEs and steal wakes, FUTEX_WAKE signalling (SPEC §11); (c) the live capability row — RingClock/RingNet and the Env alias handed out by LoopGroup.run (the milestone that unparks the SPEC §1 overview example). Benchmark futex-wake vs MSG_RING vs eventfd; measure thread-per-core vs work-stealing tail latency/throughput on the echo bench.

Gate: all three topologies selectable via loop-group config (done — single + thread-per-core via LoopGroup.runEach + work-stealing via WorkStealingPool); the SPEC §1 example unparked and green under --verify; benchmarks committed.

E0 — Persistent raw CPU jobs ​

pool.d gains the SPEC §11.1 RawCpuPool: fixed-capacity MPMC submission and completion rings, attributed function-pointer jobs over caller-owned contexts, persistent DRuntime workers, explicit saturation/start/shutdown outcomes, and draining or cancelling shutdown. The existing batch/delegate WorkStealingPool remains source-compatible and keeps its benchmark history.

Gate: start/stop cycles, queue and completion saturation, exactly-once execution/completion, context lifetime, draining/cancelling shutdown, and ThreadSanitizer stress tests pass. The picker exercises the documented synchronous fallback when start or submit cannot accept work.

M10 — kqueue backend (macOS) ​

backend/kqueue.d: the completion-synthesizing proactor over kqueue readiness; files via a small worker pool (regular files have no readiness); scheduler parking via os_sync_wait_on_address; the same public API at both tiers. Verified on mac-bsn (built via ldc2 directly).

Gate met: the FULL EventLoop!KqueueBackend integration (tier-A loop + tier-B fibers + accept/connect/recv/send verbs) verified green two ways — the data path on real macOS (Darwin 25.3.0, scripts/verify-kqueue-macos.sh), and a complete fiber TCP echo on Linux over mheily/libkqueue (dub run --single libs/event-horizon/examples/fiber-echo.d -b checked -c libkqueue). The loop-portability refactor (loop/sched/io/scope/live/group/pool → version(Posix), DefaultBackend selection in backend/select.d) landed to enable it. Regular-file worker pool + native EV_DELETE cancel remain as refinements.

M11 — IOCP backend (Windows, Wine-tested) ​

backend/iocp.d: native completion mapping (OVERLAPPED ⇄ op slots); scheduler parking via WaitOnAddress; the Win32 cross shell ported from the window-system research branch (win32-ldc2 + headless wine64), findings labelled A[wine]; optional windows-latest CI job.

Gate: the echo example green under Wine.

This milestone shipped socket/timer/waker parity only. It did not ship process spawn, process-handle waits, overlapped process pipes, or Job Objects; those are M19, and the distinction is normative in SPEC §13.

M12 — Monadic veneer (tier C) ​

effect.d: the Effect!T pipeline layer monomorphized onto the direct core (SPEC §12), with parity tests (same behavior direct vs veneer) and a microbenchmark bounding the veneer's overhead.

Gate: the parity suite is green; the overhead bound is documented.

M13 — Showcases ​

  1. HTTP/1.1 building blocks (parser + minimal handler API) as the new sparkles:http sub-package at libs/http.
  2. apps/terminal ported onto the loop: PTY read, child reap, timers replace the per-frame non-blocking drain.

Gate: all examples verified.

M14 — Cross-runtime benchmark suite ​

A libs/event-horizon/bench/ harness (model: libs/wired/bench/; competitor orchestration + RSS/CPU collection via sparkles.core_cli.process_utils) comparing event-horizon — all three tiers, both topologies — against pinned competitor implementations: D vibe.d, Rust Tokio, Rust Glommio, C++ Boost.Asio, libuv (raw C), Node.js and Bun, and OCaml Eio, each via a reproducible Nix devshell pin. Workloads: TCP echo (throughput + p50/p99/p999 tail latency, few-large and many-small connections), HTTP/1.1 plaintext via sparkles:http where the competitor has an equivalent, a file-I/O scan, and the internal microbenchmarks (effect-op dispatch, fiber switch, callback vs fiber vs veneer overhead).

The file-I/O phase additionally plugs into the local polyglot-walks clone: an event-horizon implementation of the recursive count-files/dirs walker (work-stealing topology) registered as a walker via a flake-module.nix, so nix flake check validates correctness against the fixture and nix run .#benchmark races it under hyperfine against the rayon/go/tokio/node implementations. Caveat recorded in the results doc: getdents has no io_uring opcode, so the walk stresses the scheduler and OPENAT/STATX more than pure ring I/O.

Gate: one command runs the matrix locally; the results + methodology doc (research-docs style) is committed; event-horizon lands within its target envelopes (≥ vibe.d throughput at lower tail latency; competitive with Tokio/Glommio on echo).

M15 — Async process management v2 ​

The proc rebuild to SPEC §13, replacing the M7 sketch: the effects-side vocabulary (StdioMode/StdioSpec/ProcessConfig/ExitStatus), full per-stream stdio control with heap-staged argv/env (spawnProcess v2), kill/killGroup, signaled-vs-exited status decoding, PTY spawning (spawnPty via POSIX_SPAWN_SETSID + in-child slave open, resizePty), and the capability layer — isProc (associated-Child shape), RingProc joining Env, SimProc as the deterministic double. Portability refinements ride the backend milestones: kqueue EVFILT_PROC/NOTE_EXIT lowering for OpWaitid, plus a planned IOCP reap. M19 supersedes the latter with the documented wait-registration contract in SPEC §13.9.

Gate: the agent-tooling example upgraded to v2 (stdin round-trip, stderr capture, kill, signaled exit); pty-drain reaps in-ring via spawnPty; a SimProc unit test drives an app-shaped "run, parse, decide" path with no process.

Status clarification: the low-level implementation shipped on POSIX and is exported/tested on Linux. kqueue child reap has since shipped, but macOS public export and process tests remain M19. IOCP process support did not ship here.

M16 — UI-loop substrate ​

The SPEC §5.6/§14/§15 primitives, in dependency order: the external Waker (eventfd / EVFILT_USER / PostQueuedCompletionStatus; drain accounting excludes the armed wake op; destroy disarms), OpPollAdd (single + multishot; uring POLL_ADD, kqueue filter-only) with the waitReadable/waitWritable verbs, sleepUntil + Ticker (absolute-deadline frame clock, missed ticks skipped), Channel!T (bounded intra-worker; park/wake through the executor seam; close-then-drain semantics), and Sched.tick (the O7 embedding hatch).

Gate: unit tests for each primitive (cross-thread wake; poll on a pipe; tick cadence + missed-tick skip under a deliberately slow consumer; channel producer/consumer/close/cancel; tick-driven echo identical to run-driven); the io.steadyState.zeroAllocations gate still green with a waker armed.

M17 — Owning the application loops ​

The SPEC §15.3 shapes delivered through the UI adapters, then the applications:

  1. sparkles:ui-tui gains the ring-driven input source (chunked stdin reads through a resumable escape assembler over sparkles:tui's pure decoders; SignalFd([SIGWINCH]) resize; events into a Channel).
  2. sparkles:ui-raylib gains the Ticker-paced frame driver (SetTargetFPS(0); sampled input per tick; the plain-loop fallback arm for loop-creation failure — Android seccomp).
  3. apps/hue: the workspace TUI loop moves onto the input/UI-fiber shape; GitStatusCache re-shaped from a std.parallelism thread to a spawned git status --porcelain child (deleting the nested 150 ms wait); the resident oracles' non-blocking tryReadLine polling becomes parked pipe reads. Then the --gui frame loop onto the frame driver. (Done: the loops landed with M17; the git child rides capture behind a GitStatusCache.asyncSpawn driver seam gated on OpWaitid, and the oracles keep their ResidentProcess wire while daemon watcher fibers park waitReadable on their stdout and wake the loop through the event channel — a ~10 s tree-open oracle-active session measures 8 ring waits, no 33/150 ms wakeups.)
  4. apps/terminal follows (the M13 port, now with its substrate in place): PTY master reads and the child reap park instead of the per-frame EAGAIN drain; the idle no-swap behavior is preserved. (Done. The loop landed via the terminal-view extraction — the idle gate's 99.7→0.3% is ring parking replacing busy-wait pacing — and the pty/reap half shipped as TVW8 on the HST15 host errands plus this library's waitPid raw-pid reap: parked master reads with a sync burst drain, in-ring reap, idle at zero pty syscalls, churn +1.4 pp recorded as the saturation cost.)

Coordinate with the sparkles:ui-app plan: the host package's tui_loop.d/gui_loop.d arms are the natural final home of these loops (HST9); the adapters land the machinery so either migration order works.

Native desktop windowing follows the separate sparkles:wsi delivery plan. Its M2 Wayland/X11/Win32/AppKit spikes are the acceptance tests for this section's foreign-fd and OS-owned-loop seams. If a spike needs a new primitive, it lands in Event Horizon first as the smallest backend-neutral host hook; WSI never carries a parallel wait or scheduler.

The two OS-owned-loop acceptance slices are delivered. On Win32, IOCP's completion-mirror event plus runHostedOnce/tickHosted shares one blocking wait with User32; the bounded Wine smoke runs an HWND lifecycle beside an IOCP timer and foreign-thread waker. On AppKit, the same host seam wraps Event Horizon's native kqueue descriptor in a CFFileDescriptor source; the arm64 macOS smoke runs an NSWindow lifecycle, kqueue timer, and foreign-thread waker through one CFRunLoop wait. The direct X11/XCB consumer is now delivered: it drains XCB's buffered queue, waits on the connection fd through single-shot OpPollAdd, and uses cancelAndWait as its borrowed-callback teardown barrier. The Wayland consumer is also delivered without a bootstrap round-trip: registry sync, prepare-read pairing, flush backpressure, configure/ack and cancellation all run through the same single-shot poll. XIM remains the unresolved part of the X11 input path.

Gate: hue TUI runs with zero polling timeouts on an idle document (strace-verifiable: the loop makes one blocking ring wait, no 33/150 ms tick wakeups); the apps/terminal render-CPU benchmark's idle scenario stays at its near-zero baseline after the port.

M18 — Fork + shared-memory server (SPEC §17) ​

The zygote/grandchild ForkServer (forkserver.d): single-threaded-start gate (refusal, not assert), SOCK_SEQPACKET frames + anonymous shared-arena slots, serial serve loop, crash/overflow/throw status contract, blocking and non-blocking takes, the completion fd for pollAdd. Tested single-threaded (dub test :event-horizon -- -t 1 -i forkserver; the suite's default multi-threaded run skips them): echo round-trip, crash → -SIGSEGV with continued service, slot backpressure (EAGAIN) and input size guard (EMSGSIZE), non-blocking take. First consumer: hue's format preview (FPR10 in docs/specs/hue/format-preview.md) — the in-process dmd-fmt provider's execution backend, with initDMD as childInit so every format inherits initialized frontend globals by CoW. Follow-up (not built): parallel grandchild fan-out; dmd-lsp fork-per-analysis (PRJ13 lifted); the live-types oracle.

M19 — Cross-platform supervised processes for apps/ci ​

Deliver SPEC §§13.5–13.9 as one parity milestone above the shipped M15 low-level POSIX API:

  1. Export proc/live and the supported process surface under version (Posix), not Linux-only gates; run spawn, separate/merged pipes, environment, cwd, exit, kill, concurrent drain, and exactly-once reap tests against io_uring and kqueue (libkqueue on Linux plus native macOS).
  2. Add inherited-environment overlays and the supervised run/event API: independent stdout/stderr line framing over arbitrary bytes, raw capture, cumulative resource-sample events, and a final exit event.
  3. Add timeout and cancellation teardown: tree-scoped TERM, monotonic grace, KILL, concurrent EOF drain, and one reap on every race/error path.
  4. Add the bounded scheduler-integrated BlockingPool; completions wake the owner loop, cancellation retains job storage through terminal completion, and no blocking call or sleep runs on a scheduler thread.
  5. Add native Windows spawning with CreateProcessW, UTF-16 argv/environment/ cwd staging, restricted handle inheritance, IOCP-associated overlapped named pipes, process-handle wait registration posting to IOCP, and Job Object tree ownership/termination/resource accounting.
  6. Port the process execution layer used by apps/ci from sparkles.core_cli.process_utils to the supervised surface without changing its user-visible line streaming, timeout status, separate capture, or resource-reporting behavior.

Gate: one behavioral suite runs the same supervision cases on Linux, native macOS, and Windows (Wine for bounded smoke plus windows-latest for Job Object, console-control, and real IOCP semantics). Cases cover environment set/replace/ unset and PATH lookup; LF/CRLF/split/final-partial/binary lines; stdout/stderr ordering and merge; chatty dual streams; natural exit, spawn failure, timeout, external cancellation, TERM-exit, grace-expiry KILL, descendants, and every exit-vs-EOF race; exactly one exit event/drain/reap; cumulative samples; blocking pool saturation/cancel/shutdown. apps/ci's affected commands pass on all three hosts. Linux traces contain io_uring_enter and no epoll_*; Windows traces contain IOCP waits and no pipe/process polling loop.

Status (September 5, 2026): items 2–4 shipped on Linux as SPEC §§13.5–13.8 — the environment overlay and spawn-driven PATH search (live), the bounded LineFramer (proc), the BlockingPool with its termination-critical lane (blocking_pool), the cgroup v2 containment tiers (cgroup), the bounded transactional tree sampler (sampling), and the supervised run itself (supervise: shielded workers, one-shot ring-free clocks, the sampler handshake before the reap, the tree-state machine with ResidualPolicy, the protected terminal sequence, and the pre-join emergency phase). The substrate they stand on landed in the same series: guaranteed in-flight cancellation with a reserved internal slot and a no-return fatal hook (loop, SPEC §5.2), the scheduler-owned blocking inbox and wake-only Alarm (sched), and package-private shielded scope children (scope_, SPEC §8.2). The "no polling loop" gate holds: an idle three-second supervised sleep makes nine io_uring_enter calls, no epoll_*, and no timer loop (strace -f -c). Remaining: item 1's macOS export and process tests (the Darwin sampler is a sampled == false stub), item 5 (Windows, §13.9), item 6 (the apps/ci port), and a clone3(CLONE_INTO_CGROUP) spawn path so that a child forked before the post-spawn migration lands is contained by construction rather than by the process group alone (SPEC §13.7).