Sparkles

Appearance

Process Supervision in Event Loops ​

How completion-first and readiness-first runtimes spawn a child, own its output, its tree and its exit, and how a supervisor stays honest when the OS refuses to cooperate — the survey behind sparkles:event-horizon's supervised runs (SPEC §13.5–§13.8).

FieldValue
Scopechild creation, exit observation, output draining, tree containment, termination policy, resource accounting
Systemslibuv, Go, .NET, Java, Tokio, systemd, Erlang/OTP ports, Bazel, kqueue/EVFILT_PROC, Linux pidfd/waitid/cgroup v2
CategoryConcept / technique (cross-cutting over the master catalog)
FeedsSPEC §13; PLAN M19

Last reviewed: September 5, 2026

NOTE

This is a technique survey, not a per-system deep-dive: each section names the systems that made a given choice and cites the mechanism, then states what event-horizon took from it. The deep-dives for the runtimes themselves (libuv, Tokio, Go, .NET, Java) remain the source of truth for those systems.

Overview ​

What it solves ​

A supervised run is not "spawn and wait". Every runtime that grew a process API discovered the same set of problems in the same order:

  1. A child that writes faster than you read deadlocks the wait. A pipe buffer is finite; a child blocked in write never exits; a parent blocked in waitpid never reads. Every mature API drains both output pipes concurrently with the wait.
  2. The exit is an event, the reap is a right. A child's termination can be observed without consuming it (waitid(WNOWAIT), EVFILT_PROC, pidfd readability) and must be consumed exactly once; a second consumer (SIGCHLD=SIG_IGN, a library that calls wait(-1)) steals it and turns every later pid-based operation into a hazard.
  3. A child is a tree. Shells fork; build tools fork; daemons double-fork. Killing the root does not kill the tree, and a descendant that inherited the pipe holds EOF hostage.
  4. Blocking calls do not belong on the loop thread. waitpid, cgroupfs waits, a /proc walk — anything the completion backend cannot express must go to a pool, and the pool must not be able to refuse the one job that ends a run.
  5. Accounting must say what it is the truth about. Sampled peaks are lower bounds; kernel cumulative counters are exact only over an enforced boundary; a replacement process with a reused pid must never be folded as the original.

Design philosophy — what the field agrees on ​

Across the systems below, four positions hold with remarkable consistency:

  • Concurrent drains, single reap. Go's os/exec copies each pipe in its own goroutine, and Cmd.Wait is documented to wait for the exit and for the stdin/stdout/ stderr copying to complete before releasing the command's resources; libuv's uv_process_t delivers the exit through the loop while the pipes are ordinary streams; Tokio's tokio::process documents that dropping a Child does not by default kill the process (kill_on_drop opts in) and that its stdio handles are independent AsyncRead/AsyncWrite objects.
  • Tree ownership is an OS capability, not a pid list. .NET's Process.Kill(entireProcessTree: true) walks descendants on Unix but uses a Job Object on Windows; Java's ProcessHandle.descendants() is explicitly a snapshot; systemd puts every service in its own cgroup because a pid list cannot track a forking tree, and its KillMode=control-group default signals the whole cgroup (cgroup.kill); Bazel's sandboxing relies on a PID namespace so that killing the sandbox init kills everything.
  • Two-stage termination with a monotonic grace. systemd's KillMode/TimeoutStopSec (SIGTERM, then SIGKILL after the timeout, to the whole cgroup), Go's Cmd.WaitDelay, .NET's Kill after WaitForExit(timeout), Erlang's port {exit_status, ...} with os:cmd kill-on-timeout — the shape is always request, wait a bounded time, force.
  • Never poll the loop thread. libuv routes blocking file and DNS work to its threadpool; Tokio has spawn_blocking; Go's os package observes a child's exit with waitid(WNOWAIT) (its wait_waitid.go) before the consuming wait4, so the observation never steals the reap right; Java's Loom parks virtual threads on a ProcessHandle completion rather than spinning.

How it works — the mechanisms ​

Observing exit without consuming it ​

MechanismPlatformsConsumes the status?Used by
waitid(P_PID, WEXITED | WNOWAIT)POSIXNoGo (os.Process.Wait), event-horizon's root observer
IORING_OP_WAITIDLinux ≥ 6.7Yes (or WNOWAIT)event-horizon (OpWaitid, SPEC §13.2)
pidfd_open + pollLinux ≥ 5.3NoTokio (pidfd reaper, tokio), systemd
EVFILT_PROC / NOTE_EXITBSD, macOSNolibuv (kqueue), event-horizon's kqueue lowering (kqueue)
SIGCHLD + waitpid(WNOHANG)POSIXYeslibuv (Unix), Node
Process handle wait → IOCP postWindowsYes (handle)libuv, .NET, SPEC §13.9

The non-consuming observation is what makes sample-before-reap possible: /proc/<pid> exists until the zombie is reaped, so the final resource sample must run while the observed exit is still pinned. event-horizon orders it explicitly — observation, final sample, reap (SPEC §13.8) — and treats a post-observation ECHILD as lost to an external reaper rather than as success or failure, because only ECHILD proves the right is gone.

Draining and framing ​

Every system drains the two pipes concurrently; they differ in what they hand the caller. Go returns whole buffers or lets the caller attach a Scanner; .NET's OutputDataReceived delivers lines (with the known null-at-EOF sentinel); libuv delivers raw chunks. event-horizon frames lines independently per stream (SPEC §13.6) with a payload-byte cap: a line longer than the cap is emitted once as a truncated head and the remainder discarded until the next terminator, so a hostile child cannot make the supervisor's buffer grow without bound, and the framer stays linear (the shipped one was quadratic on long lines — a scan restarted from the buffer's start on every chunk).

Tree containment: pgid, cgroup, job object, namespace ​

BoundaryRecursive?Escapable by the child?Kill primitiveSystems
Process groupYes (inherited)Yes (setsid/setpgid)kill(-pgid, sig)every POSIX runtime; event-horizon belt
cgroup v2Yes (inherited)Only with write access to the treecgroup.kill (Linux ≥ 5.14)systemd, Docker, event-horizon tier B/A
Job ObjectYes (inherited)Not once KILL_ON_JOB_CLOSE is setTerminateJobObject.NET, libuv (Windows), SPEC §13.9
PID namespaceYesNokill the namespace initBazel sandbox, bubblewrap

The consensus is belt and braces: the process group is always there (cheap, portable, escapable), and a stronger boundary is used where the host grants it. event-horizon's tiers (SPEC §13.7) follow that exactly — none (pgid only), owned (cgroup.kill + cgroup.events + cgroup.procs), accounted (controller-backed peaks) — and issue both the pgid SIGKILL and cgroup.kill on every hard kill, because a populated 0 reading is authoritative only at its instant.

One consequence is easy to miss: a post-spawn cgroup.procs write is not containment by construction. The kernel migrates the named process only, and a child that forked before the write lands stays outside (its zombie parent, if it already exited, migrates as a silent no-op). systemd avoids the race by having the service manager itself sit in the target cgroup before it forks, and Linux ≥ 5.7 offers clone3(CLONE_INTO_CGROUP) so the child is born inside. A posix_spawn-based library cannot use either without re-implementing spawn; it can only issue the write inline the moment the spawn returns (microseconds against the hundreds a fresh program needs to reach its first fork) and say so. event-horizon does, and names the clone3 path as the by-construction answer (PLAN M19).

Termination policy and the residual tree ​

What happens to descendants that outlive the root is a policy, and the field offers three:

PolicySemanticsPrior art
Own the treeroot exit + EOF ⇒ kill what remains; pipe-holders get a bounded output gracesystemd KillMode=control-group; Go WaitDelay
Wait for the treereturn only when the boundary reads emptysystemd KillMode=none + cgroup.events; Bazel sandbox reaping
Detachnever kill residuals; force the pipes closed after a windowGo Cmd.WaitDelay with a detached child; nohup-style daemons

event-horizon's ResidualPolicy { bounded, wait, detach } is that table verbatim, with two refinements the survey argued for: the grace under bounded is an output grace (a daemon that cleanly closed stdio gets no lifetime grace, so sh -c 'x & exit' returns at once), and wait is cgroup-scoped by definition — a descendant that left the cgroup while staying in the pgid is neither waited for nor killed, and the policy degrades to bounded with telemetry when the evidence is unavailable.

Blocking work and the one job that must not be refused ​

libuv's threadpool, Tokio's spawn_blocking, and Java's ForkJoinPool are all bounded pools with backpressure; none distinguishes a reap from an ordinary job. event-horizon's BlockingPool (SPEC §13.8) adds a termination-critical lane: intrusive, frame-resident admission (no queue slot to run out of), a reserved worker that takes nothing else, and a fairness rule under which ordinary workers help the lane only while the public queue is empty. Its guarantee is eventual FIFO service under finite jobs — which is why the lane is used for cgroup.kill (one kernel write) and the waitpid fallback, and why the cleanup's populated wait is deadline-bounded on the cgroup.events descriptor rather than open-ended.

Resource accounting that says what it means ​

Go's ProcessState.SysUsage() returns the child's own rusage; systemd's systemd-cgtop reads cgroup counters; Bazel's --experimental_collect_local_sandbox_action_metrics reads cgroup and rusage. None of them accounts a tree that mutates under a sampler. The survey's position, adopted in SPEC §13.8: every metric carries a source and a quality; sampled peaks are lower bounds by construction; tree CPU is always a lower bound while containment is unenforced; only the run cgroup's own cumulative counters are exact. Two mechanisms make that honest under pid reuse:

  • a fail-closed root anchor — /proc/<pid> held open from spawn to reap, its start time validated before and after each scan, the whole sample discarded on a mismatch (a replacement root with fresh descendants satisfies every numeric test);
  • a bounded CPU ledger — a live map with its own capacity, a scalar of retired contributions, a bounded retired-key set; disappearance (or a changed start time) retires an identity, omission never does, and once the retired set fills a later death becomes a zero-CPU tombstone in the same merge, so the total never jumps.

Analysis spine ​

Applied uniformly to the systems this survey draws on:

DimensionlibuvGo os/exec.NET ProcessTokio processsystemdevent-horizon
Exit observationSIGCHLD + waitpid; kqueue procwaitid(WNOWAIT) then wait4SIGCHLD reaper thread / handleSIGCHLD or pidfdSIGCHLD + cgroup notificationsIORING_OP_WAITID (WNOWAIT), EVFILT_PROC
Drain modelstreams on the loopgoroutine per pipethread per pipe, line eventsAsyncRead per pipejournal captureshielded drain fiber per pipe, framed on the supervisor
Tree boundarypgid (opt-in), Job Objectpgid via SysProcAttrpgid walk / Job Objectpgid (opt-in)cgrouppgid + cgroup v2 tiers
Termination policycaller'sWaitDelayKill(entireProcessTree)kill_on_dropTERM → timeout → KILL, cgroup-wideResidualPolicy, TERM+CONT → grace → KILL, both mechanisms
Blocking workthreadpoolruntime threadsthread poolspawn_blockingn/aBlockingPool + termination-critical lane
Accountinguv_getrusagerusage of the childTotalProcessorTime, peak WSnonecgroup counterssourced, qualified metrics; ledger; anchor

Strengths (of the consensus) ​

  • The observe-then-reap split is available on every platform event-horizon targets and is what makes the final sample, the sample-before-reap rule and the "lost to an external reaper" diagnosis possible.
  • Belt-and-braces containment degrades gracefully: a host without cgroup delegation still gets the process-group semantics every runtime has shipped for a decade.
  • Bounded two-stage termination is universally understood by the programs being supervised (SIGTERM handlers are the convention) and by operators (systemd's timeouts).

Weaknesses (of the consensus) ​

  • No runtime accounts the tree honestly. They report the child's rusage or a snapshot and let a reader assume it is the whole story.
  • Post-spawn containment is racy everywhere posix_spawn is the spawn primitive; only a spawn path that starts the child inside the boundary closes it.
  • Pools treat the reap like any job. A saturated spawn_blocking can refuse the one call that ends a run; a dedicated lane is the missing piece.
  • SIGCHLD is process-global. Any library that installs a handler or sets SIG_IGN breaks every other supervisor in the process; completion-based observation (waitid in the ring, pidfd, EVFILT_PROC) is the way out, and event-horizon uses nothing else.

Key design decisions and trade-offs (as adopted) ​

DecisionRationaleTrade-off
Observe with WNOWAIT, reap at the terminal boundary onlythe zombie pins pid and pgid identity for the whole run; no post-reap group signal existsa zombie lingers until streams are terminal and the tree is resolved
pgid + cgroup.kill on every hard kill, unconditionallya populated 0 reading is authoritative only at its instant; kernel idempotence makes it safea trivial run pays one kill(-pgid, SIGKILL) against a group holding only the zombie
Output grace, not lifetime grace, under boundeda daemon that closed stdio has nothing left to say; waiting idles the callera daemon that wanted to finish silently is killed at root exit + EOF
Inline cgroup.procs write, documented windowmicroseconds against a fresh program's first fork; the only lever without a new spawn pathnot containment by construction until clone3(CLONE_INTO_CGROUP) lands
Termination-critical pool lane with a reserved workerthe reap and cgroup.kill must never be refused by a full public queueone extra thread per process; eventual-FIFO, no numeric bound
Fail-closed root anchor + transactional samplesa replacement root passes every numeric test; nothing may fold until both validations passa run whose /proc/<pid> handle cannot be opened loses pid-based sampling entirely
Bounded CPU ledger with tombstonesO(live members) per sample; no overcount under pid reuse or saturationretired contributions are last-known values — lower bounds, never exact lifetime usage
Sourced, qualified metricsa consumer reads what a number is the truth abouta wider result struct; "exact" is claimable only for the run cgroup's own counters

Sources ​