sparkles:shaders — Testing and evidence
Oracles, acceptance scenarios and the evidence ledger for the requirements in SPEC.md. The library spans three implementations — the vocabulary, the compiler's fragment stage and the build step — so each requirement says which of them it checks and which oracle backs it.
Oracles
| Oracle | Independent because | Checks |
|---|---|---|
| Hand-derived values | Worked from the GLSL specification's definitions, not from this library's code | every built-in at ordinary, boundary and sign-changing inputs (SHV5, SHV6) |
| Two host compilers | DMD and LDC compile the vocabulary with different representations (SHV2) and different math libraries | host behaviour that must not depend on the compiler (SHV1–SHV6) |
spirv-val | The Khronos reference validator, a separate implementation of the SPIR-V rules | the compiler's device module (SHP4) |
glslangValidator | The Khronos reference GLSL front end | every generated file is a valid, complete shader (SHP6) |
| SPIR-V disassembly | spirv-dis output read as text, independent of the compiler's code generator | interface variables, locations, bindings, names and handle storage (SHF1–SHF3, SHF5, SHF6) |
| GPU readback | A GL driver (the software rasterizer in CI) executes the generated GLSL; the host executes the D function | host/device agreement within the D12 bounds (SHV7, SHF4) |
| Recorded dub behaviour | Scratch dub packages whose configurations are resolved by dub itself and inspected with dub describe | configuration selection and leakage (SHB5, SHB6) |
| A build without the device compiler | A Nix sandbox or CI leg whose toolchain lacks ldc2-vulkan: an opt-in that leaks cannot build there | default configurations need no device compiler (SHB5, SHB6, SHT4) |
The compiler is both the subject of section 6 and the tool that produces the evidence for it, so its output is checked by readers that share no code with it: the Khronos validators and the SPIR-V disassembly. Differential tests against GLSL that a person writes by hand are not used; they would test the person.
Scenarios
Vocabulary (SHV1–SHV8)
- Construct each vector type with each constructor, read every component and swizzle, and compare with hand-written expectations, under DMD and LDC.
- For each built-in, evaluate negative, zero, positive and non-integer inputs, and inputs on both sides of each discontinuity (
floorat −0.5,modwith a negative dividend,stepat its edge,smoothstepat both edges). Expectations come from GLSL's definitions. - Agreement. A probe unit with one fragment entry point per group of built-ins is compiled by the build step; each probe reads its arguments from a float texture, one sample per texel, and writes four results to a float render target. The results are read back and compared with the same functions evaluated on the host, within the
D12bound for each operation (ui_raylib.shader_readback.hostAndDeviceAgree, run in CI under a virtual display on the software rasterizer). The GLSL ES output is not read back. - Compile a module that uses every vocabulary symbol and a
@computeattribute with DMD, without a device target, and check that it builds and that no device rule fires.
Device modules (SHM1–SHM5)
- Classify modules spelled with each mode, with
@computequalified, preceded by comments,deprecatedand other attributes, and withhostAndDeviceappearing outside@compute's arguments or inside a string; expect the classificationSHM5defines. - Seed each device rule's violation into a device module and expect the device compilation to fail with LDC's message. The editor analysis checks the same rules separately (
TGT7).
Fragment entry points (SHF1–SHF7)
- Compile entry points with zero, one and several parameters of each kind, in interleaved orders, and check in the disassembly that each
@inputhas the next location in parameter order, eachSampler2Dthe next binding from 1, the sampler binding 0, each uniform the parameter's name, and the outputfinalColorat location 0. - Compile a parameter with no marker, a parameter the calling convention rewrites, and a stage other than
@fragment, and expect a diagnostic for each. - Check in the disassembly that no image or sampler type appears as a member of a composite or as the pointee of a Function-storage variable.
spirv-valunder the universal rules does not check this; the Vulkan environment would, but also rejects the uniforms ofSHF2.
The build step (SHP1–SHP10)
- Parse
dub describeoutput with resolution chatter before it, repeated flags and several targets, and check the settings, the target and the sources taken from it. - Point the step at an unknown configuration, one without
-mdcompute-targets, one naming two targets and one without a device-only module, and expect exit 2 with the cause; with noldc2-vulkanonPATH, and with that and an unknown configuration, expect exit 3 and instructions; at a stock LDC, expect exit 3 from the capability probe. - Edit an input while the step compiles and expect the stamp to be stale.
- Run with
DFLAGSset to a host-only flag (--threads=3,-fsanitize=address) and expect identical output. - Generate twice, from two directories, and expect byte-identical files and provenance comments.
- Run several instances of the step against one output directory at once and expect complete files and a stamp that vouches only for them.
Build integration (SHB1–SHB7)
- Generate, then change each kind of input in turn — a device module, a dependency's device module, a contributing recipe — and expect the stamp to go stale after each; delete an output and expect the same.
- Write a
prebuiltstamp and expect freshness with no compiler present. - Add a device module that no existing input imports and expect the stamp to go stale, then remove it and expect the same (
SHB4). Compute the stamp's digest for the same tree in two directories and on two platforms and expect it to be identical. - Resolve, with dub, a consumer that selects the opt-in configuration in a parent configuration it does not use, and expect the dependency's default configuration.
Evidence ledger
Statuses follow Writing Specification Docs: verified means the named tests passed for the named configuration, partial names the missing case, unverified has no evidence yet, planned is scheduled in PLAN.md, and deferred waits for a prerequisite PLAN names. Host tests run under DMD and LDC on every CI leg; device evidence comes from the builds that generate GLSL, on x86_64 Linux, aarch64 Linux and aarch64 macOS.
| Requirement | Status | Evidence | Gap |
|---|---|---|---|
| SHV1, SHV3, SHV4 | verified | shaders.types.constructAndRead, shaders.types.arithmeticIsComponentWiseAndBroadcasts | — |
| SHV2 | verified | shaders.types.oneLayoutOnBothSides asserts every vector's size and alignment, a struct and an array holding vec3, and the packed variants, under DMD and LDC; every generated-shader build compiles the device vec3 | — |
| SHV5, SHV6 | verified | shaders.math.builtinsMatchGlslDefinitions (hand-derived values); ui_raylib.shader_readback.hostAndDeviceAgree (sin, cos, exp and pow against the GPU's, within the D12 bounds) | — |
| SHV7 | partial | ui_raylib.shader_readback.hostAndDeviceAgree: every built-in, read back from the GLSL 3.30 output on the software rasterizer, within the D12 bounds (CI x86_64 Linux) | the GLSL ES output (highp, D15) is not read back: the readback has no OpenGL ES context |
| SHV8 | verified | ui.effect.builtins.areHonouredByACellGrid runs hostAndDevice shader code under DMD with no device target | — |
| SHM1, SHM2 | partial | shaderCompile.unitOf.repositoryUi; ui_raylib.effect_gpu.builtinsCarryTheirGpuHalf passes only when the device build ran | no test compiles a deviceOnly module and checks it is absent from the host build |
| SHM3 | partial | the pinned compiler's own pass; dmd_lsp.testing.dcompute.rules mirrors it | no seeded-violation test of the device build itself |
| SHM4 | partial | a violating module fails the device build (SHM3) | nothing flags it before the device build does |
| SHM5 | verified | shaders.compute_mode.computeModeOf | — |
| SHF1, SHF3 | verified | shaderCompile.fragment.interfaceFromTheSignature: the generated GLSL of an entry interleaving every parameter kind and of a void one equals apps/shader-compile/tests/golden/ — entry points named after the D functions, finalColor at location 0, no output for void | — |
| SHF2 | verified | shaderCompile.fragment.interfaceFromTheSignature (inputs and plain uniforms named after the parameters in the golden GLSL; input locations and image and sampler bindings in the module); shaderCompile.cli.interfaceContractIsEnforced (an unmarked parameter rejected naming it; two entries giving fragColor different locations rejected naming both) | — |
| SHF4 | partial | ui_raylib.shader_readback.hostAndDeviceAgree (probeSample returns each texel exactly, GLSL 3.30) | not read back from the GLSL ES output |
| SHF5 | verified | shaderCompile.cli.interfaceContractIsEnforced (an int input, a uint uniform, an array input, a vec3 return: exit 1 naming the parameter and the reason); ui_raylib.shader_readback.hostAndDeviceAgree (probeUniforms: an int and a struct of float, int and vec3 set by name, read back) | — |
| SHF6 | verified | shaderCompile.cli.interfaceContractIsEnforced (a uniform struct holding a Sampler2D is rejected); the golden GLSL of shaderCompile.fragment.interfaceFromTheSignature holds no local sampler | — |
| SHF7 | verified | shaderCompile.cli.interfaceContractIsEnforced: another stage is rejected, naming it | — |
| SHP1 | verified | shaderCompile.parseDescribe.rootSettingsAndEverySource; shaderCompile.cli.exitCodes (exit 2 for a configuration without the target flag) | — |
| SHP2 | verified | shaderCompile.withoutHostFlags.dropsDflagsOnly; CI test legs, whose DFLAGS carry --threads=N, generate successfully | — |
| SHP3 | verified | shaderCompile.unitOf.repositoryUi | — |
| SHP4 | verified | shaderCompile.cli.aFailedValidationWritesNothing (exit 1, the earlier output and stamp untouched) | — |
| SHP6 | verified | every generated-shader build runs glslangValidator; shaderCompile.cli.aRejectedTranslationWritesNothing (a sampler named texture: exit 1, the earlier output and stamp untouched) | — |
| SHP5 | verified | shaderCompile.renameCombinedSamplers.givesTheImageItsNameBack, shaderCompile.entryPoints.readsFragmentEntriesInOrder | — |
| SHP7 | verified | shaderCompile.header.namesTheEntryModulesRelativeToTheirPackage | — |
| SHP8 | verified | shaderCompile.cli.staleOutputGoesAndConcurrentRunsAgree (a stale file removed, no temporaries left, four concurrent runs leave complete output and a fresh stamp) | — |
| SHP9, SHP10 | verified | shaderCompile.cli.exitCodes (0 generated and fresh; 2 for an unknown configuration, one without the target flag, one naming two targets, one without a device-only module; 3 without the compiler, also over a configuration error; 1 for a compile error); shaderCompile.cli.stockCompilerIsAToolchainError (3 by the capability probe) | — |
| SHB1–SHB3 | verified | shaderCompile.stamp.freshUntilAnInputChanges; shaderCompile.cli.anInputEditedDuringTheRunLeavesItStale (hashes taken before compiling) | — |
| SHB4 | deferred | M2 | a new device module is unseen until another input changes |
| SHB5 | verified | ui.effect.builtins.carryOnlyTheirCpuHalfWithoutGpuEffects, ui_raylib.effect_gpu.builtinsCarryTheirGpuHalf | — |
| SHB6 | verified | CI builds the tui-only examples in a sandbox without ldc2-vulkan; ui_raylib.effect_gpu asserts hasGpuEffects | the dub leak itself is recorded by hand, not by a test |
| SHB7 | verified | the consumer recipe uses --temp-build; shaderCompile.cli.staleOutputGoesAndConcurrentRunsAgree runs the built step in four processes at once on one output directory | — |
| SHT1, SHT2 | verified | the flake pins every component | — |
| SHT3 | verified | .github/workflows/ci.yml: the content hash of ui-shaders built on x86_64 Linux and aarch64 macOS agreed (#620) | aarch64 Linux is not compared |
| SHT4 | verified | CI test legs: generated-shader builds on x86_64 Linux, aarch64 Linux and aarch64 macOS; default configurations on Windows | — |