Testing strategy

Quirl tests the same implementation at progressively wider boundaries. The canonical gate is:

cargo xtask check

It checks Rust and Quirl formatting, denies every Clippy warning, rejects undocumented public Rust APIs, builds workspace Rustdoc with warnings denied, runs the full workspace, executes 128 deterministic generated C1 differential cases against each available reference shell, runs the fixed 43-check real-PTY matrix and a seeded 12-journey keyboard session on Unix, and runs guest-side Lua tests.

Local Linux validation with Docker or Colima

The pinned container image runs the same gate with Bash and Zsh installed. It supports native Linux ARM64 and x86-64 execution. Docker Desktop or a Docker-compatible Colima daemon is sufficient; Windows is outside this workflow. CI runs the canonical gate on both Ubuntu and macOS and verifies both reference shells are available.

docker build -f scripts/linux-check.Dockerfile -t quirl-linux-check:1.99.0 scripts
QUIRL_GIT_COMMON_DIR=$(git rev-parse --path-format=absolute --git-common-dir)
docker run --rm --init --cpus 4 --memory 6g --pids-limit 1024 \
  --mount "type=bind,source=$PWD,target=/workspace,readonly" \
  --mount "type=bind,source=$QUIRL_GIT_COMMON_DIR,target=$QUIRL_GIT_COMMON_DIR,readonly" \
  --mount type=volume,source=quirl-linux-cargo-1971,target=/home/quirl/.cargo \
  --mount type=volume,source=quirl-linux-target-1971,target=/home/quirl/target \
  quirl-linux-check:1.99.0

Run from the workspace root. The common Git directory mount also supports linked worktrees and preserves accurate build identity. Source and Git mounts are read-only. Tests run as an ordinary user so root privileges cannot hide permission failures. Linux artifacts and Cargo downloads stay in separate named volumes instead of overwriting macOS artifacts. The VM needs at least 6 GiB of memory; compilation and tests each use two concurrent workers. The container does not need privileged mode or access to the Docker socket. The --init flag is required: it reaps orphaned descendants as a normal Linux init process would. Running Cargo as PID 1 can leave killed descendants as zombies and invalidate lifecycle assertions.

To run a larger simulation, append cargo xtask simulate --seed 2026090401 --sessions 2048 --steps 24 --output /home/quirl/target/simulations/linux to the same docker run command after the image name. The explicit output path retains reports in the writable target volume instead of the read-only source mount. Keep the base image version and digest in scripts/linux-check.Dockerfile aligned with rust-toolchain.toml when upgrading Rust. Container PTY checks complement, rather than replace, real terminal checks on supported hardware.

The workspace Cargo configuration omits unused full-text and spatial modules from bundled SQLite. Quirl's internal catalog, history, and intelligence stores use ordinary tables. Keep those build settings when reproducing release-size measurements. ADR 0036 makes executable size advisory by default: retain exact byte counts and the warning above 8 MiB, and investigate growth without a clear user benefit. An explicit --max-binary-bytes still enforces the caller's requested maximum. Missing measurement or identity evidence remains a failure. Historical reports retain their original limits; all latency and cleanup gates remain unchanged.

The local-completion-discovery PTY fixture opts into a fixed five-second provider budget through QUIRL_TEST_LOCAL_PROVIDER_PERSISTENCE. This isolates its metadata-persistence assertions from host scheduling, including when the harness selects an optimized artifact. The ordinary provider budget remains 400 ms, the fixture cannot extend the enclosing refresh deadline, and separate deadline and cancellation tests continue to prove child cleanup. The real Zsh adapter regression also admits an insecure function directory and proves initialization does not wait for an interactive confirmation, safe nested candidates still return, and unsafe autoloaded completion code is never run.

The durable-discovery fixture opts into QUIRL_TEST_CATALOG_PUBLICATIONS in a private temporary directory. It waits for a bounded marker naming the exact published catalog fingerprint before submitting the next command. A visible database replacement alone does not prove the worker has notified the editor. This handshake tests adoption between editor turns without input retries or changing the normal session's refresh behavior.

Asynchronous completion regressions use explicit callback-start and release signals to place cancellation between old and new queries. The first-paint test holds the extension callback while checking catalog results. Wall time bounds the tests' waits; it does not establish callback ordering or stand in for the separate enforcing release-latency measurements.

The trusted-plugin registration fixture uses an explicit five-second Lua wall budget for its real isolated syntax check, registration, and process calls. Memory and instruction limits remain the production defaults, and both metadata mismatches and incorrect process grants must fail. Production callers retain the 100 ms config policy; dedicated deadline and cancellation tests continue to verify worker termination and cleanup. Isolated-adapter functional fixtures likewise use an explicit five-second callback budget so protocol, snapshot, and output-limit assertions report their own failures. The dedicated runaway-adapter test retains its five-millisecond deadline and requires deadline-specific failure.

The retained download-channel fixture waits for an actual stalled read before cancelling it, and keeps the data sender open until it observes Interrupted. Its five-second watchdog bounds test failure and cleanup; it does not measure production HTTPS download latency or change the 25 ms cancellation poll.

Native release performance samples have one total PTY deadline (two seconds by default), shared by readiness, input, cursor replies, frame stabilization, and normal shutdown. After the latency endpoints, successful samples clear their input and request EOF, requiring exit status zero before starting the next sample against the shared fixture. Shutdown does not enter the latency metrics. The harness uses nonblocking terminal I/O and performs no terminal writes during forced cleanup. It stops on the first failed sample, preserves the requested sample count, and fails the evidence gate rather than reporting the remaining samples as successful. Stage and sample progress go to stderr so a stalled runner can be diagnosed independently of the final JSON report. Ordered timing and shutdown observations are limited to the first 101 successful samples.

Terminal dimension queries use the platform API directly. An unavailable terminal retains the caller's conservative fallback or operating error; it must not start an external tput process, including during redirected prompt probes.

The command-typeahead-redraw PTY regression holds a foreground command behind a private gate while sending the next input. It verifies the rendered spaces and the exact accepted argument bytes after control returns to the editor. The outer terminal remains raw while the private child terminal echoes input; a bounded completion handshake recovers unread child input, including an unfinished canonical line, as editable text requiring explicit Enter.

generic-terminal-session uses an unknown executable, a wrapper, and a launcher to verify all three terminal descriptors, local cursor-query replies, alternate screen redraws, resize, keyboard input, Ctrl-C, nonzero exit, stopped-program resume/cancel, output retention, and explicit pipe/file endpoints. Readiness requires a fresh input lease, a completed Quirl editor frame, and actual outer raw mode; child-terminal initialization alone is insufficient. Bash fixtures use a bounded timed read so newer Bash dispatches WINCH traps while awaiting input. Assertions inspect reconstructed screen cells when redraw bytes split text across cursor/style sequences.

The managed-clone journeys project-clone-default, project-clone-navigation, and project-clone-always exercise the original-command default, managed clone and existing-checkout navigation, persistent opt-in, and explicit-destination preservation. They use isolated Git fixtures and private project/state roots to check exact destinations, policy state, project-picker publication, and restoration of unfinished input. Service tests cover remote validation, root precedence, conflicts and symlinks, retained partial data, and one absolute metadata-planning deadline shared across sequential Git probes. Recognizer tests exclude executable paths, forced external commands, expansions, options, and compound source. These are the required verification scope; an execution claim belongs to the exact artifact's completed test report.

For optional visual evidence from these journeys, set QUIRL_PROJECT_CLONE_ARTIFACT_DIR to an existing absolute directory (at most 4,096 bytes in the path). The fixture writes reconstructed .svg screens and .screen.txt text for its fixed navigation checkpoints. Files use create-new semantics: an existing capture is an error, so choose a fresh directory for each run. This export records the tested terminal state without changing assertions.

Pipe-task descriptor assertions run in isolated, bounded helper processes. BrokenPipe and EOF require every copy of an endpoint to close; unrelated tests can briefly inherit close-on-exec descriptors between fork and exec. Isolation keeps immediate closure assertions meaningful without retrying away leaks. The parent requires a successful exit and an explicit completion marker, and bounds helper runtime, captured output, and process-tree cleanup.

Cold-start regressions use controlled worker gates to prove project database admission runs off the input thread and publishes cached repositories before discovery. Draw/provider ordering checks require the initial rich prompt context request to follow a successful flush, and reject enqueueing after a draw failure. These ordering checks complement the enforcing native latency measurements; see ADR 0035.

Layers

Release and asset contract evidence

Release tests operate on temporary repositories and fixtures. They cover the first-release 0.1.0 rule; patch, minor, pre-1.0 breaking-minor, and stable breaking-major Conventional Commit histories; unreachable and malformed tags; non-releasing commits retained in notes; idempotent preparation; curated changelog preservation; and refusal to rewrite protocol, schema, plugin API, fixture, or historical version literals.

Package tests require stable entry ordering and metadata, explicit target admission, exact candidate version/commit reporting, bounded reads, duplicate target rejection, and byte-identical aggregation. Strict versioned JSON fixtures must reject unknown fields, oversized collections and strings, invalid hashes, path traversal, unsupported schema versions, missing targets, and candidate identity disagreement. Formula tests exercise all four OS/architecture branches and prove the offline test invokes only the installed binary while asserting that every redistribution license/notice file is installed.

Runtime asset tests use an injected downloader or bounded local server, never the internet. Cross every byte, deadline, redirect, retry, manifest-entry, and retained-state boundary. Prove that the first prompt does not wait; duplicate background requests coalesce; cancellation and every partial-write fault remove staging; corrupt, truncated, incompatible, and unexpected files are rejected; permanent errors stop automatic retries; transient retry state remains bounded and survives restart; and an admitted old generation remains usable until a fully verified replacement is atomically installed.

Native command catalog evidence

The curated external native catalog has a source/artifact boundary distinct from Catalog::builtin() and the mutable CLI intelligence cache. Its tests must prove all of the following:

• valid strict KDL produces the expected typed command tree, while malformed KDL retains source identity, UTF-8 byte spans, and actionable help;
• unknown nodes/properties, duplicate properties and identifiers, extra positional values, typed KDL annotations, invalid platforms/actions, and impossible flag or argument combinations fail closed;
• source bytes, string bytes, total commands/flags/arguments/documents, per-command values, root-inclusive depth, database bytes, query bytes, documents scanned, and result counts trip at the declared boundary and report configured plus observed usage;
• compiling the same typed tree twice yields byte-identical QCNC SQLite, and reader admission rejects wrong identity, corrupt integrity, unknown snapshot fields, or any normalized-row change relative to the exact snapshot;
• command/alias, flag, argument, platform, and semantic projections have stable ordering; child platforms narrow rather than widen parent support; lexical ranking is platform-filtered and deterministic; and no completion action runs during compilation or lookup;
• publication fault injection after staging creation, write, content sync, staged reread, destination recheck, and before rename preserves the previous admitted image and removes the current staging file; contention and unsafe file shapes fail without partial publication; and
• a pinned Carapace import is bounded and writes only a review draft. Tests must show that dirty pinned files, duplicate facts, cross-command action leakage, and invented positional metadata fail or become explicit omissions; a partial/invalid import cannot change curated KDL; formatting is byte-idempotent; unchanged canonical KDL preserves the database checksum; and no runtime test requires Carapace or network access.

CI should run the implemented format-check, strict-check, and build operations, compile twice, compare exact bytes, reopen through the hardened reader, and publish the database checksum with the source revision. The tooling command names and corpus paths are deliberately deferred to the integration change; tests and docs must use the four operation roles from the catalog contract until then.

Reproducing generated failures

The default generator is intentionally stable. To increase exploration locally:

cargo xtask test --seed 123456789 --cases 2048

Case counts are restricted to 1..=10000. A failure message contains everything needed to replay it. Never seed from wall-clock time inside a test; a scheduled or external swarm may choose seeds, but it must record them before execution.

Stateful compatibility simulation

The inspectable compatibility swarm is a separate, wider gate:

cargo xtask simulate --seed 123456789 --sessions 2048 --steps 12

Each generated session has 3..=steps stateful operations. It starts in an isolated directory, changes into a session workspace, exports a value, and then mixes bounded C1 command lists, pipelines, redirects, append operations, parameter and arithmetic expansion, command substitution, here-strings, status propagation, stdout, and stderr. The final observation exposes retained environment and filesystem state. Quirl, clean Bash, and clean Zsh each receive an identical fresh filesystem and an environment reduced to explicit locale, path, home, temporary-directory, and terminal values.

Both references are required. A Bash/Zsh disagreement is reported as reference_divergence, because the generator has escaped the promised common subset or a reference changed. When the references agree and Quirl differs in status, exact output, or the bounded filesystem manifest, the result is native_mismatch. No majority vote can hide a divergence.

The seed-specific directory under target/simulations/ contains:

• summary.json, including result counts, exact interpreter paths and versions, the original Quirl path, its tested snapshot SHA-256 and byte size, and the first divergent session;
• report.jsonl, with every source, step list, outcome, filesystem manifest, deadline result, and classification;
• failures/session-N/, with standalone .sh and .qrl replay sources plus the complete case record; and
• issue.md, only on mismatch, with the exact focused replay command.

To rerun only a reported session while preserving generator position:

cargo xtask simulate --seed 123456789 --sessions 2048 --steps 12 --session 731 --output target/simulations/replay

Sessions are bounded to 1..=10000, steps to 3..=32, source and retained output to 64 KiB, the filesystem manifest to 64 files and 64 KiB at depth eight, and every interpreter to five seconds. Timeouts terminate the isolated process group before the result is recorded.

Before the first session, the simulator copies Quirl from one admitted regular file handle into a private temporary directory. The copy uses a 64 KiB buffer, accepts at most 256 MiB of executable bytes, and hashes exactly what is copied. Observed in-place changes reject admission; later rebuilds cannot switch the binary between sessions. All sessions execute that owner-only snapshot, which is removed by the run's cleanup guard. Summary schema 2 adds snapshot identity; individual report.jsonl records retain schema 1.

The daily GitHub Actions swarm chooses the explicit seed from the workflow run ID, installs both references, runs 2,048 sessions, and uploads the trace. It opens a public issue only when a completed summary.json proves a compatibility mismatch; checkout, installation, build, or runner failures fail the workflow without being mislabeled as shell defects.

Keyboard-driven user sessions

The compatibility swarm compares shell execution. The session soak exercises the interactive product through real Unix PTYs, using the same isolated configuration and terminal model as the rich-terminal checks:

cargo build -p quirl-cli
cargo xtask session-soak --seed 2026090501
cargo xtask session-soak --seed 2026090501 --sessions 100 --journeys 60

The short default runs four sessions with 12 journeys each. The explicit 100-session invocation runs 6,000 journeys. At the explicitly assumed rate of 60 command journeys per hour, that is 100 modeled active hours with think time removed. It is an action-coverage model, not a measured human usage rate or evidence of 100 hours of continuous uptime. Reports record actual elapsed time and input counts separately. A journey can contain several commands and navigation actions.

The harness uses an immutable, hashed copy of the selected prebuilt binary. Pass --binary /absolute/path/to/quirl to test another build. Replaying a zero-based session requires the original seed and workload parameters, for example --seed 2026090501 --sessions 100 --journeys 60 --session 17. Keep the original executable when investigating a failure: rebuilding changes what a replay tests even when its input sequence is identical.

The workload combines editing, completion, history, mode changes, typed data, help, invalid-input recovery, and terminal resizing. It records input actions and representative visible-screen checkpoints under target/session-soak. Open the run's index.html to browse its screen gallery and measured counters; manifest.json records replay inputs and summary.json records the outcome. Each session retains actions.jsonl, a summary, and bounded final screen/output evidence. A failed session also retains failure.svg when artifact writing succeeds. Logs record input before delivery; artifact or cleanup failures also fail the run. Reports use private, newly created directories and never replace an earlier run.

Screens are escaped, bounded SVG styled terminal-cell models of the actual PTY output. They preserve ANSI, indexed, and truecolor foreground/background colors and common SGR styles using an explicit fixed palette. They make clipping, selection, stale overlays, wrapping, and misplaced cursors inspectable; they do not reproduce the user's terminal palette, font shaping, or exact pixels. Each cell admits at most 256 UTF-8 bytes; exceeding that bound fails the model and capture instead of silently publishing truncated evidence. Visual checkpoints wait for Ratatui's completed cursor-finalization sequence within the same bounded screen deadline, so a matching word halfway through a redraw cannot publish a partially painted gallery image. Failed checkpoints are not added to the gallery index. Inspect failure traces and images before attributing a failure to Quirl rather than the harness. Each run allows at most 1,000 sessions, 200 journeys per session, 180 seconds per session, two hours overall, and 20 failed sessions. Journals are capped at 1 MiB each and run artifacts at 256 MiB, with space reserved for summaries. Per-action screen waits are limited to eight seconds. The workload stops with failure when a bound prevents completion.

The fixed rich-terminal matrix also reopens the same private profile to verify durable history and tests multiline bracketed paste against filesystem side effects. Pasting and selecting history must not execute a command; explicit submission must execute the intended commands exactly once. Session fixtures never use the developer's personal home, configuration, or history.

The fixed matrix also covers pasted control sequences, oversized paste rejection, fragmented committed Unicode and grapheme editing. Transport-level checks exceed pending-paste, escape-sequence, and terminal-reply queue limits, requiring resource diagnostics, no partial execution, and terminal restoration on both rich and simple surfaces. ADR 0032 records the dependency ingress patch. Both editors bound source to 64 KiB. The rich editor rejects the entire paste and preserves existing input; the simple editor fails closed on overflow and restores the terminal. ADR 0033 also bounds simple undo retention and Vi repeat expansion, and requires displayed controls to remain inert. An actual idle unfinished-paste check waits for the 30-second admission deadline without sending another byte. A separate canonical zero-poll probe verifies immediate kernel and parser-queued events; it tests the harness's linked terminal library separately from the selected Quirl executable.

The vi-repeat-admission PTY check sends rapid insertion and Escape in one write and verifies exact post-edit source. It also rejects excessive decimal counts, multiplied counts, and overflowing prefixes before queued submission, proving terminal restoration and absence of command side effects. The simple editor bounds both raw collection and expanded actions as specified in ADR 0033.

Keyboard clipboard tests select Unicode output rows, issue copy commands, and verify exact single-line and multiline OSC 52 payloads. They do not access the operating-system clipboard. Fixed PTY checks pin one immutable binary for the whole matrix and print its SHA-256, so a concurrent rebuild cannot switch the product halfway through a run. Large PTY writes must drain output concurrently under the original send deadline: a real terminal continues reading while a shell paints, and a write-only harness can deadlock against the child's output buffer. Query replies, retained output, and screen parsing keep their existing bounds; backpressure must not be hidden by increasing timeouts.

sustained-session keeps one shell alive across 128 output bursts (over 64 MiB), 64 error/pipeline/editor-cancellation rounds, and four foreground interruptions. The first 32 MiB warms child scrollback and session state before measured churn. Compact terminal snapshots mean this does not establish parent transcript wrap; separate UI retention tests exercise that limit. Linux samples descriptors, threads, resident memory, and direct children from bounded /proc reads at settled prompts. The warmed growth envelope is eight descriptors, eight threads, and 32 MiB RSS; successful exit must reap owned children. Other platforms verify responsiveness and child cleanup. The fixed workload has a 240-second aggregate admission budget, accommodating slower hosted macOS debug runners without reducing any work or resource checks. Every screen, input-readiness, and exit observation must still complete within five seconds and the remaining aggregate budget, so a stalled command fails promptly. This checks accelerated child scrollback eviction and session churn, not 100-hour continuous uptime or proof of zero leaks.

No finite generated workload emulates every terminal or usage pattern. Keep the focused process, job-control, sandbox, protocol, and release tests. Real terminal checks remain necessary for colors, fonts, IME input, clipboard integration, accessibility, terminal-specific key protocols, remote sessions, and real Codex authentication/network behavior. The accelerated soak does not replace actual long-duration memory, idle, suspend, or clock-change testing.

Lifecycle simulation

The process lifecycle simulator adapts TigerBeetle's safety-to-liveness simulation to a local shell process group:

1. A bounded safety phase schedules stop, continue, exit, duplicate, and stale child notifications in a seeded order. Every step checks that invalid transitions leave committed state unchanged and that the aggregate job state matches the child states.
2. The liveness phase freezes completed children permanently, heals the live core by continuing stopped children, stops injecting faults, and requires the process group to reach done within at most two transitions per live child.

The same seed also schedules real construction failures after each of the first four process spawns on Unix. Dropping PipelineConstructionGuard must kill and reap every child already owned by the incomplete transaction. These OS-process cases are capped at 32 per run even when the pure simulator is asked to run more cases; both bounds are intentional so the canonical gate remains predictable.

The model is not a replacement for real PTY evidence. It makes state-machine and cleanup schedules cheap to explore and replay; cargo xtask rich-pty separately proves native Ctrl-C/Ctrl-Z, jobs/bg/fg transitions, process-group construction cleanup, foreground ownership, stopped-job and prompt termios restoration, dialect-island noninteraction, and rendering against the operating system.

The PTY checks live in xtask/src/pty.rs and xtask/src/rich_pty.rs. The Rust-owned driver reuses the workspace's existing nix and unicode-width dependencies and requires no separate scripting-language toolchain. Its small VT screen model answers cursor-position queries from modeled screen state, so assertions distinguish visible cells from stale bytes in the raw transcript. Each session limits retained output to 16 MiB, each read/write/wait to 15 seconds locally (100 seconds on GitHub Actions), the screen to 262,144 cells, and forced child cleanup to two seconds. Session teardown kills both the foreground process group and the PTY session group before reaping the leader.

external-command-compatibility installs an isolated GHQ-shaped executable and Fish completion file. It proves that an scp-style repository argument reaches the child unchanged, colored progress becomes visible before a one-second child finishes, carriage-return progress replaces its prior value, alternate-screen ownership remains stable, and completion exposes real subcommand paths and descriptions rather than importer implementation names. Fixtures use no network and mark completion independently so a final-output-only renderer cannot pass by racing the assertion.

local-completion-discovery uses the same GHQ shape with controlled fake Fish and Zsh executables. Neither host shell nor host completion root is consulted. The check proves that initial top-level discovery persists framed subcommands, flags, descriptions, and provider attribution, then that an editor-observed unknown nested path is coalesced, probed off the render thread, published, and adopted only after the next editor boundary. The focused CLI model tests cover warm positive and negative restarts, expiry and identity invalidation, malformed/excessive frames, deadlines, cancellation, descendant cleanup, and queue/concurrency limits without requiring Fish, Zsh, Carapace, or network.

Run the harness model tests and one focused end-to-end interaction with:

cargo build -p quirl-cli
cargo xtask rich-pty --check mode-switch-and-palette-screen
cargo xtask rich-pty --check external-command-compatibility
cargo xtask rich-pty --check local-completion-discovery

The focused interaction sends the legacy-terminal encoding for Alt-Q and Alt-Q leader chords exactly. It asserts that repeated mode switches retain the edit buffer without feedback scrollback, the palette status is on the physical bottom row, and dismissal erases the expanded viewport before restoring the compact prompt.

Adding coverage

• Add a frozen fixture for a discovered regression before fixing it.
• Extend a seeded generator only with syntax inside the supported native C1 contract; unsupported Bash/Zsh syntax belongs in explicit-island tests.
• Prefer a real implementation or a small independent model as the oracle.
• Test both success and failure, including cancellation and cleanup after a partially completed operation.
• Formatting fault tests inject failure after temporary creation, partial output, flush, content sync, permission update, metadata sync, original retention, rename, and parent sync. Every returned failure must preserve the original source bytes and remove transaction files. Authoring discovery tests cross every depth, directory, per-directory entry, total entry, supported-file, retained-path-byte, and scanned-name-byte limit and cover symlink, permission, and disappearing-entry behavior without recursive traversal.
• For a liveness claim, separate a fault-exploration phase from a healthy phase with a named progress bound. Do not let later random recovery hide a stuck state.
• Bound generated input size, case count, subprocess output, and elapsed time.
• For terminal work, verify NO_COLOR, TERM=dumb, narrow widths, Unicode graphemes, wrapped lines, and hostile control characters.
• For external-command regressions, assert an intermediate state before child exit as well as final bytes/status. Run the same bounded argv/output fixture through clean Bash and Zsh when the claim concerns shell semantics; use the PTY model when the claim concerns visibility or terminal state.

The rationale and non-goals are recorded in ADR 0011.