CI efficiency and test signal

Context

Maintainers need useful PR feedback without paying for repeated preparation, redundant proof or unrelated cache invalidation. A historical expensive CLI run took 2612.83 seconds; updater, release-recovery and built-CLI portfolio files accounted for 93.1% of test-body time. These are diagnostic timings from a failed run on earlier code/hardware, not current 2-vCPU benchmarks or promised savings.

The accepted scope combines the complete suite audit with the deeper execution traces. It reduces actual work and preserves the cheapest meaningful proof through owned interfaces. The user explicitly removed the earlier 80% target and prohibited sharding. There is no monthly minute ceiling, fixed elapsed-time target or test-count quota.

The primary owner is apps/cli for updater/release/command test capabilities. .github/workflows, root verification scripts and Turbo own CI selection, cache identity and runner execution; each other app/package retains its own tests. apps/wiki owns these requirements; implemented operator instructions remain in docs and change with implementation.

Non-Goals

• No test sharding, new shard matrix, or duplicate worker jobs distributing test subsets. Capability cache tasks may execute within existing verification jobs.
• No product feature removal, weakened ownership/containment checks, release-authority redesign, new prior-PR Candidate Evidence lookup or withdrawal of platform support.
• No new cache service, semantic-source hashing engine, test-title authority ledger, paid sticky-disk dependency, ARM migration or monthly quota enforcement.
• No production release-planner memoization or metadata-first classification in this test-fixture optimization scope.
• No claim that every historical builder remains continuously compatible after full replay moves on demand.

Requirements and Outcomes

OUT-001: Retained tests earn their execution cost

Audit the full current portfolio against TDD's High-Signal Test standard. Delete registration, source spelling, prose, fixture-history, provider-imitation and shape-only tests when they protect no owned capability. Consolidate redundant scenarios and move useful breadth to the cheapest sufficient owned interface. Preserve meaningful runtime decoding and public compatibility; static typing is not a substitute for untrusted-input validation.

Every meaningful removed or relocated scenario must have an explicit retained proof mapping covering actor, input class, authority, observable outcome, side effects and isolation. A test with no owned invariant needs no replacement. Reconcile the eight currently unselected root/operator files by current supported consumers: select unique supported proof at its proper owner and remove obsolete/duplicate proof. Do not silently add the whole unreviewed set to every run.

OUT-002: Updater tests execute only relevant expensive operations

Filesystem, ownership, receipt, authoring and failure-orchestration tests retain actual update/reconciliation behavior while controlling external installation when installation itself is incidental. Reuse immutable prepared fixture seeds with private mutable copies. Avoid full preparatory updates and repeated complete traversals merely to classify one file, manifest or proof state.

Keep distinct interruption states, live-input publication races, invalid file-kind refusal, authored-file preservation, independent partial work and convergence. Retain actual cold installation, tool executability, lifecycle-script suppression and hook integration through focused real-boundary witnesses. Controlled installed-state fixtures must satisfy real follow-up checks rather than replacing readiness with an invented healthy result.

OUT-003: Release recovery proves its protocol without full historical reconstruction

Routine CI exercises the actual planner and validator against a minimal real Git history and tiny valid artifacts. Keep first-parent relationships, exact recovery declaration/notes/tree/version binding, local remote tags, tamper rejection, partial-anchor exclusion, completed retry and provider readback behavior where owned.

The complete historical monorepo/toolchain replay stays explicitly runnable on demand and leaves ordinary CI. Avoid repeated full installs/CLI builds in those minimal fixtures. The fixture's self-validation case is not a standalone retained product proof. Product release behavior and authority remain unchanged.

OUT-004: Built CLI journeys prove composition rather than every policy permutation

Move catalog ownership, missing/old/current catalog variants, dependency-section permutations, prepare composition and preservation breadth to actual planner/reconciler/materialization interfaces with meaningful file/receipt outcomes. Keep focused executable proof for artifact startup and dispatch, aliases where packaging matters, JSON/exit/stderr semantics, scaffold-to-update composition, cold/warm cache wiring, controlled failure without consumer mutation, and meaningful delegation/transport behavior.

Remove the weak generic command matrix and registry/help accounting whose success permits arbitrary typed errors. Preserve any unique supported refusal contract in a focused test. Move the full issue-215 diagnostic benchmark out of ordinary tests and keep it on demand; the normal cache witness has a bounded cold/warm purpose. Direct shell grammar and application tests must not depend on a built artifact solely because they share its test file.

OUT-005: Cached proof invalidates only for relevant inputs

Use small capability-owned cache tasks within the existing job topology. Distinguish source, fixture, generated/shipped assets, runtime/tool/platform capabilities, environment and trust requirements according to actual consumers. Unrelated documentation changes preserve product task results; consumed prompts/skills invalidate the tasks whose outcomes depend on them.

Exclude tests and unrelated content from build keys only where build copying/generation does not consume them. Browser or Docker identity changes must not invalidate tasks without those capabilities. Retain signed cache trust, all relevant transitive dependencies, required live-readiness checks and failure behavior when reuse identity cannot be established. Keep existing affected selection and complete Stable Aggregate Check accounting.

OUT-006: CI setup and supporting suites stop repeating unnecessary work

Every automated worker uses 2-vCPU. The user subsequently authorized aligning repository and CI Bun pins to 1.4.0 if local frozen installation, builds and affected tests establish compatibility; keep historical fixture pins where their versions are the input under test. Remove automated macOS CLI integration, cross-platform cache restoration and weekly filesystem jobs. Keep meaningful Linux proof and the existing GitHub-hosted 2-vCPU Linux npm trusted-publishing boundary.

Provision Chromium only when browser work is selected, with cache identity reflecting the actual required capability. Give root cache-policy proof one execution owner without weakening the required result. Reuse a suite-owned PostgreSQL server with isolated databases/migration state where currently duplicated containers prove the same infrastructure boundary. Consolidate redundant wiki runner startup where its actual synchronization/navigation/route proof can remain. Avoid cross-package shared mutable infrastructure and new scheduling machinery.

OUT-007: Savings are demonstrated without changing verification authority

Retain PR and main verification, existing cancellation semantics, signed result reuse and current publication authority. Report passing comparable 2-vCPU execution before/after for forced test execution with warm dependency stores, normal result reuse and fully cold setup when evaluated. Separate job/workflow elapsed time from summed runner work, setup and retries. Label historical or reused timings honestly.

Completion means the identified unnecessary operations are eliminated or equivalently consolidated, required proof remains selected, and observed costs are reported. There is no minimum percentage or maximum elapsed-time acceptance gate. A faster failure or a cache hit cannot be presented as faster fresh verification. Retain the accepted branch lineage through #222's preserved head branch into #223, and document the final test/cache/runner operating model with the implementation. PR #222 is closed without merge by explicit user instruction; its inherited code remains in this branch.

Acceptance Criteria

• AC-001: The final inventory gives each existing test file a retained, moved, merged or removed disposition, includes the parent-added tests, and identifies each retained capability owner. Covers: OUT-001.
• AC-002: Every removed/moved meaningful scenario maps to retained outcome proof; no-owned-invariant culls explicitly say so. Runtime decoder/compatibility and safety proof remain selected. Covers: OUT-001.
• AC-003: All eight previously unselected root/operator files are reconciled against current consumers; no unsupported blanket selector is added. Covers: OUT-001.
• AC-004: Updater scenarios whose assertion is file/receipt/ownership behavior do not invoke a real dependency resolver solely as incidental setup; actual transaction outputs and failure handling remain asserted. Covers: OUT-002.
• AC-005: Prepared seeds eliminate repeated convergence work, and independent scenarios cannot observe each other's file, process, database or cache mutations. Covers: OUT-002, OUT-006.
• AC-006: Retained real installation/hook witnesses demonstrate tool executability and lifecycle-script suppression; controlled-process scenarios still demonstrate rollback, truthful partial results and withheld proof after failure. Covers: OUT-002.
• AC-007: Routine recovery tests use actual planner/validator and real Git/artifact boundaries to prove exact binding, tamper refusal, partial resume and completed retry without replaying the full historical monorepo builders. Covers: OUT-003.
• AC-008: Full historical replay and the complete update diagnostic benchmark have explicit on-demand entrypoints and are absent from normal test selectors. Covers: OUT-003, OUT-004.
• AC-009: Broad catalog/prepare/dependency-section policy inputs retain real manifest/receipt outcomes at their owning seams; focused built-command witnesses cover the remaining composition boundaries. Covers: OUT-004.
• AC-010: The generic command/registry matrix is removed; retained command tests assert meaningful outcomes rather than any string-valued status/error. Covers: OUT-001, OUT-004.
• AC-011: Direct application/grammar proof can run without building the CLI when it does not consume that artifact. Covers: OUT-004, OUT-005.
• AC-012: Changing unrelated documentation preserves affected product task keys/results; changing a consumed prompt/skill or transitive relevant source causes the appropriate miss. Covers: OUT-005.
• AC-013: Browser/Docker identity changes invalidate only tasks that require them; missing relevant identity or invalid trust prevents proof reuse. Covers: OUT-005.
• AC-014: Cached outputs restore correctly and every required proof is accounted for in the aggregate; cache hits do not bypass applicable live-readiness checks. Covers: OUT-005, OUT-007.
• AC-015: Workflow configuration contains only 2-vCPU worker selections, no automated macOS jobs, and no introduced test shard matrix or subset-worker fan-out. Linux trusted publishing retains its provider boundary. Covers: OUT-006, OUT-007.
• AC-016: A selection without browser work does not provision Chromium, and root cache-policy proof is no longer redundantly requested by both verification paths. Covers: OUT-006.
• AC-017: Shared PostgreSQL preparation preserves independent database/migration state; wiki proof retains its actual outcomes after runner consolidation. Covers: OUT-006.
• AC-018: The final measurement record distinguishes passing fresh execution from task reuse and reports elapsed time, total runner work and retry/setup costs without a percentage gate or monthly cap. Covers: OUT-007.
• AC-019: PR/main triggers, required aggregate accounting, current release authority and the accepted parent/child PR relationship are retained. Covers: OUT-007.
• AC-020: Operator/test instructions and affected rule verification routes describe the implemented selectors, diagnostics and cache boundaries truthfully. Covers: OUT-001, OUT-007.

Constraints

Only 2-vCPU workers; no sharding. Capability task decomposition is for ownership and reuse, not replicated execution across added runner jobs. Existing independent job responsibilities need not collapse into one workflow job. Bounded local concurrency must respect CPU/disk/process isolation rather than oversubscribe the runner.

TDD governs new or changed behavior: test the public capability, observe a meaningful RED, then implement and verify GREEN. Test-only culls and equivalent fixture moves do not need manufactured production failures or replacement tests mirroring implementation. Preserve supported source/asset ownership and release integrity.

Relevant input closure takes precedence over cache-hit rate. Purely irrelevant files can be excluded; unknown dependencies cannot. Preserve current untrusted/trusted cache separation. No new external service, spending allocation or publication permission is implied.

Dependency Readiness

Ready.

• Parent code from PR #222, branch team/stefan/cli-update-quick-fixes, is the accepted dependency. On September 22 the user explicitly requested refreshing this rebase and closing #222 before compiling the spec. Provider readback confirms #222 is closed without merge; its branch is retained.
• Grounded inherited validation/generation/commit behavior at immutable parent commit cef8e31e8a4fc7c8ac57ca7439b7590683c77a0e. Child research HEAD 9430c62496aed8b5f9a94c399d877b9f1aa7a39d contains it.
• Fresh fetch and September 22 provider readback identify parent head 4d4b7b71622ee90228143af57f995f0f6de4bff8. Rebase reports this child is up to date; Git ancestry proves that exact parent is included. Dependency readiness is based on the accepted branch/base intent and included code, not a claim that #222 landed in main.
• Parent-added project-settings and validation-plan tests are part of the current portfolio; retain their supported proof rather than duplicating parent implementation.

Branch/Base Intent

The user originally requested main ← #222 ← #223, then explicitly requested refreshing the rebase onto #222's branch and closing #222 before Create Spec. The retained branch lineage is main ← team/stefan/cli-update-quick-fixes@4d4b7b71 ← team/stefan/issue-217-ci-cost. Draft PR #223 still targets team/stefan/cli-update-quick-fixes; closing #222 neither merged its code into main nor changed #223's base. Keep the inherited parent code and branch intact.

Accepted Technical Decisions

• Optimize complete proof as a system: owned behavior, smallest adequate seam, exact relevant cache inputs, private mutable execution state, complete required aggregate.
• Keep existing planner, reconciliation, materialization, update, installer and release capabilities as owners. A small behavior-preserving testability refactor is valid where no truthful cheap seam exists; do not expose private helpers merely to assert intermediate shape.
• Use controlled external process outcomes for updater response semantics, real processes for installer/tool/hook safety, and representative built commands for composition.
• Use existing Turbo/package/workflow mechanisms for capability caches, selected provisioning and proof accounting; no new scheduler or proof registry.
• Reuse immutable fixture preparation only after portability and isolation checks. Do not share writable roots or copy large installed trees blindly.

Accepted Testing Decisions

Use the research inventory and September 22 seam map as evidence, not a blind deletion script. Preserve filesystem confinement, authored-content ownership, truthful partial failure, receipt interruption/races, missing-tool refusal, script suppression, release binding/tamper refusal, database constraints/migrations, authentication/session behavior and critical browser recovery. A meaningful regression remains useful regardless of its issue name.

Minimum retained executable categories are production dispatch, relevant alias packaging, public output/status semantics, scaffold/update composition, cold/warm cache composition, no-mutation failure and owned transport/delegation. Their broad policies belong at cheaper owned interfaces. Historical replay and broad benchmarks are diagnostic entrypoints rather than ordinary gates.

Verification Seams

Parked Decisions

• Production release-planner memoization or metadata-first selection: CLI release owner; resume only on explicit behavior-change scope.
• New cache services, paid storage or ARM migration: CI maintainers; resume only on a new infrastructure requirement.
• Publication eligibility redesign: release maintainers; resume only on explicit authority changes, independent of this performance scope.

Additional Delivery Instruction

September 22: prioritize useful local validation to avoid wasting hosted runner time. The user explicitly requested moving the Bun 1.3.5 repository/CI pin to 1.4 if compatible. The available local version is 1.4.0; align pins after local compatibility proof. This does not request a broader dependency upgrade or rewrite historical fixture versions.

Decision Log