Harness Intelligence Wiki
SpecsCLICLI Release Test Caching

Cache Deterministic CLI Verification Across Development, CI, and Release

Spec: Cache Deterministic CLI Verification Across Development, CI, and Release

Context

CLI developers currently repeat deterministic build, typecheck, and test work across local development, pull-request CI, and release runs. The public release command enters Turbo, but the publisher runs its verification internally, so Turbo cannot reuse those results. The present test surface also combines cache-safe behavior with tests that still observe uncontrolled operating-system or filesystem state.

Developers need passing local verification to accelerate compatible pull-request checks. Release operators need the same reuse without allowing developer-originated artifacts into the protected release trust boundary. Maintainers need complete test coverage, explicit cache eligibility, and proof that a cache hit is equivalent to fresh execution.

Non-Goals

  • Caching npm publication, npm dist-tag changes, Git tag changes, or GitHub release changes.
  • Changing the external release-state reconciliation behavior in this delivery.
  • Reusing cached results across incompatible operating systems, architectures, runtimes, or result-affecting environments.
  • Declaring every test cacheable before its inputs and execution state are controlled.
  • Allowing the current real child-process wall-clock assertion to use cache before it gains a controllable boundary.
  • Defining implementation files, task names, commands, worker assignments, or delivery order.

User Stories

US-001: Reuse compatible local verification in pull-request CI

As a CLI developer, I want a passing local verification result to satisfy the same required pull-request check when its complete cache identity matches, so unchanged deterministic work does not run again.

US-002: Preserve complete release-test behavior while caching eligible work

As a CLI maintainer, I want every release test assigned to an explicit cache policy while preserving the accepted test schedule and inventory, so caching changes latency without reducing behavioral evidence.

US-003: Reuse deterministic verification during safe release execution

As a release operator, I want new-version releases to reuse trusted build, typecheck, and test results while existing-version retries keep their fast recovery path, so release retries do not repeat unnecessary verification or unsafe mutations.

US-004: Publish only trusted, reproducible build artifacts

As a release operator, I want a protected cached build to be publishable only when its identity, signature, restored outputs, and repository state are valid, so remote reuse does not weaken package integrity.

Acceptance Criteria

  • AC-001: A passing local verification result can satisfy the corresponding required same-repository pull-request check when source, fixtures, declared root inputs, relevant environment, operating system, architecture, Bun version, Node version, and lockfile identity match.
    • Covers: US-001
  • AC-002: A mismatch in any declared result-affecting input or runtime identity causes a cache miss.
    • Covers: US-001, US-004
  • AC-003: Authenticated developer machines and same-repository pull-request CI can read and write signed development-cache entries, while fork pull requests receive no cache credentials.
    • Covers: US-001
  • AC-004: Protected-branch and release verification uses a separate signed cache namespace and never consumes a development-cache artifact.
    • Covers: US-003, US-004
  • AC-005: Every release test belongs to exactly one cacheable or uncached execution group, and an unassigned or duplicate test fails the suite-inventory contract.
    • Covers: US-002
  • AC-006: New ordinary test files start uncached; ordinary tests use file-level classification, and registered update cases carry explicit cache-policy metadata.
    • Covers: US-002
  • AC-007: Release verification runs the four update shards as one parallel wave before the ordinary CLI suite, while typecheck may run independently, and the complete accepted test inventory remains represented.
    • Covers: US-002, US-003
  • AC-008: Effect-managed timing and test-owned loopback behavior can qualify for caching when controlled by deterministic test services; the real child-process wall-clock assertion always executes uncached until it gains a controllable boundary.
    • Covers: US-002
  • AC-009: The release flow checks whether the package version already exists before deterministic verification; an existing version follows the current retry path, while a new version runs verification and checks npm again immediately before publication.
    • Covers: US-003
  • AC-010: npm publication, npm dist-tag changes, Git tag changes, and GitHub release changes always execute uncached.
    • Covers: US-003, US-004
  • AC-011: The CLI build cache identity includes every release-changelog and bundled-baseline input that affects package bytes, and a cache hit restores both the distributable output and generated bundled-baseline identity.
    • Covers: US-003, US-004
  • AC-012: The generated bundled-baseline identity is derived output and its previous generated contents do not participate in the build input hash.
    • Covers: US-004
  • AC-013: A protected cached build is eligible for publication only when its signature is valid, its complete cache identity matches, every declared output is restored, fresh and cached artifacts are byte-identical, and the worktree remains clean.
    • Covers: US-003, US-004
  • AC-014: Acceptance evidence proves forced execution, first-run misses, identical second-run hits, output restoration after deletion, invalidation for every declared input and runtime-identity change, exact test assignment, execution of uncached exceptions, and clean fresh and cached build paths.
    • Covers: US-001, US-002, US-003, US-004

Constraints

  • Cache policy applies to a complete Turbo task execution, not to one test inside a shared process.
  • A task is cacheable only when all result-affecting inputs are declared or controlled and all required file outputs can be restored.
  • Development-cache authority is accepted as sufficient provenance for required same-repository pull-request checks.
  • Protected-cache authority is separate from development-cache authority and is required for release use.
  • Cache artifacts must be signed and verified within their trust boundary.
  • Cache misses caused by incompatible local and CI environments are accepted behavior.
  • Existing external release behavior remains unchanged except for the accepted placement of deterministic verification and npm version probes.

Dependency Readiness

No Stack Required.

Remote-cache namespaces, credentials, runtime identity, and signature verification are part of this capability boundary rather than an upstream delivery dependency. No prototype or separately landed stack is required before planning.

Branch/Base Intent

Not applicable. The confirmed grill did not establish a parent branch, base branch, or child-branch constraint for downstream planning.

Accepted Technical Decisions

  • Use an uncached release dispatcher to probe npm before verification and route existing-version retries away from deterministic gates.
  • Re-probe npm immediately before a new-version publication attempt.
  • Separate cacheable and uncached tests into different Turbo task executions.
  • Classify ordinary tests by file and registered update cases through explicit cache-policy metadata; avoid title-based selection as the durable contract.
  • Preserve the four-update-shard wave before the ordinary CLI suite.
  • Treat the generated bundled-baseline identity as build output only.
  • Include platform, architecture, exact Bun and Node versions, relevant environment, source, fixtures, required root inputs, and lockfile state in cache identity.
  • Use signed development and protected cache namespaces with separate authority.
  • Keep receipt evidence limited to measured runtime and output bytes; establish protected provenance through pinned Turbo execution, protected signing authority, read-only remote-hit evidence, and attestation.
  • Permit protected release publication from a valid restored protected-cache build.

Accepted Testing Decisions

  • New tests remain uncached until explicitly classified.
  • Effect-managed time uses TestClock or an equivalent deterministic Effect test service.
  • Network-like tests use test-owned loopback services or explicit mocks when eligible for caching.
  • The current child-process wall-clock assertion stays uncached until its operating-system timing boundary becomes controllable.
  • Cache validation covers forced execution, misses, hits, restoration, invalidation, byte equality, test inventory, uncached execution, and clean repository state.
  • Local development results may satisfy required same-repository pull-request checks when the complete development-cache identity and signature match.

Verification Seams

  • Turbo's resolved task graph and cache summaries expose cache policy, dependency identity, misses, and hits.
  • The release command exposes the existing-version and new-version routing boundary before any external mutation.
  • The release-test inventory contract exposes missing, duplicate, cacheable, and uncached assignments while preserving the full suite schedule.
  • Restored distributable and generated identity outputs expose build-cache restoration and byte equality.
  • Pinned Turbo execution, signature verification, cache namespace selection, remote-hit summaries, and attestation expose development-versus-protected provenance; the receipt exposes measured runtime and restored output equivalence only.
  • Repository status after fresh and restored builds exposes unintended tracked-file mutation.

Parked Decisions

  • External release-state reconciliation
    • Owner: CLI release reliability follow-up.
    • Resume trigger: a separate request to complete missing npm dist-tags, Git tags, and GitHub release state after partial publication.
    • Preserved direction: reconciliation completes the requested channel state without republishing and stops if an existing Git tag points to another commit.

Decision Log

DecisionEvidenceRationale
Preserve existing-version fast retriesGrill Q1, Q11Static verification dependencies would run before the publisher can detect an existing npm version.
Cache the broad suite through explicit eligibilityGrill Q2, Q5, Q6, Q12Source hashing is safe only when ambient state and test assignment are also controlled.
Preserve the two-wave release-test scheduleGrill Q3Cache adoption must not silently change accepted concurrency or coverage.
Separate development and protected cache trustGrill Q7, Q10, Q15, Q17, Q22Local-to-PR reuse is accepted, while release artifacts require protected provenance.
Treat generated bundled-baseline identity as output onlyGrill Q13A generated file that is also an input can invalidate its own build key.
Cache controlled timing but exclude real OS timingGrill Q18, Q21Effect time can be deterministic; child-process wall time remains uncontrolled.
Require complete cache-behavior proofGrill Q20One observed cache hit does not prove restoration, invalidation, artifact equality, or coverage.
Park external release-state reconciliationGrill Q16Dist-tag and release-state completion is adjacent reliability work, not required for verification caching.

On this page