Maximal Test Caching and Green-Main Release Gate Research

Scope and method

Three readonly lanes inspected:

1. every retained GitHub Actions run and the relevant failed job logs;
2. the current local, pull-request, protected-main, scheduled, and release-dispatch graphs;
3. every current CLI cache class plus representative generic workspace tests.

The live Actions inventory covered all 384 retained runs from 2026-07-14 10:19 UTC through 2026-08-10 12:47 UTC. All runs belong to the single Behavior contract workflow. The audit then inspected every green-PR-to-red-main case, every main and release-dispatch run since PR #100, and representative older failures. Static source evidence is from main at caee8be43d38bfaaaff4cc3c90cb14c8b82c245a.

This report records facts and implications. It does not select the remaining product policies.

Executive conclusion

The observed problem is not one general class of tests that passes on a PR and fails after merge.

• Three verified green-PR-to-red-main cases came from protected-only attestation, cache authentication, or task-graph behavior. PR #115 fixed that graph and then passed on both the PR and main.
• PRs #119 and #120 were already red before merge, but the repository still allowed them to merge.
• The following seven failing main commits had no associated pull request. They were direct pushes containing both real contract drift and recurring test-harness flakes.
• The repository is private under the wearedevpunks GitHub Free organization plan. Live ruleset and branch-protection API requests returned HTTP 403. GitHub documents that private organization repositories need GitHub Team or Enterprise for protected branches.

The cache concern is also valid. Several current uncached groups are conservative buckets rather than irreducibly uncached work. update:uncached can become host-capability cacheable, and most of the ambient file can move into the same class after it is split. Generic workspace tests need equivalent portable, host-capability, and live-integration boundaries before they receive shared credentials.

Complete Actions inventory

Primary evidence: the live GitHub Actions API for workflow 312920364 and the repository's Actions history. The API reported exactly 384 retained runs and zero schedule events.

Since PR #100 merged, the exhaustive focus window contains 71 runs. In that window, main had one success, twelve failures, and four cancellations.

What actually failed after green pull requests

PR #100: protected-only timeout

PR #100 passed PR run 31225493732. Its merge 89cd741c failed in main run 31226447260.

Protected attest job 93024651885 reran update shard 2. Eighteen of twenty-three tests reached the 30-second timeout. This was a protected-only graph/runtime defect, not a newly discovered product assertion.

PR #112: protected cache authentication

PR #112 passed PR run 31234609691. Its merge 7a88d085 failed in main run 31235511499.

Protected attest job 93050565938 reported remote-cache authentication failure and insufficient write permission, then could not restore @punks/cli#build. This was cache authority/configuration failure.

PR #114: protected task-graph mismatch

PR #114 passed PR run 31239750346. Its merge 3f5f9622 failed in main run 31240693216.

Protected attest job 93063704387 restored build but missed release:attest. The grouped task invocation had a different hash from the producer. This was a protected task-graph defect.

PR #115: corrected fixed point

PR #115 passed PR run 31244179458. Its merge c4dcca31 passed main run 31245179530, including the grouped protected producer/restore proof.

The verified green-PR-to-red-main cluster is therefore three failures while the protected graph was converging. It is not evidence that ordinary product tests routinely become red only after merge.

Red merges and direct pushes

Two pull requests merged while already red

• PR #119 failed in PR run 31373515283, then merged into failing main run 31373518818.
• PR #120 failed in PR run 31373970443, then merged into failing main run 31373984194.

Their logs include 15/30-second timeouts, spawnSync bun ETIMEDOUT, and missing temporary commands.jsonl files. These are recurring test-runner/orchestration faults, but they were visible before merge.

Seven later main commits had no pull request

GitHub's commit-to-PR API returned no associated pull request for each commit below.

These are unvalidated direct pushes, not green pull requests that changed behavior after merge.

Release-only failure evidence

• Dispatch 31247845153 restored protected build and attestation, then stopped at Refusing to publish without an authenticated npm session. The failure was npm authentication, not testing.
• Dispatch 31368684924 failed twice before npm because update shard 2 repeatedly timed out; one attempt was cancelled.
• Dispatch 31385584319 was cancelled after approximately 29 seconds and has no failure diagnosis.

Four main runs after green PRs #109, #113, #116, and #117 were cancelled. Retained GitHub metadata does not identify the cancelling actor, so manual cancellation versus workflow supersession remains unresolved.

Why current PR and main graphs differ

The workflow triggers on every pull request, every push to main, weekly schedule, and manual release dispatch (.github/workflows/behavior-contract.yml:3-17).

Root scope

Pull-request root verification is diff-aware. Workspace-only changes use Turbo affected selection. Global or unknown changes, missing refs, empty diffs, and non-PR events run the full graph (scripts/behavior-contract/run-test-scope.ts:20-98). A green affected PR can therefore cover less root work than its full main run.

CLI scope and authority

Internal PRs use the signed development cache and execute typecheck, four update shards, deterministic tests, four native-capability partitions, and ambient tests (.github/workflows/behavior-contract.yml:57-210).

Main and schedule use the separate protected authority, repeat update and deterministic work, force all native tests uncached, run ambient, and then run protected attestation (.github/workflows/behavior-contract.yml:332-473).

Protected attestation creates a detached clean producer, performs a frozen install, reruns build/typecheck/tests, produces release evidence, and requires grouped build plus release:attest restoration (apps/cli/scripts/run-cli-verification.mjs:267-319, 672-777). It is structurally post-merge-only today because it requires a committed tree and live GitHub OIDC.

The attestation producer also strips LANG, LC_ALL, and NODE_ENV, while ordinary verification canonicalizes them (apps/cli/scripts/run-cli-verification.mjs:53-69). Main therefore repeats suites under a release-shaped environment that PRs do not use.

Missing release-candidate checks

test:browser and validate:packaged-product exist but are not selected by PR, protected-main, or release-dispatch verification (package.json:58-63). There is also no semantic release-impact classifier in the workflow.

Branch enforcement facts

The repository is private and owned by the wearedevpunks organization. The live organization record reports the Free plan. Both:

• GET /repos/wearedevpunks/harness-intelligence/rulesets, and
• GET /repos/wearedevpunks/harness-intelligence/branches/main/protection

returned HTTP 403 with the instruction to upgrade or make the repository public.

GitHub documents that protected branches are available for public repositories on Free, but private organization repositories require GitHub Team or Enterprise: About protected branches.

This explains how red PRs and direct pushes reached main. Repository workflow code cannot itself prevent a write to an unprotected branch.

Cacheability findings

Current CLI slices

Primary implementation sources: turbo.json:13-455, apps/cli/scripts/host-test-capabilities.mjs:434-665, apps/cli/scripts/release-test-policy.mjs:97-145, and apps/cli/scripts/run-release-tests.mjs:35-136.

Why the current portable class is not yet cross-OS

The deterministic task hashes platform, architecture, Node, Bun, CI, namespace, and canonical environment (turbo.json:239-290). This prevents a macOS local result from satisfying Ubuntu CI.

The current deterministic inventory also contains tests that branch on process.platform, including apps/cli/src/scaffold/gitignore.test.ts, output.test.ts, and skill-snapshot.test.ts. Those files must move to a native class, or their platform behavior must be controlled, before removing OS/architecture from the portable hash.

Ambient split

The ambient file mixes two different kinds of work:

• genuine simultaneous writers, FIFO coordination, stale-lock contention, spawned-process shutdown, and elapsed-time assertions;
• controlled hard-link, symlink, descriptor, long-name, oversized-input, and permission behavior.

Transient scheduling after task start cannot be represented completely in a pre-run cache key without removing the concurrency property under test. The controlled filesystem cases can use the existing capability profile. Sources: apps/cli/src/data/scripts/sync-subagents.native.test.ts:350-881, 1254-1575, 1756-1787, and 1868-2729.

Update shard redesign

The current uncached update policy is declarative rather than forced by an observed ambient dependency (apps/cli/src/update/run.shard-3.test.ts:6-10). Its cases exercise subprocesses, symlinks, modes, atomic parent swaps, receipt confinement, projection lifecycle, and ownership overlap. Those inputs match the existing kernel/filesystem/tool capability profile. A host-capability task can therefore cache them while preserving native semantics.

Generic workspace tests

Generic tasks currently hash only CI, NODE_ENV, and TZ; HOME, temporary directories, Git configuration, and XDG roots are pass-through (turbo.json:457-604). The root behavior-contract job has no authenticated remote cache setup.

The current generic suite mixes pure tests with:

• Docker/Postgres container lifecycle in packages/auth and packages/db;
• real signals and child shutdown in backoffice;
• loopback servers in API;
• spawned build/runtime-product validation.

One shared generic cache class would be unsound. The maximally cacheable design separates portable pure tests, capability-keyed subprocess/filesystem/container tests, and the smallest remaining live/fresh witness set. Internal and fork PR jobs must also split before adding OIDC and signed remote-cache authority; fork jobs must remain credential-free.

Local verification facts

There is no repository pre-commit or pre-push test hook.

The root bun run test is broad: it runs Effect contracts, root behavior contracts, the non-CLI workspace graph, and the canonical CLI verification runner (package.json:58). A shorter coverage-equivalent CLI command is:

CI=true HI_CACHE_NAMESPACE=development TZ=UTC bun run test:release

Exact PR reproduction also needs Node 24.19.0, Bun 1.3.5, the workflow service environment, committed base/head refs, and the development cache credentials. Affected selection cannot classify uncommitted changes because it compares commit objects. Before commit, a full root run is the faithful safe fallback (scripts/behavior-contract/run-test-scope.ts:20-33).

Protected attestation cannot run exactly in an ordinary local shell because it requires a clean committed tree, protected signing authority, Vercel/Turbo credentials, and GitHub OIDC (apps/cli/scripts/run-cli-verification.mjs:97-218).

Synthesized implications

These are evidence-backed architecture implications, not yet accepted product decisions.

1. No test discovery after merge. A required PR release-candidate gate can run the full prospective merge tree, including the release-shaped environment and complete package/baseline assembly. The post-merge job can then verify immutable evidence for the exact repository tree and perform only classification, reconciliation, and publication.
2. Defense in depth for release eligibility. Even with branch protection, automatic publication can refuse a main commit that has no associated green PR evidence. An audited emergency push can remain possible without automatically publishing it.
3. Upgrade or remain dormant. Private-repository branch gates cannot be enforced on the current organization plan. The accepted automatic release workflow must remain dormant unless the organization upgrades to GitHub Team/Enterprise, makes the repository public, or explicitly changes the earlier gate requirement.
4. Make the PR gate a full graph. Diff-aware tests remain useful for quick feedback, but release eligibility needs the complete graph so main does not expand coverage after merge.
5. Use one canonical developer command. A repository command should reproduce the complete PR candidate graph with the pinned runtime and canonical environment. A short affected pre-commit hook and a complete pre-push gate are separate product choices.
6. Cache by behavior, not current labels. Portable pure work should share local and trusted PR entries. Host-dependent work should use capability-keyed cache entries. Only external mutations and the smallest explicit fresh/concurrency witnesses should bypass cache.
7. Separate semantic failures from harness flakes. Bundled-source drift, missing baseline assets, stale wiki/skill contracts, and fixture digest mismatches must remain fail-closed. Repeated timeout, ETIMEDOUT, and commands.jsonl failures form a separate test-harness reliability problem.
8. Do not rerun the long graph inside npm publication. Release dispatch should consume exact successful tree-bound evidence, verify artifact/version/digests, authenticate, and publish idempotently. External authentication and network failure may still occur after merge; product test failure should not.

Conflicts and unresolved decisions

• The stated feature-branch-to-main workflow conflicts with live history: red PRs and seven direct pushes reached main because no branch protection is enforceable on the current plan.
• The stated platform-neutral deterministic class conflicts with current code: several files inside it branch on platform, and the task intentionally hashes OS/architecture.
• The existing policy requires full fresh protected native execution. Maximizing cache use suggests capability-cached native coverage plus a much smaller fresh sentinel. The acceptable fresh witness must be selected explicitly.
• A full pre-commit graph provides the earliest evidence but can take many minutes on a cold cache. The repository currently has no hook. The choice between advisory local command, affected pre-commit, and mandatory full pre-push remains open.
• Four main cancellations lack actor metadata. The evidence cannot determine whether a person or overlapping workflow caused each cancellation.
• No scheduled workflow execution exists in retained history despite the weekly trigger. The reason is not proven by retained Actions data.