PR #6 Telemetry Review Findings

Initial Debt

Codex PR review found telemetry/reporting issues after M4 delivery:

• cli.artifact.pull.completed emitted only for control-plane baseline materialization, so the default GitHub stable-baseline fallback path undercounted successful artifact pulls.
• cached stable-baseline reuse also emitted cli.artifact.pull.completed, so repeated scaffold/update runs against the same cached artifact could overcount adoption.
• punks report awaited best-effort telemetry after the report had already been submitted, so a stalled telemetry call could make a successful report command appear stuck.
• telemetry/report timestamps were modeled as plain strings, so invalid datetime values could pass contract validation and fail later or persist inconsistently.
• loopback control-plane URLs such as 127.0.0.1 and ::1 were classified as normal telemetry instead of local-dev, so local testing could be counted as adoption traffic.
• API telemetry persistence was gated on full Better Auth configuration, so CLI-token deployments with DATABASE_URL still used in-memory telemetry and lost adoption data across restarts.
• database telemetry dedupe used a pre-read before insert, so concurrent retries with the same eventId could race on the unique index and reject instead of returning accepted/not-counted.
• @punks/db still imported the full server env contract, so CLI-token-only database persistence could fail when Better Auth variables were intentionally absent.
• artifact-pull telemetry did not receive the target repository cwd, so scaffold/update runs using --input could attribute pulls to the launcher directory.
• telemetry/report datetime validation accepted impossible calendar dates because JavaScript date parsing normalizes values like February 31 instead of rejecting them.
• 0.0.0.0 control-plane URLs were classified as normal, so common local API configurations could be counted as adoption traffic.
• telemetry without explicit project usage or repository remote metadata derived ProjectUsage from per-event ids, fragmenting adoption into one project per event.
• reports without explicit project usage or repository remote metadata derived ProjectUsage from titles, merging unrelated reports with similar names.
• CLI telemetry still probed git repository metadata before discovering telemetry transport was disabled, adding avoidable latency in large repositories.
• telemetry database outages were mapped to 400-level TelemetryEventRejected, misclassifying server persistence failures as client input errors.
• report database outages were mapped to 400-level HarnessReportRejected, misclassifying server persistence failures as client input errors.
• CLI telemetry read plain git config for user.name and user.email, allowing global git identity to leak into telemetry outside safe repo-local context.

Resolution

• Centralized artifact-pull telemetry and emitted it for successful remote baseline materialization from both control-plane and GitHub stable-release paths.
• Stopped emitting artifact-pull telemetry when reusing cached control-plane or GitHub stable baselines.
• Restored punks report telemetry to best-effort fire-and-forget after successful report submission.
• Added IsoDateTimeString contract validation for telemetry occurredAt and report createdAt.
• Classified localhost, .localhost, IPv4 loopback, and IPv6 loopback control-plane URLs as local-dev.
• Decoupled API database persistence selection from Better Auth environment readiness; DATABASE_URL plus non-memory storage now selects durable telemetry/report persistence while Better Auth remains an auth-provider gate.
• Made database telemetry insertion idempotent through insert conflict handling, returning accepted/not-counted when an event id already exists.
• Removed the hidden Better Auth env dependency from @punks/db bootstrap; database creation now requires only DATABASE_URL.
• Threaded target repository cwd through stable baseline resolution so artifact-pull telemetry uses the scaffold/update target repository.
• Tightened ISO datetime validation to reject impossible calendar dates before telemetry/report persistence.
• Classified 0.0.0.0 control-plane URLs as local-dev.
• Stopped manufacturing ProjectUsage records when telemetry or reports lack both explicit projectUsageId and repository remote identity; events/reports remain persisted with nullable project linkage instead.
• Skipped CLI event construction and repository probing when telemetry transport is disabled.
• Added TelemetryPersistenceUnavailable as a 503 contract error for telemetry persistence outages while keeping TelemetryEventRejected for 400-level input rejection.
• Added HarnessReportPersistenceUnavailable as a 503 contract error for report persistence outages while keeping HarnessReportRejected for 400-level input rejection.
• Restricted telemetry git identity collection to repo-local git config.
• Added regression tests for GitHub fallback artifact telemetry, cached baseline reuse, target-repo artifact telemetry attribution, disabled-transport telemetry short-circuiting, repo-local git identity boundaries, loopback telemetry source classification, 0.0.0.0 local-dev classification, database persistence selection, database retry idempotency, remote-less project linkage, telemetry/report persistence outage mapping, database bootstrap env independence, and invalid telemetry/report timestamps.

Validation

• bun run test --filter=@punks/contract --filter=@punks/cli: passed.
• bun run check-types --filter=@punks/contract --filter=@punks/cli: passed.
• bun run test --filter=@punks/cli: passed after follow-up review fixes.
• git diff --check: passed after follow-up review fixes.
• bun run check: passed after follow-up review fixes.
• bun run check-types: passed after follow-up review fixes.
• bun run test --filter=@punks/api: passed after API persistence follow-up review fixes.
• bun run check-types --filter=@punks/api: passed after API persistence follow-up review fixes.
• bun run check-types: passed after API persistence follow-up review fixes.
• bun run test --filter=@punks/cli: passed after target-repository artifact telemetry follow-up review fixes.
• bun run test --filter=@punks/db: passed after database env-independence follow-up review fixes.
• bun run test --filter=@punks/api: passed after database env-independence follow-up review fixes.
• bun run check-types --filter=@punks/db --filter=@punks/api --filter=@punks/cli: passed after latest follow-up review fixes.
• bun run check: passed after latest follow-up review fixes.
• bun run check-types: passed after latest follow-up review fixes.
• bun run test --filter=@punks/contract: passed after datetime validation follow-up review fixes.
• bun run test --filter=@punks/cli: passed after 0.0.0.0 local-dev follow-up review fixes.
• bun run check-types --filter=@punks/contract --filter=@punks/cli: passed after latest follow-up review fixes.
• git diff --check: passed after latest follow-up review fixes.
• bun run check: passed after latest follow-up review fixes.
• bun run check-types: passed after latest follow-up review fixes.
• bun run test --filter=@punks/api: passed after remote-less project linkage follow-up review fixes.
• bun run check-types --filter=@punks/api: passed after remote-less project linkage follow-up review fixes.
• git diff --check: passed after remote-less project linkage follow-up review fixes.
• bun run check: passed after remote-less project linkage follow-up review fixes.
• bun run check-types: passed after remote-less project linkage follow-up review fixes.
• bun run test --filter=@punks/contract: passed after disabled-transport/persistence-outage follow-up review fixes.
• bun run test --filter=@punks/api: passed after disabled-transport/persistence-outage follow-up review fixes.
• bun run test --filter=@punks/cli: passed after disabled-transport/persistence-outage follow-up review fixes.
• bun run check-types --filter=@punks/contract --filter=@punks/api --filter=@punks/cli: passed after disabled-transport/persistence-outage follow-up review fixes.
• git diff --check: passed after disabled-transport/persistence-outage follow-up review fixes.
• bun run check: passed after disabled-transport/persistence-outage follow-up review fixes.
• bun run check-types: passed after disabled-transport/persistence-outage follow-up review fixes.
• bun run test --filter=@punks/contract: passed after report-persistence/git-identity follow-up review fixes.
• bun run test --filter=@punks/api: passed after report-persistence/git-identity follow-up review fixes.
• bun run test --filter=@punks/cli: passed after report-persistence/git-identity follow-up review fixes.
• bun run check-types --filter=@punks/contract --filter=@punks/api --filter=@punks/cli: passed after report-persistence/git-identity follow-up review fixes.
• git diff --check: passed after report-persistence/git-identity follow-up review fixes.
• bun run check: passed after report-persistence/git-identity follow-up review fixes.
• bun run check-types: passed after report-persistence/git-identity follow-up review fixes.

Implementation Notes

• apps/wiki/specs/cli/IP-105-runtime-telemetry/IMPLEMENTATION-NOTES.md
• apps/wiki/specs/cli/IP-106-cli-harness-reporting/IMPLEMENTATION-NOTES.md