Verifying pi extensions
The runnable recipe — target, launch, doctor, feature map, cleanup — is
.pi/verify/SKILL.md. This file explains what each
check covers and where its evidence stops.
Run npm run check for repository conventions, types, lint, formatting, unused
code and tests. npm run verify loads every root-manifest extension through a
registration stub and checks the local skill library. The stub captures
registrations; it does not dispatch events or execute tools. Missing TypeSafe
credentials are informational offline. npm run verify -- --live explicitly
opts into an external TypeSafe request using your configured credentials.
Run npm run smoke with pi 1.4.0 or later on PATH (or set
PI_SMOKE_BIN to its executable path). This uses the actual pi RPC host, loads
all manifest extensions, the owned workflow skills and prompt templates. It
checks discovery and expansion of those resources from an unrelated working
directory using the local provider, including /workflow. It checks /skill-search
and /usage --json in RPC without provider calls, then enters plan mode, checks that
plan_mode_question, plan_mode_complete, and plan_fetch_url are offered to
the model, attempts a harmless touch smoke-blocked command through real tool
dispatch, verifies it was blocked and that the file does not exist, exits plan
mode, checks those three tools are no longer offered, explicitly enables
subagents,
and completes a scout child through the actual child-session implementation.
It then has the fixture model write a file and stop, and checks that done-gate
buys exactly one extra turn with its "Done check" message and keeps the reminder
out of the transcript (display: false); checks that a write outside the
workspace buys no turn; and runs /until test -f smoke-done, checking the agent
is sent back until the file exists.
The isolated host also loads built-in codemode explicitly. It runs controlled
measurement trials and rejects empty work, then fails the same check after two
distinct edits. It checks that native writes, shell writes and both nested
variants are blocked with files absent, executes a real diagnostic observation,
and checks that a subsequent write succeeds.
It starts a real failed background job through deferred terminal_* tools,
checks the actual exit status/output, proves one completion follow-up resumes the
agent without steering, and drives an optional Jev attention watch
against the redacted local fixture. The watch is inference, not verification;
notify: false must not start another model turn.
It also runs deferred find_code from codemode against a local TypeSafe
fixture, checks URL-userinfo redaction at that HTTP boundary, and uses a
structured MCP-shaped tool fixture (not an actual MCP transport) to check
warning propagation to the outer codemode result without changing its schema.
It fails on missing capabilities, failed assertions, or a 30-second RPC timeout.
The smoke creates a temporary working directory and agent configuration, uses
an allowlisted environment, disables discovery/MCP/startup networking, and
serves scripted OpenAI-compatible responses on loopback only. It needs no
provider credentials and incurs no external model cost. Child sessions use the
same temporary agent directory. TypeSafe requests point at the local fixture:
semantic-search and injection-screen phases receive scripted typed answers;
other phases exercise the existing unavailable-judge fallback. An explicit base
URL overrides the classifier's catalog host, so a registered TypeSafe model
still does not call api.typesafe.ai during smoke. scripts/ classifiers have
no Pi registry and keep the direct HTTP transport against that same base URL.
Normal user settings and sessions are not changed. The host, server and fixtures
are cleaned up in finally on normal success or failure. Forced process
termination can leave a pi-smoke-* directory in the system temporary directory.
npm run coverage runs the unit suite, offline registration verifier and real-Pi
smoke under Node's built-in LCOV reporter, so it requires pi on PATH too. It
works on Node 24+ without --test-coverage-include-all (which is Node 26-only). Generated coverage
is ignored by Git. Run from a clean state on the CI runtime too; a local green
result is not evidence of hosted CI success.
This smoke checks host integration and resource/extension wiring, not whether
a model follows a skill well, model judgment,
interactive rendering, external providers, or successful TypeSafe judgments.
For an explicitly authorized live/UI check, start pi in a disposable directory
with a separate PI_CODING_AGENT_DIR, provision only the intended provider
credential there, and repeat /plan start, the harmless blocked bash command,
/plan exit, /subagents on, and a child request. Record the actual tool result
and child completion. Do not count a provider response or the registration stub
as evidence that the blocked action or child execution worked.
For the local discovery/accounting commands alone, run
node .pi/verify/local-commands.mjs. It loads all extensions in an isolated real
Pi RPC host and verifies skill-search results, empty-branch JSON accounting and
/workflow discovery without a listener or model calls. It can run where loopback
listeners are prohibited. It does not cover prompt expansion, numerical model
usage (covered by unit tests), or interactive skill-picker rendering.
Project recipe
Target: record the requested base/head and git rev-parse HEAD before running.
For uncommitted work include staged, unstaged and untracked content identity;
subagent results provide assessment.revision.fingerprint when Git is available.
Any intervening edit invalidates earlier evidence for the final state.
| Step | Command / fixture | Expected observation |
|---|---|---|
| Setup | npm ci only when dependency installation is authorized; otherwise use the existing installation |
Node, dev tools and pi are available; report missing setup |
| Static and behavior checks | npm run check |
Exit 0; all repository checks and tests pass |
| Registration | npm run verify |
Every manifest entry registers; each owned workflow ranks for its sample task; no live TypeSafe call |
| Workflow resources | npm run smoke |
Three owned skills and review/test/design/refactor prompts discover and expand in a fresh host outside this repo; no imported personal skill collection |
| Coverage and change risk | npm run coverage, then crap packages/skill-select scripts --diff --coverage coverage/lcov.info and deadwood . --fail-on high |
Nonempty LCOV matches changed JS modules; inspect changed-function coverage, CRAP and unused-code findings. Report tool limitations rather than scoring unmatched files as covered |
| Host startup and fixture | npm run smoke starts pi RPC and its disposable local provider |
All manifest extensions load in the actual host |
| Failed-fix and measurement contracts | npm run smoke plus node --test packages/diagnosis-nudge/test/checkpoint.test.mjs packages/done-gate/test/measurement.test.mjs |
Native/shell/nested mutations pause after two failed revised checks; executed checkpoint recovers; invalid evidence cannot unlock or generate a measurement report |
| Changed behavior | Smoke attempts touch smoke-blocked during /plan start, then exits and runs a child |
The three plan tools are offered inside plan mode and gone after exit; the tool is blocked, file absent, real child completes; done-gate continues once, quietly, and not for an outside-cwd write; /until loops to its predicate |
| Jev question sets | Part of npm run check: node --test scripts/jev-questions.test.mjs |
Every call site's questions pass the production validator against the state its caller builds; a case that silently asks nothing is itself a failure |
| Jev decision ledger | Part of npm run check: packages/typesafe/test/ledger.test.mjs and scripts/judge-thresholds.test.mjs |
A call appends a local jev-decision entry and does not store state, question text, or the API key; a drifted reported model warns once; guardrail entries export to the replay-corpus shape without the command |
| Judgment quality, live | node scripts/judge-live.mjs |
Exit 0 when every labelled case reached the judge; the labels are printed for a human to read, because a model's reading is a hypothesis and the wiring is what the unit tests prove |
| Judgment paths, live | node scripts/judges-live.mjs (or --only <name>) |
Exit 0 when every judgment path reached its judge: review lint, findings verification, web_answer, plan review, CI triage, correction promotion. Each line prints what came back; the readings are a model on a fixture, not ground truth. Costs one real request per case, so it is not part of npm run check |
| Workbench/footer, real terminal | node .pi/verify/workbench.mjs --upgrade and node .pi/verify/workbench.mjs --regular |
Exit 0 in both modes: isolated tmux terminal with real pi and Neovim, local scripted provider, footer upgrade/reload, live filter/select/cancel, recoverable drafts across reload, file references, non-editable staged/unstaged diffs, next/previous navigation with retained query/selection/scroll, read-only Neovim handoff, /eixt interception without execution or model calls, Unicode/narrow resize, and actual working → ready state. No personal config or credentials; fixture/server/socket removed in finally. Does not prove every terminal forwards Ctrl+Alt identically or exercise Matt's Neovim plugins. The target is the current working tree and printed HEAD, not a hosted revision. |
| Background jobs, real terminal | node .pi/verify/jobs.mjs |
Real Pi live logs, follow/scroll/resize/cancel, same-PTY input and detach/reattach, Ctrl+C to the program, explicit human line input, real failed exit status, and owned-process cleanup across reload/session/shutdown. Isolated tmux is only the test terminal, not an extension dependency; no personal config or model requests. |
| Cleanup | Smoke's finally closes processes/server and removes temporary files |
No user config/session edits; report cleanup failure |
The current deadwood build does not scan this repository's .mjs modules
and reports unresolved sibling imports plus the intentionally host-provided Pi
SDK as missing dependencies. Its failure is not a useful dead-code verdict here;
record that limitation and use the repository's Knip/Fallow gates rather than
adding runtime dependencies or suppressions to satisfy it.
Choose additional acceptance-specific checks for changes outside that smoke's coverage. A green registration stub alone is insufficient. For UI/native work, use an explicitly configured project verifier with those tools or report the capability gap. No universal extra tool grant is implied by this recipe.