interhelm

keep priority P2 effort M wave 2

What it does: A pattern-teaching plugin (no MCP server, no runtime code shipped) for “agent-as-operator” runtime diagnostics. Per its files it provides: 4 skills (runtime-diagnostics scaffolds a /diag/* HTTP server with Health/Diff/Assert/Smoke-Test/UI-state/Schema endpoints plus /control/* mutations; smoke-test-design teaches the smoke test as an executable contract; cuj-verification does screenshot-free CUJ checks via /diag/ui/state state-before→action→state-after; diagnostic-maturation walks a 6-level ladder from domain-health to REPL/watch and supports an audit mode), 1 sonnet review agent (runtime-reviewer: pattern completeness + security + perf, P0/P1/P2 findings), 3 advisory PostToolUse hooks (nudge away from screenshots on native apps, auto-health-check after Rust edits, CUJ reminder on git commit), and 2 reference templates (rust-hyper server, CLI client). Core thesis: agents query structured JSON endpoints instead of OCR-ing screenshots, citing ~7.5x token savings. Validated against a real reference (Shadow Work sw-agent).

Rationale: The core thesis sits squarely on the right side of 2026 SOTA. The “verification is the new bottleneck” finding and the empirical result that structured, tool-grounded feedback (tests, executors, runtime signals) drives self-correction while generic NL/LLM-judge prose does not (“From Code to Courtroom”) directly validate interhelm’s bet: replace visual inspection with parseable runtime state. Token-economics findings (“the token bill comes due”, context-rot from verbose tool output) make the screenshot→JSON savings genuinely load-bearing rather than cosmetic. The plugin is mature: tight scope, framework-agnostic, grounded in a production reference, clean separation of read (/diag) vs mutate (/control), advisory-only hooks (correct for blast-radius), and a security/perf review agent that already encodes the right P0s (debug-gated endpoints, localhost-only bind, no lock on main thread, no secrets in dumps). This is the kind of well-formed, single-purpose plugin the prompt says should default to keep. The gaps are enhancements, not structural problems.

SOTA gap: Two frontier ideas it lacks. (1) Code Mode / code-execution-with-MCP framing (Cloudflare Code Mode ~99.9% input-token reduction; Anthropic code-execution-with-MCP ~98.7%): the diagnostic surface is currently driven by hand-written curl/CLI calls in the skills; SOTA says agents are far more reliable and token-efficient writing code against a discoverable API than issuing individual tool calls. The /diag/schema endpoint is the natural hook for presenting the diag API as a filesystem-discoverable/code-callable surface with progressive disclosure (RAG-MCP), which interhelm does not yet articulate. (2) Reliability-science evaluation (Beyond pass@1, ReliabilityBench): smoke-test-design treats a single pass/fail run as truth (“Deterministic — same state produces same results”) but does not advocate k-trial variance, perturbation robustness, or graceful-degradation-under-fault measurement — which is exactly where runtime verification of a stochastic agent-built app should be heading. Minor: the runtime-reviewer agent hardcodes model: sonnet rather than deferring to the ecosystem’s interspect/interrank routing.

Redundancy: Adjacent but not duplicative. clavain /smoke-test (smoke tests against a running app, walks critical user journeys) and the verification-before-completion / verify skills overlap conceptually with smoke-test-design and cuj-verification — but clavain’s are generic (dev server + journey walk), while interhelm’s niche is teaching an app to EXPOSE a structured diagnostic HTTP/CLI surface in the first place; that scaffolding niche is unique. tuivision (Playwright-for-terminals, visual TUI testing) is the visual counterpart interhelm explicitly defers to (its browser-on-native hook even matches tuivision’s get_screenshot tool); complementary, not redundant. intercheck (code-quality guards + session health) and interflux/quality-gates touch verification generally but not runtime diagnostic-server design. No merge warranted; worth a cross-reference so clavain /smoke-test can detect and prefer an interhelm /diag/smoke-test endpoint when present.

Actions:

Add a ‘diagnostic-as-code-surface’ section to runtime-diagnostics (or a new skill): teach exposing /diag/schema as a discoverable, code-callable API and have agents write a short script against it rather than hand-curling each endpoint — citing Code Mode / code-execution-with-MCP token wins and RAG-MCP progressive disclosure for large endpoint sets.
Modernize smoke-test-design with a reliability-science tier: optional k-trial repeat (variance/flake detection), perturbation robustness, and fault-injection (kill a subsystem, expect graceful ‘degraded’) — moving from ‘did it pass once’ to ‘how reliably does it pass’, per Beyond pass@1 / ReliabilityBench.
Stop hardcoding model: sonnet in runtime-reviewer; route via the ecosystem’s interspect/interrank economy/quality toggle so the security/perf reviewer scales model choice to risk.
Add an explicit interop note so clavain /smoke-test and verification-before-completion detect a project’s /diag/smoke-test endpoint (via the CLAUDE.md Diagnostic Server convention) and prefer it over generic journey-walking — closes the adjacency gap without merging.