← roadmap

interstat

keep priority P2 effort M wave 2

What it does: Token-efficiency benchmarking + cost analytics for agent workflows. Two-phase collection: a PostToolUse:Task hook (post-task.sh) captures subagent dispatch events into SQLite in real time, then SessionEnd + analyze.py backfills actual token counts (input/output/cache_read/cache_create) from JSONL transcripts, resolving subagent hash IDs to semantic names via .meta.json. A second PostToolUse:* hook logs every tool call into tool_selection_events (seq, preceding-tool, retry, outcome) for failure classification. report.sh emits subagent-type breakdown, main-vs-subagent split, top consumers, p50/p90/p95 percentiles, and a hardcoded decision gate (120K effective-context p95 -> BUILD/SKIP hierarchical dispatch). cost.py computes API-equivalent cost + subscription leverage from a hardcoded PRICING table. cost-query.sh is a declared ~17-mode JSON interface for L1 Intercore (ic cost baseline) and L2 Galiana, with bead/phase correlation, cost-per-landable-change baseline, shadow-routing ROI, and agent effectiveness ranking. Emits budget-tier alerts to interband. Session search was deliberately migrated out to intersearch; interstat now retains bead-correlated token metrics only.

Rationale: This is a mature, well-scoped, well-tested plugin (bats + structural pytest, failed-insert replay, WAL/busy_timeout for concurrent hook writes, fail-open hooks, idempotent upsert with multi-strategy matching). It directly serves the #1 enterprise pain point in the 2026 SOTA findings — token economics (“the token bill comes due”, per-dev consumption up ~18.6x, context-rot snowballing) — and does so as deterministic hook-driven instrumentation, exactly the “harness engineering” discipline the findings endorse. Scope is already disciplined: session search was correctly handed to intersearch, leaving interstat as the cost/token-metrics specialist. The architecture (hooks -> SQLite -> queryable JSON interface consumed by other layers) is sound and not duplicated wholesale elsewhere. The defects are bounded and fixable in place — none warrant modernize/deprecate. The two real gaps are a hardcoded pricing table that drifts (Opus 4.7 default, while the ecosystem now references Opus 4.8/Fable) and a single-threshold dispatch verdict that predates current orchestration economics — both are targeted improvements, not a rewrite.

SOTA gap: Two frontier ideas it lacks. (1) Pricing/model data is hardcoded and goes stale — the SOTA findings establish interrank (cost_leaderboard, get_model, refresh_snapshot) as the canonical, refreshable model/cost authority; interstat should source pricing from interrank instead of pinning a PRICING dict that already lags current Opus/Fable rates. (2) The decision gate (“BUILD vs SKIP hierarchical dispatch” on a single 120K effective-context p95) predates the consolidated 2025-2026 orchestration economics: orchestrator-worker carries ~15x token cost and is justified only for provably-parallelizable, breadth-first work, with context-isolation-inside-a-single-agent as the default (HALO arXiv:2505.13516; Anthropic context-engineering write/select/compress/isolate with +29-39% lifts). interstat measures the right primitive but frames the verdict on a dated heuristic. It also does not yet feed its rich per-agent cost-effectiveness data into the embedding/evidence-based routing loop (interspect) as a closed-loop cost signal — a missed self-improvement opportunity, though arguably out of scope.

Redundancy: Pricing table in cost.py overlaps directly with interrank, which is the canonical model/cost-leaderboard MCP (cost_leaderboard, get_model) — interstat maintains a parallel, drifting copy of pricing. The cost-effectiveness/agent-ranking mode (effectiveness) overlaps conceptually with interspect’s routing/cost calibration; AGENTS.md already notes the intended division of labor (interstat = raw cost, interspect = quality-weighted), so this is coordinated rather than accidental. Session analytics overlap with intersearch was already resolved by an explicit migration. tool-time also captures tool-usage-via-hooks analytics, partially overlapping the post-tool-all.sh / tool_selection_events path — worth confirming they are not double-instrumenting the same PostToolUse events.

Actions:

  • Source pricing dynamically from interrank (cost_leaderboard / get_model) with the hardcoded PRICING dict as a fallback, so API-equivalent cost stops drifting; at minimum update the default pin from claude-opus-4-7 to the current default and add Opus 4.8 / Fable entries.
  • Reframe the report.sh decision gate to current orchestration economics: report measured fan-out cost multiplier and parallelizability rather than a single 120K BUILD/SKIP verdict, citing the ~15x orchestrator-worker cost / context-isolation-default consensus.
  • Verify no double-instrumentation between interstat post-tool-all.sh (tool_selection_events) and tool-time; if overlapping, declare one canonical capture path in AGENTS.md and document the division with interspect (raw cost vs quality-weighted routing).
  • Close the loop: expose interstat effectiveness/cost-per-bead data to interspect as a cost signal for routing calibration, turning passive measurement into a self-improvement input.