← roadmap

intertrack

keep priority P2 effort S wave 2

What it does: A small, hooks-and-skills-only plugin (no MCP server, same pattern as interstat) that tracks feature-level success metrics for the Demarch project. It stores YAML-defined metric definitions (config/metrics.yaml: 12 metrics across 4 features F1-F4, each tied to a producing plugin: interknow/interflux/intersynth/interspect) in a WAL-mode SQLite DB at ~/.claude/intertrack/metrics.db. Public API is two bash scripts: track-record.sh (write an observation: value + session/bead/context, always exits 0 to never break callers) and track-query.sh (read modes: status / metric / feature / baseline-snapshot, all JSON). A SessionStart hook auto-inits the DB and emits a one-line metric/observation count as additionalContext. Two skills (/intertrack:metrics dashboard, /intertrack:feature-report) format the JSON into tables. Philosophy is Demarch’s “receipts over narratives”: it measures only — routing/policy is left to interspect/clavain.

Rationale: This is a deliberately minimal, mature, single-purpose measurement substrate and it does that job cleanly: tight schema, indexed columns, graceful-degradation write API, structural pytest suite, standard plugin scaffolding, and a clear scope boundary vs interstat (cost) / interspect (trust) / interpulse (pressure). It is not LLM-shaped — there is no model call, no agent loop, no tool catalog — so most of the 2025-2026 agent SOTA (orchestration, context engineering, RAG-MCP, progressive disclosure, memory tiers) simply does not apply to its core function, and bolting any of it on would be over-engineering. The honest verdict for a correct, small, scoped plugin is keep. However two real defects keep it from being clean: (1) ADOPTION GAP — a grep across the entire interverse for track-record.sh / intertrack callers returns ZERO; none of the four “instrumented plugins” the README claims “record observations automatically” actually emit anything, so the DB is an empty harness measuring a single historical PRD. (2) DOC BUG — README.md (Hooks + Data Flow) advertises a PostToolUse hook that records observations, but hooks.json defines only SessionStart (the vision doc correctly says SessionStart-only). These are fixable in-place, not redesign signals.

SOTA gap: No frontier-architecture gap that matters for its purpose — this is a metrics-store, not an agent. The one genuinely relevant SOTA touchpoint is the reliability-science evaluation trend (Beyond pass@1 / ReliabilityBench, arXiv:2603.29231 & 2601.06112): a feature-effectiveness tracker that reduces everything to a single AVG(value) and a binary GREEN/YELLOW/RED-vs-target is exactly the “one success run is not evidence” anti-pattern the field is moving away from. intertrack stores per-observation rows (good raw material) but its query/skill layer computes only mean + latest; it has no variance/spread, no k-trial consistency, no confidence interval, and no significance gate on “did this feature actually move the metric.” Adding variance/CI and a min-sample threshold before declaring GREEN would align it with reliability-science eval at very low cost. Secondary, lower-priority: track-record.sh and track-query.sh build SQL by raw string interpolation ($metric_name, $value, $name, $feature) — a classic injection vector. Trust model is internal (callers are sibling plugins, not untrusted repo content), so severity is low, but the 2026 agent-supply-chain/tool-poisoning trend argues for parameterized inserts on principle.

Redundancy: Low and intentional. It explicitly partitions the observability space with sibling plugins and even cross-references them in skill descriptions: interstat = token cost/efficiency, interpulse = context pressure, interspect = agent trust/routing, intercheck = code quality, intertrack = feature-outcome effectiveness. No other interverse plugin tracks “did a shipped feature hit its declared success metric.” The one adjacency worth watching: interlab (autonomous experiment/optimization loop with metric logging) and intermix (cross-repo matrix eval) both record metrics over time — intertrack’s metric_observations table overlaps conceptually with interlab’s log_experiment. They are not duplicative today (interlab is optimization-loop scoped, intertrack is PRD-feature scoped), but if a unified ecosystem metrics store ever emerges, these three are the merge candidates.

Actions:

  • Fix the README doc bug: remove or correct the PostToolUse hook claim in the Hooks and Data Flow sections so it matches hooks.json (SessionStart-only), as the vision doc already states.
  • Close the adoption gap: either wire at least one real track-record.sh call into the four named producer plugins (interknow/interflux/intersynth/interspect) so the harness has live data, OR file a beads issue scoping that instrumentation and mark the plugin’s current metrics as illustrative — do not ship an empty tracker advertised as auto-instrumented.
  • Add reliability-science statistics to track-query.sh and the skills: emit variance/stddev or a confidence interval and an observation-count gate, and only declare GREEN once a minimum sample size is met (cite Beyond pass@1, arXiv:2603.29231) — turns ‘meets target on average’ into ‘reliably meets target’.
  • Parameterize the SQL in track-record.sh and track-query.sh (bind values instead of interpolating $metric_name/$value/$name/$feature) to remove the injection vector while keeping the always-exit-0 contract.