interlab

What it does: A Go MCP server (mark3labs/mcp-go, stdio, auto-build launcher) implementing an autonomous benchmark-driven optimization loop with three tool families verified in code: (1) experiment — init_experiment writes a JSONL config header + creates an interlab/<name> git branch; run_experiment executes a bash -c benchmark, tees stdout to interlab.run.log, parses METRIC name=value lines via regex, and checks a 3-limit circuit breaker (max_experiments=50, max_crashes=3, max_no_improvement=10); log_experiment records keep/discard/crash, path-scoped git add of FilesInScope (never -A), commits with Metric-Name/Metric-Value trailers or reverts via git checkout --. All state is reconstructed from interlab.jsonl on every call (no server-side state machine, free crash recovery). (2) orchestration — plan/dispatch/status/synthesize_campaigns use beads (bd CLI) as the coordination layer for parent-epic/child-campaign DAGs, detecting file-scope conflicts at plan time. (3) mutation — a WAL-mode SQLite store (modernc.org/sqlite) recording each approach with task_type/hypothesis/quality_signal/inspired_by; mutation_record computes is_new_best inside a transaction (TOCTOU-safe via MaxOpenConns(1)); mutation_query returns prior approaches sorted by quality; mutation_genealogy traverses inspired_by chains into a tree (recursion correctly collects child IDs before closing rows to avoid the single-conn deadlock). The /autoresearch and /autoresearch-multi skills drive these tools. ic CLI and bd are best-effort/optional.

Rationale: This is genuinely mature, careful infrastructure: stateless file-based recovery, structural git-safety guarantees (path-scoped staging, branch isolation), graceful degradation when bd/ic are absent, and a correctly-engineered SQLite store (finite-value guard, transactional is_new_best, deadlock-aware genealogy recursion, indexed columns). The mutation provenance layer is ahead of the curve — inspired_by + is_new_best + cross-campaign task_type queries is a working instance of the 2026 ‘distill trajectories into a retrievable library of named principles/skills, then inject the relevant ones at inference’ self-improvement pattern (EvolveR, ExpeL, SAGE/RLEP skill libraries), and the /autoresearch skill already queries the store at campaign start to seed hypotheses. That is the right architecture. The verdict is ‘improve’ rather than ‘keep’ for one concrete, code-confirmed reliability gap (below) plus minor doc/version drift (main.go hard-codes server version ‘0.2.0’ while plugin.json is ‘0.4.8’; AGENTS.md and README still say ‘7 tools’/‘3 tools’ but the binary registers 10 — the 3 mutation tools are undocumented in the top-level docs). None of this undermines the core; it is a targeted hardening pass on an otherwise solid plugin.

SOTA gap: Single-run keep/discard is the frontier gap. run_experiment executes the benchmark exactly once and log_experiment makes the keep/revert decision on that one metric value (state.BestMetric is a scalar, no repeated-trial aggregation). The 2026 reliability-science findings (Beyond pass@1, arXiv:2603.29231; ReliabilityBench, arXiv:2601.06112) are explicit that a single success run is not evidence — you must measure variance across k trials and robustness to perturbation. interlab’s own skill even warns ‘avoid metrics that fluctuate >5%,’ acknowledging noise, but the engine offers no n-trials / median / confidence-interval / significance gate. A noisy benchmark can therefore record a spurious is_new_best and a kept commit on a one-off lucky run. The grounding-critique finding (‘generic NL critique is ineffective; structured tool-grounded feedback drives improvement’, arXiv:2510.24367) is satisfied (interlab IS executable-signal-grounded), but the variance dimension is the missing reliability primitive. A secondary, lower-priority gap: the mutation store is keyword/SQL-filtered only (task_type + min_quality), with no embedding/semantic retrieval over hypotheses — RAG-MCP/MemRouter-style semantic seeding would let a new campaign pull the most relevant prior hypothesis rather than just the highest-quality one in the same task_type bucket. qmd/intersearch in the ecosystem could supply this without a new embedding stack.

Redundancy: Low and mostly complementary. The beads-backed orchestration (plan/dispatch/status/synthesize_campaigns) overlaps conceptually with clavain’s campaign/sprint-dag DAG sequencer and dispatching-parallel-agents, but interlab’s DAG is scoped specifically to metric-optimization campaigns with file-conflict detection on files_in_scope — a narrower, justified specialization rather than duplication. The mutation provenance/genealogy store overlaps in spirit with interknow (durable pattern repo with provenance) and intermem (memory synthesis), and with interspect’s evidence/calibration history; these are different substrates (SQLite quality-signal genealogy vs. markdown knowledge vs. routing evidence) but a future consolidation of ‘provenance-tracked outcome stores’ across interlab/interknow/interspect is worth watching. interrank is named in the SOTA as pairing with interlab for benchmark-driven loops — complementary, not redundant. No tool here duplicates another plugin’s tool surface.

Actions:

• Add repeated-trial reliability to the experiment engine: a trials parameter on run_experiment (or init_experiment) that runs the benchmark N times, aggregates median + stddev/IQR, and surfaces variance; gate is_new_best/keep on a significance or min-improvement-over-noise threshold rather than a single sample — directly closes the Beyond-pass@1 / ReliabilityBench gap.
• Fix the documented drift: bump the hard-coded server version in cmd/interlab-mcp/main.go (‘0.2.0’) to match plugin.json (0.4.8) and ideally read it from the manifest; update AGENTS.md and README.md to document all 10 tools (the 3 mutation tools and the SQLite store are absent from both).
• Add semantic seeding to the mutation store: an optional embedding-backed query path (delegating to intersearch/qmd rather than a new embedding stack) so a new campaign retrieves the most relevant prior hypotheses across task_types, not just the top-quality rows in one bucket — realizes the RAG-MCP/MemRouter retrieval pattern the self-improvement loop depends on.
• Record per-trial variance into the mutation metadata and expose it in mutation_query/genealogy, so the compounding ‘what worked’ library distinguishes robust wins from lucky one-offs — turning provenance into reliability-aware provenance.