← roadmap

intermix

keep priority P2 effort M wave 2

What it does: Go MCP server (mark3labs/mcp-go, mirrors interlab) exposing 6 tools — init_matrix, run_cell, classify_result, report_matrix, run_batch, poll_batch — that run the Skaffen coding agent against a (repo × task) matrix of unfamiliar codebases. Per cell it clones (bare-repo cache), runs language setup (extensive uv/pyenv Python version handling), spawns skaffen -mode print (optionally -iterate N -test-cmd), applies SWE-bench test_patch, runs targeted-then-full validation (pytest/Django runtests.py/go test/npm), and classifies via a fixed-signal taxonomy (timeout/crash/context_limit/tool_failure/no_progress/success/partial) plus optional free-text LLM analysis. State is JSONL-reconstructed (crash recovery), circuit breakers cap 100 cells / 5 consecutive failures, parallel mode fans cells into intermux-discoverable tmux sessions, harvests Skaffen evidence JSONL, and auto-files per-cell debug beads + failure-cluster pattern beads. Reports include pass-rate, ASCII heatmap, outcome distribution, token accounting (in/out/cache), and segment delta (fixed/regressed/stable). Real data: SWE-bench Lite 38%→50%→78% across 50-instance runs.

Rationale: This is a mature, genuinely working eval harness that has produced real, hard-won benchmark progressions (38%→78% SWE-bench Lite) documented in committed campaign data — not a stub. The architecture is sound and already aligns with several SOTA patterns the roster calls out: stateless JSONL reconstruction (crash recovery / replay-friendly, the LangGraph lesson), git-worktree-equivalent tmpdir isolation per cell, tmux-based parallel fan-out with isolated execution, token accounting including cache, circuit breakers, and bead-integrated regression tracking. Code quality is high (UTF-8-safe truncation, atomic temp-file+rename for batch state, shell-escaped prompt injection avoidance, 11 test files across the package). The single real frontier gap — reliability-science k-trial measurement — is an additive enhancement to an otherwise-correct design, so it does not warrant a modernize verdict. Last commit 2026-03-26 reflects active benchmark iteration, not neglect. Keep, with the k-trial enhancement filed as the highest-value follow-up.

SOTA gap: Single-run-per-cell evaluation. No k-trial repetition, variance, or seed control anywhere in the code (grep for trial/variance/repeat/seed/pass@ is empty). The 2026 “Beyond pass@1: Reliability Science” trend (arXiv:2603.29231, ReliabilityBench arXiv:2601.06112) argues that for a stochastic agent a single pass conflates capability with luck — exactly intermix’s measurement model. The delta report’s fixed/regressed counts inherit this: a “regression” may just be run-to-run variance. Secondary: classify_result’s free-text LLM analysis lacks the Agent-as-a-Judge bias guards (distinct judge, masked author, reward-hacking watch); and validation runs untrusted repo commands via bash -c with only tmpdir/tmux isolation, no sandbox (security trend’s controlled-blast-radius framing).

Redundancy: Low and intentionally complementary, not overlapping. Integrates with interlab (campaigns/optimization loops sit above it), intermux (tmux session discovery via /tmp mapping files + sanitized session names), and beads (debug/pattern bead creation). interlab does mutation-provenance experiment loops; intermix is the eval execution substrate beneath it. interrank ranks published model/benchmark scores; intermix generates first-party benchmark data. No merge candidate — distinct layer in the eval stack.

Actions:

  • Add k-trial support: run_cell/run_batch gain an optional trials arg (default 1); report_matrix surfaces pass@k, per-cell pass rate, and variance, so the existing delta report can distinguish true regressions from run-to-run noise (drives the only modernize-worthy gap — arXiv:2603.29231 reliability science).
  • Harden classify_result against Agent-as-a-Judge bias: keep the deterministic signal classifier as ground truth (already good), but document that the free-text llm_analysis is advisory only and should be produced by a model distinct from the agent under test.
  • Reduce validation blast radius: route the bash -c validation/setup step through the existing Docker path by default for untrusted repos (the RunCellDocker plumbing already exists), making sandboxed execution the default rather than an opt-in flag.
  • Promote it for cross-ecosystem use: intermix is the only interverse plugin producing first-party agent benchmark data — wire its report JSON into interrank/interspect so routing calibration can consume real harness outcomes, not just published leaderboards.