← roadmap

interrank

improve priority P2 effort M wave 2

What it does: A pure read-only MCP server (no skills/commands/hooks) that loads a published AgMoDB JSON/JSON.gz snapshot (GitHub release asset, URL, or local file), caches it in memory with interval refresh + in-flight coalescing, and serves 15 ranking/lookup tools: snapshot_info, refresh_snapshot, list_models, get_model, list_benchmarks, leaderboard, benchmark_leaderboard, recommend_benchmarks, recommend_model, compare_models, domain_leaderboard, cost_leaderboard, and the routing bridge (resolve_routing_name, routing_compare, list_model_families). Benchmark records carry contamination/freshness/metadata-status/caveats fields, and recommendModels chains task->benchmark relevance->weighted model score with a 0.7x discount for BenchPress-predicted cells and a min-coverage gate.

Rationale: The core thesis is exactly on-frontier: harness-agnostic ranking/routing across many models, with benchmark metadata that already encodes contamination risk, freshness type, and prediction flags — a direct answer to the SOTA findings that SWE-bench is saturating and headline benchmark scores have lost signal, and that the value is in “tracking which benchmark predicts real-world performance and routing accordingly.” The engineering is mature (snapshot coalescing, clean stdio/stderr separation, background prefetch, graceful missing-data handling). This is not a deprecate/modernize candidate — the architecture is sound. But it falls short of “keep” on two concrete, code-visible gaps: (1) the task->benchmark->model recommendation engine routes purely on string.includes() keyword tokenization against a 15-entry hand-authored TASK_DOMAIN_MAP, when the SOTA-validated pattern (Select-then-Solve embedding routers, RAG-MCP) is a sub-millisecond embedding gate — and the ecosystem already ships intersearch embedding infra that interrank ignores; and (2) docs are stale (README/CLAUDE.md/AGENTS.md list 7 tools, code registers 15) and version drift exists (plugin.json 0.3.1 vs McpServer name 0.2.0). These are real, bounded signals that warrant improve, not keep.

SOTA gap: Embedding-based semantic gating for task->benchmark and task->model recommendation. recommend_benchmarks/recommend_model use keyword token overlap + a hardcoded TASK_DOMAIN_MAP; SOTA (Select-then-Solve / paradigm-routing via all-MiniLM-style encoders, RAG-MCP retrieval-augmented selection >3x accuracy) says the recommendation step should be a cheap embedding similarity gate over benchmark relevantUseCases/descriptions. The ecosystem’s intersearch already provides shared embedding infra to plug in. Secondary frontier idea lacked: reliability-science framing (k-trial variance, robustness) is only partially present via contaminationRisk/predicted flags — leaderboards still rank on single point scores with no variance/confidence band surfaced to the caller.

Redundancy: Low/complementary. interrank is the data/ranking substrate; interspect (routing optimizer/calibration) and Clavain model-routing (economy/quality toggle) are explicit downstream consumers via resolve_routing_name/routing_compare — that is intended layering, not overlap. intermix is a cross-repo matrix EVAL harness (it produces benchmark data) vs interrank which SERVES ranking metadata — adjacent but distinct (intermix could feed AgMoDB). No tool-surface duplication with any other interverse plugin. The only redundancy risk is if interspect grows its own model-comparison logic instead of calling interrank’s routing_compare.

Actions:

  • Add an optional embedding-gated recommendation path: index benchmark relevantUseCases/descriptions via the existing intersearch embedding infra and use cosine similarity for the task->benchmark scoring in recommend.ts, keeping the keyword scorer as a deterministic fallback when no embedding backend is present (cite Select-then-Solve / RAG-MCP).
  • Fix doc drift: regenerate README.md, CLAUDE.md, and AGENTS.md to list all 15 tools (recommend_model, recommend_benchmarks, cost_leaderboard, domain_leaderboard, compare_models, resolve_routing_name, routing_compare, list_model_families are undocumented), and reconcile the version (plugin.json 0.3.1 vs McpServer version ‘0.2.0’ in index.ts).
  • Surface reliability signals in leaderboard/recommend output beyond a single point score: when available, expose a confidence/variance band or k-trial note, and propagate contaminationRisk/freshnessType into recommend_model results (currently only recommend_benchmarks carries them) so routing decisions inherit the benchmark-skepticism the SOTA mandates.
  • Consider exposing a thin Skill (SKILL.md) wrapper so the recommend_model/routing_compare flow is discoverable via progressive disclosure rather than only as raw MCP tools, aligning with the ‘thin skill orchestrating one MCP per system’ packaging pattern.