interfer

keep priority P2 effort M wave 2

What it does: A custom local MLX-LM inference server for Apple Silicon (M5 Max 128GB), packaged as a Claude Code plugin via a stdio MCP server. Confirmed from code, not the hint: Starlette/uvicorn HTTP front (OpenAI-compatible /v1/chat/completions SSE) + a spawned Metal subprocess (spawn-not-fork to avoid GPU semaphore leaks) owning all MLX, communicating over a priority multiprocessing.Queue. ~6,160 lines across 21 server modules + 26 test files. Real, working frontier-grade internals: speculative decoding with draft models (inference.py/benchmark.py), KV-cache quantization (kv_bits 2/4/8, benchmarked), entropy-based early-exit layer skipping, a ConfidenceCascade that probes the smallest model first and escalates/fails-to-cloud (cascade.py), reservoir-routing task-classifier MLP, thermal-aware scheduling (macOS notify API), flash-MoE SSD expert streaming for a 397B model (~13 tok/s, streaming_switch.py), Prometheus metrics (prom.py), prompt caching, batch scheduler, shadow logging, and a holistic benchmark harness (quality+latency+memory+reliability, LiveCodeBench-v6 pass@1, LLM-judge). Wired into Clavain routing as Track B5 (currently shadow mode). The MCP layer (server/mcp.py) is a hand-rolled JSON-RPC stdio loop exposing only 3 read-only tools: interfer_health, interfer_models, interfer_load.

Rationale: This is a mature, very actively developed plugin (last commit 2026-06-22, dense bead/handoff history) whose core thesis is directly validated by the SOTA findings: ‘on M-series Macs MLX is now the fastest path’ and ‘local LLM inference bifurcated: Ollama for dev, vLLM for prod, MLX fastest on Apple Silicon.’ The plugin already implements the techniques the frontier prizes for local serving — speculative decoding, KV-cache quantization (the ChunkKV/KeyDiff family’s motivation), confidence-based cascade routing (the model-routing/economy-mode pattern), and a holistic eval that goes beyond pass@1 into variance/reliability (the ‘reliability science: beyond pass@1’ finding) and grounds the LLM-judge in executed code (the ‘verification must be tool-grounded, not free-form prose’ finding). It owns hooks inside the forward pass that off-the-shelf servers (Ollama, vLLM-mlx, llama.cpp) cannot expose — a deliberate, defensible bet, not NIH. Verdict is keep: the engineering is sound and ahead of most competitors on the dimension that matters (custom-loop local inference on Apple Silicon). It is not ‘modernize’ because the inference internals already track or lead SOTA; the only soft spot (MCP surface) is a small, low-effort improvement rather than an architectural deficiency.

SOTA gap: Two narrow gaps, both at the MCP/integration surface rather than the inference core. (1) The MCP server is hand-rolled JSON-RPC (protocolVersion 2024-11-05, no SDK) exposing only 3 read-only tools and does NOT expose the plugin’s genuine differentiator — confidence-cascade routing / ‘should this run local or fail-to-cloud’ — to the orchestrating agent. The whole point per SOTA model-routing/economy-mode findings (Plan/Execute splitting, BYOK routing, distilled-local-model decision fast-paths) is that an orchestrator should be able to ask interfer ‘can you handle this cheaply/privately?’ and get a routed answer; today that intelligence lives only behind the HTTP API and Clavain’s shadow-mode config, invisible over MCP. (2) No progressive-disclosure / Tool Search alignment — moot today (only 3 tools, no context-bloat) but relevant the moment generation/routing/cascade-stats tools are added: they should be deferred-loadable. The inference internals lack no major frontier idea — speculative decoding, KV-quant, early exit, and cascade are present.

Redundancy: Low and complementary, not overlapping. It is the only plugin that actually runs models; interrank ranks models/benchmarks (AgMoDB) and interspect profiles agent/skill performance + routing optimization — interfer is the execution substrate those two reason about (vision doc explicitly registers interfer’s local models into interrank’s snapshot and feeds interspect evidence). interlab runs the A/B experiment campaigns interfer’s optimizations enter as (declared companion in integration.json). Clavain owns the task-to-model routing decision; interfer owns within-request cascade/escalation. The model-routing concept is shared with interrank/interspect/Clavain but the layers are distinct (rank vs. profile vs. orchestrate vs. execute). No merge candidate.

Actions:

Expose the confidence-cascade routing decision over MCP: add a deferred-loadable tool (e.g. interfer_route or interfer_can_handle) that takes a prompt + privacy/complexity hints and returns the cascade’s accept/escalate/cloud decision + chosen local model + estimated tok/s, so the orchestrator can make local-vs-cloud calls without parsing Clavain config. This surfaces the plugin’s real differentiator to agents.
Migrate server/mcp.py off the hand-rolled JSON-RPC loop to the MCP Python SDK (FastMCP) and bump protocolVersion; this removes a maintenance liability (manual notifications/initialized handling, no schema validation) and makes adding generation/stats tools safe. Mark non-essential tools defer_loading-friendly per the progressive-disclosure SOTA.
Add an interfer_generate (or interfer_complete) MCP tool that calls /v1/chat/completions so agents in this harness can use sovereign local inference directly for routine/sensitive subtasks — the ‘sovereignty / sensitive code never leaves the machine’ mission promise is currently only reachable via raw curl, not via the agent’s own tool surface.
Surface cascade + Prometheus stats (accept_rate, cloud_rate, thermal, tok/s) as a read-only MCP resource or tool so interspect can ingest local-model quality evidence programmatically rather than out-of-band, closing the evidence loop the vision doc describes.