intersearch

Rationale: Well-scoped, documented, tested infrastructure that correctly implements the ‘one shared embedding layer, many consumers’ consolidation thesis the SOTA validates (agent memory as a distinct architectural layer; embedding-based gating; RAG-MCP-style retrieval). The mechanism/policy split is the right design. But three confirmed code-level deficiencies hold it below current retrieval state of the art: (1) it embeds whole files as a single vector (store.index_file calls embedder.embed(content) on the full file; nomic’s 8192-token cap silently truncates large files) where SOTA retrieval is chunk-based; (2) query is a brute-force O(N) linear scan in a Python loop (SELECT all rows, NumPy dot product each) with no ANN index (sqlite-vec/FAISS/hnswlib); (3) it is pure dense semantic with no hybrid dense+sparse retrieval, which findings call ‘now standard for accuracy’ — ironically the cass engine in its own bundled session-search skill already runs —mode hybrid. embed_batch exists but indexing never uses it (the vision’s own ‘batched embedding’ direction is unbuilt). These are incremental upgrades to a sound foundation, not a rewrite — hence ‘improve.’ Python MCP infrastructure providing the ecosystem’s shared embedding/search mechanism. server.py exposes two MCP tools (embedding_index, embedding_query); store.py persists vectors in per-project SQLite (~/.intersearch/index//embeddings.db, WAL, SHA256-gated incremental indexing, model-version invalidation); embeddings.py wraps sentence-transformers (default nomic-embed-text-v1.5 768d, legacy all-MiniLM 384d) with lazy load and L2-normalized vectors; exa.py is an async Exa web-search client with multi-query dedup and fail-open on missing EXA_API_KEY. A bundled session-search skill delegates to the external cass binary for session history (orthogonal). Deliberately ‘mechanism, not policy’: indexes and retrieves by cosine similarity, leaves ranking/intent to consumers. interject consumes it as a path dependency.

SOTA gap: Chunk-level + hybrid retrieval with an ANN index. Current SOTA retrieval folds chunking, dense+sparse hybrid scoring, and approximate-nearest-neighbor indexing into the baseline; intersearch does whole-file dense vectors with linear scan. As the shared substrate every consumer builds on, this single layer caps the recall/precision ceiling of the whole ecosystem. The graph/agentic-retrieval frontier (GraphRAG, agentic multi-hop) is correctly out of scope here — that is consumer policy, not this mechanism’s job.

Redundancy: Three real overlaps. (1) intercache still ships its own src/intercache/embeddings.py — the vision claims intersearch superseded it, but consolidation is incomplete; intercache should migrate to the intersearch path dep or be confirmed deprecated. (2) interknow/qmd searches 2946 markdown docs and its status banner says ‘No vector embeddings yet — run qmd embed’; that planned embedding layer overlaps intersearch’s mechanism and should reuse it. (3) tldr-swinton has its own semantic_index/semantic tools for code reconnaissance — another independent embedding path the vision lists as a target consumer but never wired in. The bundled session-search/cass skill is orthogonal (session transcripts, not embeddings) and arguably belongs with interstat.

Actions:

• Add document chunking before embedding (sliding-window or semantic-section split with overlap; store chunk-id + char span per row) so large files aren’t silently truncated to one vector — the single biggest retrieval-quality win, plus an embeddings-table schema change.
• Replace the O(N) Python linear scan in store.query with an ANN index: adopt sqlite-vec (keeps the single-file-per-project SQLite model) or FAISS/hnswlib, falling back to brute force only for tiny indices.
• Wire indexing through the existing embed_batch path (batch files/chunks per encode call) to deliver the vision’s already-promised batched embedding and cut model-load/GPU overhead.
• Expose stale_paths and an index_stats MCP tool (count, model, last-updated) — both named in docs/intersearch-vision.md Direction but unbuilt — and finish consolidation by migrating intercache off its own embeddings.py and pointing interknow/qmd’s embedding layer at intersearch.