57462b52ff
Wires nomic-embed-text (iguana ollama) + pgvector on the shared
postgres18 into brain_query / brain_answer via Reciprocal Rank Fusion.
Pure BM25 stays the default; setting BRAIN_PG_DSN and BRAIN_EMBED_URL
together opts in. Setting one without the other is misconfiguration →
exit 1.
New packages:
- internal/embed
Client.Embed(ctx, text) → []float32 via POST {URL}/api/embed.
Defaults to nomic-embed-text:latest (768 dim). nil-on-empty-URL so
callers gate on a single nil check.
- internal/vectorstore
PGStore wraps a pgxpool against postgres18. Init creates
brain_embeddings(path PK, vector(768), updated_at) + HNSW cosine
index idempotently. Upsert / Delete / Search / KnownPaths.
Sync(brainDir, store, embedder) diffs brain/wiki/ against the store
and upserts new files / deletes removed ones; StartSync runs it on
a ticker (default 300s). Integration tests gated by BRAIN_PG_TEST_DSN.
- scripts/brain-embeddings-init.sql
One-time DBA setup: brain DB, brain_app role, vector extension,
GRANTs. Idempotent.
Search layer:
- search.QueryOptions gains Vector + Embedder fields.
- QueryContext is the cancellable variant; Query stays for callers.
- When both are set, BM25 (top-N) and pgvector (top-4N) candidates
merge via Reciprocal Rank Fusion (k=60, Cormack et al. 2009 — no
tuning knob, robust to scale differences between rankers).
- Vector-only hits are hydrated from disk so callers see uniform
Result records (path, title, excerpt, wing, hall, score).
- Wing/hall filters still apply to vector candidates via path-prefix.
- On embedder/vector errors the search falls back to BM25 — embedding
outage degrades quality but doesn't take the brain offline.
MCP wiring:
- mcp.Server.WithHybridRetrieval(v, e) opt-in setter, same shape as
WithReranker.
- brainQuery and brainAnswer pass the wired vector/embedder through
to search.QueryContext.
REST:
- POST /backfill-embeddings drives Sync synchronously. Returns
{added, deleted, errors[]}. 503 when feature is unconfigured.
cmd/server/main.go:
- BRAIN_PG_DSN + BRAIN_EMBED_URL together enable hybrid; one alone
→ exit 1.
- vectorAdapter bridges *PGStore (returns []Hit) to
search.VectorSearcher (which takes []VectorHit) without either
package importing the other.
- BRAIN_EMBED_SYNC_INTERVAL (default 300s) controls the background
Sync ticker.
Backend pivot from Qdrant to pgvector recorded in DECISIONS.md
2026-05-18 (supersedes 2026-04-08): postgres18 already runs in
databases/ ns, Qdrant was never deployed, one engine beats two.
Dependency: github.com/jackc/pgx/v5 — modern, native pgvector via
parametric vector literals.
Tests:
- embed.Client: empty-URL nil, request shape, dimension, upstream
error propagation, empty-text rejection.
- vectorstore.PGStore: dimension validation (unit); upsert/search/
KnownPaths (integration, BRAIN_PG_TEST_DSN-gated).
- vectorstore.Sync: adds new files, skips known, deletes
disappeared, skips _index.md, no-op when nil, collects embedder
errors.
- search.Query: hybrid promotes vector-only hits via RRF; falls
back to BM25 on embedder error.
Closes hyperguild#8.
92 lines
2.3 KiB
Go
92 lines
2.3 KiB
Go
package vectorstore_test
|
|
|
|
import (
|
|
"context"
|
|
"os"
|
|
"testing"
|
|
"time"
|
|
|
|
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
|
|
"github.com/stretchr/testify/assert"
|
|
"github.com/stretchr/testify/require"
|
|
)
|
|
|
|
// integration tests run against a real postgres18 + pgvector. Gated by
|
|
// BRAIN_PG_TEST_DSN so `task check` stays hermetic on hosts without a
|
|
// reachable database.
|
|
//
|
|
// To run:
|
|
// BRAIN_PG_TEST_DSN='postgres://brain_app:pwd@127.0.0.1:5432/brain' \
|
|
// go test ./internal/vectorstore/... -run Integration
|
|
func dsn(t *testing.T) string {
|
|
t.Helper()
|
|
v := os.Getenv("BRAIN_PG_TEST_DSN")
|
|
if v == "" {
|
|
t.Skip("BRAIN_PG_TEST_DSN not set; skipping pgvector integration tests")
|
|
}
|
|
return v
|
|
}
|
|
|
|
func freshStore(t *testing.T) (*vectorstore.PGStore, context.Context) {
|
|
t.Helper()
|
|
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
|
t.Cleanup(cancel)
|
|
s, err := vectorstore.New(ctx, dsn(t))
|
|
require.NoError(t, err)
|
|
t.Cleanup(s.Close)
|
|
require.NoError(t, s.Init(ctx))
|
|
// Clean slate per test.
|
|
_, _ = s.KnownPaths(ctx)
|
|
require.NoError(t, s.Delete(ctx, "%test-fixture%"))
|
|
return s, ctx
|
|
}
|
|
|
|
func vec(dim int, fill float32) []float32 {
|
|
v := make([]float32, dim)
|
|
for i := range v {
|
|
v[i] = fill
|
|
}
|
|
return v
|
|
}
|
|
|
|
func TestIntegration_UpsertAndSearch(t *testing.T) {
|
|
s, ctx := freshStore(t)
|
|
|
|
require.NoError(t, s.Upsert(ctx, "wiki/a.md", vec(768, 1.0)))
|
|
require.NoError(t, s.Upsert(ctx, "wiki/b.md", vec(768, -1.0)))
|
|
|
|
hits, err := s.Search(ctx, vec(768, 1.0), 2)
|
|
require.NoError(t, err)
|
|
require.GreaterOrEqual(t, len(hits), 1)
|
|
assert.Equal(t, "wiki/a.md", hits[0].Path)
|
|
assert.InDelta(t, 0.0, hits[0].Distance, 1e-5)
|
|
|
|
t.Cleanup(func() {
|
|
_ = s.Delete(ctx, "wiki/a.md")
|
|
_ = s.Delete(ctx, "wiki/b.md")
|
|
})
|
|
}
|
|
|
|
func TestIntegration_KnownPaths(t *testing.T) {
|
|
s, ctx := freshStore(t)
|
|
require.NoError(t, s.Upsert(ctx, "wiki/k.md", vec(768, 0.5)))
|
|
t.Cleanup(func() { _ = s.Delete(ctx, "wiki/k.md") })
|
|
|
|
paths, err := s.KnownPaths(ctx)
|
|
require.NoError(t, err)
|
|
_, ok := paths["wiki/k.md"]
|
|
assert.True(t, ok)
|
|
}
|
|
|
|
func TestUpsert_RejectsWrongDimension(t *testing.T) {
|
|
s := &vectorstore.PGStore{}
|
|
err := s.Upsert(context.Background(), "x", vec(100, 0))
|
|
require.Error(t, err)
|
|
}
|
|
|
|
func TestSearch_RejectsWrongDimension(t *testing.T) {
|
|
s := &vectorstore.PGStore{}
|
|
_, err := s.Search(context.Background(), vec(100, 0), 5)
|
|
require.Error(t, err)
|
|
}
|