feat(vectorstore): re-embed on file mtime > store updated_at (#23)
All checks were successful
CI / Lint / Test / Vet (push) Successful in 11s
CI / Mirror to GitHub (push) Has been skipped

Removes the TODO in Sync that left files static after their first embed.
Edits to brain/wiki/ and brain/knowledge/ now surface in subsequent
syncs without manual /backfill-embeddings calls.

Approach
- Store interface: KnownPaths → KnownPathsWithTime returning path →
  updated_at. Callers compare against file mtime to detect edits.
- PGStore: SELECT path, updated_at FROM brain_embeddings.
- Sync groups known chunks by parent path and tracks the EARLIEST
  updated_at per parent. A file is stale when its mtime is after that
  oldest chunk's timestamp — any chunk older than the file means at
  least one chunk hasn't been refreshed since the last edit.
- Stale-path rewrite: delete every old chunk for the parent (handles
  "file shrunk → fewer chunks → orphan rows at higher #NNNN" cleanly),
  then re-chunk + re-embed + re-upsert.

Tests
- New: TestSync_ReembedsFileWhenMtimeNewer — file mtime forced into the
  future vs store updated_at; Sync deletes old chunk + upserts fresh one.
- New: TestSync_SkipsFileWhenMtimeOlder — file mtime backdated; Sync is
  a no-op (no upserts, no deletes).
- Updated: stubStore.known is now map[string]time.Time. A zero value
  resolves to a far-future sentinel so existing "skip if already known"
  tests keep passing without per-test setup.
- pg_test renamed KnownPaths integration → KnownPathsWithTime; asserts
  updated_at is non-zero and within 5s of insert wall-clock.

Backward compat
- brain_embeddings rows pre-dating this change carry valid updated_at
  values (column was always populated via `DEFAULT now()` + ON CONFLICT
  `updated_at = now()`). No migration needed. Live pod will start
  re-embedding any file whose source has been edited since its chunks
  were originally written.

Closes gitea/mathias/hyperguild#23.
This commit is contained in:
Mathias
2026-05-20 09:50:45 +02:00
parent 6f1cb53295
commit 815739758e
4 changed files with 139 additions and 40 deletions

View File

@@ -8,6 +8,7 @@ import (
"errors"
"fmt"
"strings"
"time"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
@@ -120,21 +121,26 @@ func (s *PGStore) Search(ctx context.Context, query []float32, limit int) ([]Hit
return hits, nil
}
// KnownPaths returns the path set already present in the store. Used by
// the watcher to diff against the wiki/ tree and decide what to upsert.
func (s *PGStore) KnownPaths(ctx context.Context) (map[string]struct{}, error) {
rows, err := s.pool.Query(ctx, `SELECT path FROM brain_embeddings`)
// KnownPathsWithTime returns every embedded chunk path paired with the
// row's updated_at. Sync uses the timestamps to decide whether a file
// has been edited since its chunks were last embedded — when the file's
// mtime exceeds the oldest chunk's updated_at, the file is re-embedded.
func (s *PGStore) KnownPathsWithTime(ctx context.Context) (map[string]time.Time, error) {
rows, err := s.pool.Query(ctx, `SELECT path, updated_at FROM brain_embeddings`)
if err != nil {
return nil, fmt.Errorf("query paths: %w", err)
}
defer rows.Close()
out := make(map[string]struct{})
out := make(map[string]time.Time)
for rows.Next() {
var p string
if err := rows.Scan(&p); err != nil {
var (
p string
t time.Time
)
if err := rows.Scan(&p, &t); err != nil {
return nil, err
}
out[p] = struct{}{}
out[p] = t
}
return out, rows.Err()
}