Commit Graph

14 Commits

Author SHA1 Message Date
Mathias
75685e7b67 feat(brain): structured wing/hall taxonomy + obsidian-compatible layout
All checks were successful
CI / Lint / Test / Vet (push) Successful in 11s
CI / Mirror to GitHub (push) Successful in 4s
Adds a two-dimensional address (wing, hall) to brain notes. A wing is a
topic domain (e.g. jepa-fx, hyperguild); a hall is one of a closed
vocabulary of memory types (facts, decisions, failures, hypotheses,
sources). Notes route to brain/wiki/<wing>/<hall>/<slug>.md with
wing/hall/created_at YAML frontmatter, making the directory a valid
Obsidian vault.

Changes:
- new package ingestion/internal/brain (NotePath, ValidHalls, Sanitise,
  BuildWingIndex, BuildAllWingIndexes)
- api.WriteNote refactored to WriteNoteOptions; wing+hall routes to
  brain/wiki/, otherwise falls back to brain/knowledge/ (legacy)
- search.Query → QueryOptions with optional Wing/Hall filtering; Result
  carries wing/hall extracted from frontmatter or path segments
- MCP tools brain_write and brain_query gain optional wing/hall params
  (hall enum-validated); new brain_index tool regenerates _index.md MOC
- POST /index REST endpoint mirrors brain_index
- brain_write auto-rebuilds the wing's _index.md after a wing+hall write
- scripts/migrate-brain-halls.sh migrates flat brain/wiki/{concepts,entities}/
  into the new layout (dry-run by default, --commit applies)

All existing tests pass; new tests cover wing/hall write routing, scope
filtering, invalid hall rejection, _index.md generation, and migration
script paths.

Closes hyperguild#1.
2026-05-18 20:47:08 +02:00
Mathias Bergqvist
37dbd22eff feat(brain): /pass-rate aggregator and handler
Adds a new HTTP GET handler at the ingestion pod that walks
brain/sessions/*.jsonl, filters by skill name and timestamp window
(default 7d, accepts Nh and Nd), normalizes legacy status vocabulary
(ok->pass, error->fail, skipped->skip), and returns aggregated counts
plus pass_rate.

Pass rate is null when pass+fail == 0, distinguishing 'no data' from
'always passes'. Plan 6 routing pod will check for null before
making decisions.

Route registration in cmd/server/main.go lands in a follow-up commit.
2026-05-03 22:37:41 +02:00
Mathias Bergqvist
e4a94df4fc feat(ingestion): extract WriteNote helper and add brain_write MCP tool
api.WriteNote captures the file-write logic that was previously inline
in Handler.Write. The existing HTTP endpoint now delegates to it; the
new MCP brain_write tool reuses the same function. Path-traversal
guard is strengthened to explicitly reject filenames containing path
separators or "..", so the rejection is surfaced before filepath.Base
strips the suspicious component (the previous defense-in-depth prefix
check became unreachable for these inputs after Base normalisation).
HTTP error code for caller-input errors shifts from 500 to 400, which
is semantically correct and not exercised by any existing test.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 10:25:38 +02:00
Mathias Bergqvist
0a70d9e972 feat(pipeline): add POST /ingest-raw for direct batch ingestion without LLM
All checks were successful
CI / Lint / Test / Vet (push) Successful in 9s
CI / Mirror to GitHub (push) Has been skipped
Allows callers to provide pre-structured RawPage data directly, bypassing the
LLM extraction step. The pipeline still handles slug computation, frontmatter,
link canonicalization, source back-references, and dedup — only the extraction
is skipped. Useful when a more capable model or manual curation produces the
structured data.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-24 11:15:59 +02:00
Mathias Bergqvist
923a665365 fix(pipeline): skip RawPages with empty title in BuildPages instead of producing broken paths
All checks were successful
CI / Lint / Test / Vet (push) Successful in 9s
CI / Mirror to GitHub (push) Has been skipped
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-23 19:55:37 +02:00
Mathias Bergqvist
1605624668 feat(pipeline): add POST /backfill-refs endpoint to retroactively inject source back-references 2026-04-23 16:50:00 +02:00
Mathias Bergqvist
2975eadc87 feat(watcher,api): use extract.Text() for file reading — fixes PDF ingestion 2026-04-23 16:01:36 +02:00
Mathias Bergqvist
2f4b577131 fix(ingestion): address code review issues in api and watcher packages
- Strip internal error detail from 500 responses (leak prevention)
- Add path containment assertion in /write handler
- Use Go 1.22 method-prefixed mux routes for automatic 405 responses
- Clarify watch_interval log when watcher not yet wired
- Consolidate validation tests into table-driven TestIngest_Validation
- Watcher: return nil after successful quarantine to avoid double-logging
- Watcher: append timestamp suffix to processed dest if file already exists

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 22:59:39 +02:00
Mathias Bergqvist
a25bb18c54 feat(ingestion): add /ingest and /ingest-path HTTP handlers
Wires pipeline.Run into the HTTP layer so callers can ingest raw text
or files/directories without touching the filesystem directly. Rewrites
main.go to parse LLM and watcher env vars and build pipeline.Config.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 22:54:28 +02:00
Mathias Bergqvist
c9310b1079 fix(ingestion): always append .md extension to written filenames
All checks were successful
cd / Build and deploy (push) Successful in 9s
CI / Lint / Test / Vet (push) Successful in 10s
CI / Mirror to GitHub (push) Successful in 4s
brain_write with a custom filename omitted the .md extension, causing
search to skip the file (search.go filters on HasSuffix .md).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 19:23:07 +02:00
Mathias Bergqvist
3625e1268d feat(ingestion): simplify brain to knowledge/ — write and search use same dir 2026-04-22 15:36:10 +02:00
Mathias Bergqvist
24d9216474 fix(ingestion): preserve type and domain metadata as frontmatter in written notes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-17 21:22:14 +02:00
Mathias Bergqvist
d18fa0dd59 fix(ingestion): validate required query field in Query handler
Empty or whitespace-only queries would silently pass through to search,
returning meaningless results. Also removed the Domain field from
queryRequest — it was accepted but silently ignored since search.Query
has no domain parameter, which would confuse callers.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-17 20:27:02 +02:00
Mathias Bergqvist
e20edd6ca9 feat(ingestion): add query and write HTTP handlers
Implements POST /query (BM25 search via internal/search) and POST /write
(raw file persistence to brain/raw/) as an api.Handler struct. Filename
is auto-generated when absent.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-17 20:24:51 +02:00