Commit Graph

136 Commits

Author SHA1 Message Date
Mathias Bergqvist
a37d18bf7a chore(docker): add poppler-utils for PDF text extraction 2026-04-23 16:02:12 +02:00
Mathias Bergqvist
2975eadc87 feat(watcher,api): use extract.Text() for file reading — fixes PDF ingestion 2026-04-23 16:01:36 +02:00
Mathias Bergqvist
53e46781b1 feat(pipeline): resolve proposed pages against inventory before writing 2026-04-23 16:00:31 +02:00
Mathias Bergqvist
e9b5cc401c feat(pipeline): add fuzzy entity resolution to prevent slug proliferation 2026-04-23 15:59:36 +02:00
Mathias Bergqvist
bf6f497d9d feat(wiki): add Aliases to Entry and read from YAML frontmatter 2026-04-23 15:57:16 +02:00
Mathias Bergqvist
9cc6c2d053 feat(extract): implement PDF extraction via pdftotext 2026-04-23 15:53:46 +02:00
Mathias Bergqvist
43a46d07e5 feat(extract): add Text() dispatcher with md/txt passthrough 2026-04-23 15:45:20 +02:00
Mathias Bergqvist
820d1c93a7 docs: add implementation plan for PDF extraction and entity resolution 2026-04-23 15:44:13 +02:00
Mathias Bergqvist
e74320a8e8 feat(ingestion): wire watcher into server startup + fix Procfile env vars
Some checks failed
cd / Build and deploy (push) Successful in 10s
CI / Lint / Test / Vet (push) Failing after 5s
CI / Mirror to GitHub (push) Has been skipped
- Start background watcher on startup when INGEST_WATCH_INTERVAL > 0
- Procfile: add INGEST_WATCH_INTERVAL=30 and INGEST_SVC_URL for supervisor

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 23:09:00 +02:00
Mathias Bergqvist
1b0706f270 chore(brain): rename CLAUDE.md to schema.md for clarity
CLAUDE.md has a specific meaning in the Claude Code ecosystem (agent
instructions). The wiki schema for the ingestion pipeline should live
in schema.md to avoid confusion.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 23:06:32 +02:00
Mathias Bergqvist
2ae6bfe81e fix(brain): enforce mutual exclusivity and clarify brain_ingest schema
- Return error when both path and content are supplied simultaneously
- Improve tool description to clearly state the two valid call forms
- Add per-field descriptions so LLMs understand what each parameter requires

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 23:03:03 +02:00
Mathias Bergqvist
a6dce972d6 feat(brain): add path field to brain_ingest for /ingest-path routing
Adds an optional path field to brain_ingest so Claude can ingest files
or directories directly by path without embedding content in the call.
Routing: path set → /ingest-path; content+source set → /ingest; neither → error.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 23:01:05 +02:00
Mathias Bergqvist
2f4b577131 fix(ingestion): address code review issues in api and watcher packages
- Strip internal error detail from 500 responses (leak prevention)
- Add path containment assertion in /write handler
- Use Go 1.22 method-prefixed mux routes for automatic 405 responses
- Clarify watch_interval log when watcher not yet wired
- Consolidate validation tests into table-driven TestIngest_Validation
- Watcher: return nil after successful quarantine to avoid double-logging
- Watcher: append timestamp suffix to processed dest if file already exists

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 22:59:39 +02:00
Mathias Bergqvist
a25bb18c54 feat(ingestion): add /ingest and /ingest-path HTTP handlers
Wires pipeline.Run into the HTTP layer so callers can ingest raw text
or files/directories without touching the filesystem directly. Rewrites
main.go to parse LLM and watcher env vars and build pipeline.Config.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 22:54:28 +02:00
Mathias Bergqvist
78531bb238 feat(ingestion): add background file watcher for brain/raw/
Polls brain/raw/ on a configurable ticker, derives human-readable source
names from filenames, runs the pipeline, and moves files to
processed/YYYY-MM-DD/ on success or failed/ on error with a log.md entry.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 22:54:03 +02:00
Mathias Bergqvist
04fefe8e9c fix(ingestion): wrap naked error returns and harden mustJSON helper
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 22:51:19 +02:00
Mathias Bergqvist
103f4d90bf feat(ingestion): add pipeline orchestrator with prompt builder
Adds prompt.go (BuildPrompt + systemPrompt) and pipeline.go (Run, Config,
Result, mergeAll) that wire chunking, LLM calls, parse, merge, index rebuild,
and log append into a single ingestion pipeline. Includes integration tests
covering write, dry-run, and duplicate-path merge scenarios.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 22:45:19 +02:00
Mathias Bergqvist
9b11719481 feat(ingestion): add content chunking and LLM JSON output parser 2026-04-22 22:37:14 +02:00
Mathias Bergqvist
d405346f07 feat(ingestion): add wiki index rebuilder and audit log 2026-04-22 22:36:55 +02:00
Mathias Bergqvist
bf8a3fc11c feat(ingestion): add OpenAI-compatible LLM HTTP client with 429 retry 2026-04-22 22:29:24 +02:00
Mathias Bergqvist
ae5a4d04f0 feat(ingestion): add wiki page merge logic 2026-04-22 22:28:55 +02:00
Mathias Bergqvist
3a0424a6b4 feat(ingestion): add wiki inventory loader 2026-04-22 22:28:53 +02:00
Mathias Bergqvist
08dd7b9365 docs(brain): add wiki schema document for ingest prompt 2026-04-22 22:25:52 +02:00
Mathias Bergqvist
91e02b930c feat(ingestion): add wiki package with Page types and slug generation 2026-04-22 22:25:45 +02:00
Mathias Bergqvist
c7341a2607 feat(config): add IngestSvcURL and KBRetrievalURL to supervisor config 2026-04-22 22:24:27 +02:00
Mathias Bergqvist
b5a0085c0a feat(brain): add brain_ingest, brain_search tools and extend search to wiki/ 2026-04-22 22:16:02 +02:00
Mathias Bergqvist
d6daa37c71 docs: add brain ingestion pipeline implementation plan 2026-04-22 22:14:59 +02:00
Mathias Bergqvist
62fc3989f2 docs: add brain ingestion pipeline design spec 2026-04-22 22:05:19 +02:00
Mathias Bergqvist
c9310b1079 fix(ingestion): always append .md extension to written filenames
All checks were successful
cd / Build and deploy (push) Successful in 9s
CI / Lint / Test / Vet (push) Successful in 10s
CI / Mirror to GitHub (push) Successful in 4s
brain_write with a custom filename omitted the .md extension, causing
search to skip the file (search.go filters on HasSuffix .md).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 19:23:07 +02:00
Mathias Bergqvist
ca8a691241 fix(exec): strip trailing result-schema JSON from local model output
All checks were successful
cd / Build and deploy (push) Successful in 6s
CI / Lint / Test / Vet (push) Successful in 10s
CI / Mirror to GitHub (push) Successful in 3s
Small models (phi4-mini) produce correct markdown analysis but then
append the old {status/phase/skill} JSON schema out of training habit.
stripResultJSON() detects and removes these trailing fences so Claude
Code receives clean prose regardless of model behaviour.

Non-schema json blocks (config examples etc) are preserved.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 16:55:53 +02:00
Mathias Bergqvist
214f607007 fix(config): make no-JSON instruction unmissable in protocols.md
All checks were successful
cd / Build and deploy (push) Successful in 7s
CI / Lint / Test / Vet (push) Successful in 10s
CI / Mirror to GitHub (push) Successful in 3s
Local models (phi4-mini, qwen3-coder-30b) ignore soft instructions
and revert to JSON from their training. Move the prohibition to the
very top with bold caps before any other content.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 16:51:51 +02:00
Mathias Bergqvist
0e08dfffb8 fix(config): rewrite all skill discipline files for simplified model
All checks were successful
cd / Build and deploy (push) Successful in 6s
CI / Lint / Test / Vet (push) Successful in 10s
CI / Mirror to GitHub (push) Successful in 3s
Remove JSON output contracts from all skill files (debug, review, spec,
tdd, retrospective, trainer-reader, trainer-writer). Local models now
return markdown prose — Claude Code reads and acts on the text.

Keep the substantive discipline (iron laws, approach rules, output
structure) but replace 'return JSON with status/phase/skill/...' with
clear markdown format instructions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 16:46:52 +02:00
Mathias Bergqvist
caef05bea4 fix(config): rewrite protocols.md for simplified model
All checks were successful
cd / Build and deploy (push) Successful in 6s
CI / Lint / Test / Vet (push) Successful in 9s
CI / Mirror to GitHub (push) Successful in 3s
Remove JSON output contract, verification rules, escalation, and scope
limits that applied to the old Claude subprocess workers. Local models
are now consultants returning markdown prose, not JSON executors.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 16:43:04 +02:00
Mathias Bergqvist
ca1a16873c feat(ingestion): add Dockerfile and extend CD to build+push ingestion image
All checks were successful
cd / Build and deploy (push) Successful in 9s
CI / Lint / Test / Vet (push) Successful in 9s
CI / Mirror to GitHub (push) Successful in 3s
Ingestion server is a pure-Go HTTP binary — alpine runtime, no node.js.
CD now builds both supervisor and ingestion images on every push,
updates both deployment.yaml files in the infra repo.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 16:37:11 +02:00
Mathias Bergqvist
63c238c650 fix(config): update model names to match LiteLLM host/name format
All checks were successful
cd / Build and deploy (push) Successful in 6s
CI / Lint / Test / Vet (push) Successful in 9s
CI / Mirror to GitHub (push) Successful in 4s
Replace ollama/ prefix with iguana/ and koala/ prefixes to match
actual model IDs exposed by LiteLLM on this cluster.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 16:34:53 +02:00
Mathias Bergqvist
ce45592730 refactor: replace orchestrator/verifier chain with direct LiteLLM calls
All checks were successful
cd / Build and deploy (push) Successful in 6s
CI / Lint / Test / Vet (push) Successful in 10s
CI / Mirror to GitHub (push) Successful in 3s
Drop the three-layer Claude subprocess orchestration (local model →
Claude verifier → cloud escalation). Skills now call LiteLLM directly
and return plain text to Claude Code, which decides what to do with it.

- Delete executor, orchestrator, verifier, result, attempts packages
- Simplify LiteLLMExecutor: Run(Request)→Result becomes Complete(model,sys,user)→(string,int64,error)
- Replace ExecutorFn with CompleteFunc in all 6 skill configs
- Rewrite all skill handlers to call Complete and return {"text","model","duration_ms"}
- Simplify config/models: remove Verifier/LlamaSwapURL, add ModelFor
- Bump version to v0.5.0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 16:19:09 +02:00
Mathias Bergqvist
823de23213 feat(exec): log per-attempt chain verdicts for pass rate visibility
All checks were successful
cd / Build and deploy (push) Successful in 6s
CI / Lint / Test / Vet (push) Successful in 1m9s
CI / Mirror to GitHub (push) Successful in 4s
2026-04-22 15:40:15 +02:00
Mathias Bergqvist
78d3939caa feat(config): wire protocols.md into every worker as shared behavioral contract 2026-04-22 15:39:25 +02:00
Mathias Bergqvist
f2bc39b500 feat(skills): inject brain context into review, debug, spec, tdd before spawning workers 2026-04-22 15:37:56 +02:00
Mathias Bergqvist
3625e1268d feat(ingestion): simplify brain to knowledge/ — write and search use same dir 2026-04-22 15:36:10 +02:00
Mathias Bergqvist
47df642836 feat(brain): add Query client for skill handler context injection 2026-04-22 15:34:09 +02:00
Mathias Bergqvist
235d70ad0b docs: document hyperguild scope reset — drop parametric learning, simplify brain 2026-04-22 15:27:52 +02:00
Mathias Bergqvist
7d5289ac54 chore: bump version to v0.4.0
All checks were successful
cd / Build and deploy (push) Successful in 6s
CI / Lint / Test / Vet (push) Successful in 1m9s
CI / Mirror to GitHub (push) Successful in 3s
2026-04-22 13:38:23 +02:00
Mathias Bergqvist
3d8fc9dacd feat(skills): wire session.Append into retrospective and trainer 2026-04-22 13:37:43 +02:00
Mathias Bergqvist
f9f804cd49 feat(skills): wire session.Append and PrependHistory into tdd 2026-04-22 13:37:06 +02:00
Mathias Bergqvist
85f142ade0 feat(skills): wire session.Append and PrependHistory into spec 2026-04-22 13:36:35 +02:00
Mathias Bergqvist
0dfad02513 feat(skills): wire session.Append and PrependHistory into review and debug 2026-04-22 13:36:13 +02:00
Mathias Bergqvist
c44eb680b2 feat(exec): surface AttemptRecord slice on Result for session logging 2026-04-22 13:35:38 +02:00
Mathias Bergqvist
38ada998a2 feat(session): add AttemptsFrom converter for exec.AttemptRecord 2026-04-22 13:35:09 +02:00
Mathias Bergqvist
74547c2bdf feat(session): export PrependHistory for shared use across skills 2026-04-22 13:34:48 +02:00