chore(security): rotate brain_app postgres password — leaked in pre-fix pod logs #20

New Issue

mathias · 2026-05-19T11:10:55Z

mathias commented

2026-05-19 11:10:55 +00:00

Context

The original cmd/server/main.go "redaction" for the BRAIN_PG_DSN log line was wrong — it sliced up to @, which is after the password in a Postgres URL, so the password landed in the structured log line of the ingestion pod under image 7a13c756... on 2026-05-18 → 2026-05-19.

Fixed in 4af10364 via url.URL.Redacted() (now prints xxxxx). But the old log lines remain in kubectl logs --previous history and any sink the cluster logs are shipped to until that retention rolls off (typically 7–14 days for container runtime; longer if logs are shipped to a SIEM).

Action

Rotate brain_app postgres password end-to-end:

Generate new password: umask 077 && openssl rand -hex 24 > /tmp/brain_app_pg.pwd
Apply via init SQL (the script is idempotent — \if :role_exists ALTER ROLE ... WITH PASSWORD):
```
kubectl exec -i -n databases postgres18-0 -- \
  psql -U postgres -v password="$(cat /tmp/brain_app_pg.pwd)" \
  < scripts/brain-embeddings-init.sql
```
(Note: bare value, NO surrounding single quotes — :'password' does its own quoting.)

Rebuild DSN and SOPS-set:

DSN="postgres://brain_app:$(cat /tmp/brain_app_pg.pwd)@postgres18.databases.svc.cluster.local:5432/brain?sslmode=disable"
cd ~/dev/AI/infra
sops set k3s/apps/supervisor/secrets.enc.yaml '["stringData"]["BRAIN_PG_DSN"]' "\"$DSN\""

Commit the SOPS change and bump secrets-revision on ingestion-deployment.yaml to roll the pod.
rm /tmp/brain_app_pg.pwd

Acceptance criteria

New brain_app password generated and stored only in SOPS
Pod restart picks up the new DSN cleanly (brain hybrid retrieval enabled line + no auth errors)
/tmp/brain_app_pg.pwd deleted
Optional: old pod-log entries containing the leaked password purged sooner via kubectl delete pod once more after retention concern eases

Severity

Low — the leak was internal-only (kubectl logs require cluster admin), and the affected DB sits behind Tailscale. Worth doing as hygiene, not urgent.

## Context The original `cmd/server/main.go` "redaction" for the `BRAIN_PG_DSN` log line was wrong — it sliced up to `@`, which is *after* the password in a Postgres URL, so the password landed in the structured log line of the ingestion pod under image `7a13c756...` on 2026-05-18 → 2026-05-19. Fixed in `4af10364` via `url.URL.Redacted()` (now prints `xxxxx`). But the old log lines remain in `kubectl logs --previous` history and any sink the cluster logs are shipped to until that retention rolls off (typically 7–14 days for container runtime; longer if logs are shipped to a SIEM). ## Action Rotate `brain_app` postgres password end-to-end: 1. Generate new password: `umask 077 && openssl rand -hex 24 > /tmp/brain_app_pg.pwd` 2. Apply via init SQL (the script is idempotent — `\if :role_exists ALTER ROLE ... WITH PASSWORD`): ```bash kubectl exec -i -n databases postgres18-0 -- \ psql -U postgres -v password="$(cat /tmp/brain_app_pg.pwd)" \ < scripts/brain-embeddings-init.sql ``` (Note: bare value, NO surrounding single quotes — `:'password'` does its own quoting.) 3. Rebuild DSN and SOPS-set: ```bash DSN="postgres://brain_app:$(cat /tmp/brain_app_pg.pwd)@postgres18.databases.svc.cluster.local:5432/brain?sslmode=disable" cd ~/dev/AI/infra sops set k3s/apps/supervisor/secrets.enc.yaml '["stringData"]["BRAIN_PG_DSN"]' "\"$DSN\"" ``` 4. Commit the SOPS change and bump `secrets-revision` on `ingestion-deployment.yaml` to roll the pod. 5. `rm /tmp/brain_app_pg.pwd` ## Acceptance criteria - [ ] New `brain_app` password generated and stored only in SOPS - [ ] Pod restart picks up the new DSN cleanly (`brain hybrid retrieval enabled` line + no auth errors) - [ ] `/tmp/brain_app_pg.pwd` deleted - [ ] Optional: old pod-log entries containing the leaked password purged sooner via `kubectl delete pod` once more after retention concern eases ## Severity Low — the leak was internal-only (kubectl logs require cluster admin), and the affected DB sits behind Tailscale. Worth doing as hygiene, not urgent.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mathias/hyperguild#20