GOAT0.1 (Kid) · Source preview

ADR-0005: Pluggable memory: Markdown default, LLM Wiki and Hindsight optional ​

Status: Accepted · 2026-08-29 · Amended 2026-09-01

Context ​

GOAT should remember through three distinct launch providers: Markdown (local), LLM Wiki (local), and Hindsight (MIT, vectorize-io), exposed to coding agents by @vectorize-io/hindsight-coding-agents through MCP. Obsidian is a future first-class adapter over the local providers' files.

Markdown (local) is deliberately simple, readable local memory: strict note frontmatter, Markdown bodies, a derived index, and safe bounded retrieval. It is the current default. LLM Wiki (local) is a separate provider, not a cosmetic name for Markdown (local): before it is selectable it must provide immutable source material, curated/generated wiki pages and crosslinks, an explicit maintenance schema, and ingest/query/lint lifecycle operations. Its concrete local authority and tool contract are in the LLM Wiki contract. This distinction prevents GOAT from overpromising a wiki while shipping only ordinary Markdown notes.

The original decision described Hindsight as a direct retain / recall / reflect HTTP adapter to a local Docker service. That is no longer the coding-agent contract. The coding-agent package now owns per-repository bank resolution, knowledge pages, an eight-tool MCP surface, and lifecycle hooks for automatic background ingestion in supported agent harnesses. It can target Hindsight Cloud, a self-hosted server, or its local daemon according to the user's ~/.hindsight/coding-agent.json. GOAT must consume the explicit tool contract instead of rebuilding a second, drifting Hindsight client, but it does not inherit hooks merely by launching the MCP process. Hindsight is a first-class memory provider in GOAT, however, not a generic MCP row that the user must assemble and keep healthy by hand.

The file and service backends have incompatible capabilities, so the seam must describe them honestly rather than pretending every operation exists everywhere.

Decision ​

GoatMemory defines a MemoryStore with a structured index snapshot, browser entries, explicit remember operations, capability-gated deletion, and model tools. Memory file or network I/O runs in store actors or workers, never on MainActor. The store applies defensive byte, entry, and response bounds, but it does not make token-policy decisions. PromptBudgeter alone selects whole index entries up to the memory allowance, renders the digest, adds it to the canonical system turn, and plans the complete request. Memory therefore never bypasses the context budget.

Memory has an application-wide hard off switch and a durable provider binding for loose chats and each Pen. A fresh profile defaults to enabled Markdown (local) memory. The default provider applies only to loose chats and new or not-yet-bound Pens. Each Pen can inherit that default, disable memory, or choose a provider explicitly. The first memory use freezes the effective provider for that chat scope. Changing the application default never silently reroutes an existing binding. A chat resolves exactly one provider route: local providers read Global notes for loose chats or the current Pen's notes for Pen chats, while Hindsight reads its one shared bank. GOAT does not silently union providers in one prompt. When memory is off, GOAT performs no memory reads, prompt injection, tool exposure, writes, feedback synthesis, or reflection. It does not delete data or forget the binding, so turning memory back on resumes the same store.

Launch providers:

  1. WikiMemoryStore / Markdown (local) (current default). Global notes live in ~/.goat/memory; every Pen owns an independent local Markdown store at ~/.goat/projects/<short-name>_<uuid>/memory (with notes directly in that directory and a MEMORY.md index). The name makes folders discoverable while the UUID remains the stable authority; historic folder names are retained. One fact per note uses name / description frontmatter and [[wikilinks]]. PromptBudgeter policy v2 deterministically renders at most 1,536 estimated tokens from its structured snapshot, preserving whole index entries and recording omissions. Model tools are memory_list, memory_read, memory_write, and memory_delete. Writes replace notes and the derived index atomically. Every path, file type, symlink, size, and schema check fails closed. The human surfaces are an in-app Memory browser and the folder itself. It is plain Markdown, editable in anything, and syncable by the user with whatever they trust.

  2. LLM Wiki (local) (separate launch provider). It uses local files and retains immutable source inputs alongside its curated wiki, schema, derived index, and maintenance log. Its lifecycle is explicitly ingest, query, update, and lint; a generated wiki page is not source-of-truth user content. It shares the local folder convention with Markdown (local), but provider bindings and schema versions remain distinct, and no automatic migration is permitted.

  3. HindsightMemoryStore (optional). A dedicated typed adapter that launches and owns the installed @vectorize-io/hindsight-coding-agents plugin process. MCP remains the plugin's private wire protocol, but the process is not registered as a generic GOAT MCP server, is not permission-gated as an arbitrary chat tool, and cannot be replaced by a same-named server. Setup records an explicit workspace and launches with HINDSIGHT_MCP_HARNESS=goat and HINDSIGHT_MCP_PROJECT_CWD set to that captured path. GOAT bounded-reads and validates ~/.hindsight/coding-agent.json before launch because the plugin otherwise falls back to cloud defaults when the file is malformed. GOAT always runs the plugin's own daemon-start.js entry point before mcp-server.js; the helper safely no-ops for non-daemon configurations, GOAT does not reimplement or silently install Hindsight, and a successful bootstrap process exit is not treated as service health. The managed client has bounded restart backoff. After every process generation it separately recomputes an installation fingerprint over Node, package metadata, and the two executed scripts, plus a contract fingerprint over initialized server identity and the exact eight schemas. It also revalidates diagnostics, sync status, bank, workspace, endpoint, and credential-match evidence. The configured service mode is a bounded-parsed settings label, not diagnosed identity, because the current diagnose result does not report it; GOAT never infers daemon mode from a loopback address. Settings locate and test the installed plugin and show its resolved bank, endpoint, configured mode, and health. A missing plugin, invalid config, installation or contract drift, changed bank, changed endpoint, or incomplete contract requires an explicit rebind. The adapter validates and forwards the current eight tools exactly:

    • hindsight_sync_status and hindsight_diagnose
    • hindsight_search_knowledge_pages, hindsight_list_knowledge_pages, and hindsight_read_knowledge_page
    • hindsight_reflect
    • hindsight_capture_initiative and hindsight_ingest_document

    The adapter exposes a structured knowledge-page roster to PromptBudgeter and uses hindsight_ingest_document for explicit GOAT memories and feedback. It decodes the MCP server's JSON text results and treats isError, truncation, malformed JSON, or an unexpected shape as failure. It does not call undocumented HTTP routes, translate operations back to the legacy raw API, or claim a delete capability that the MCP contract does not provide. Missing tools, a stopped process, an error result, or an unexpected response makes the Hindsight backend unavailable until it is healthy again. GOAT does not silently write the same memory into the wiki as a fallback.

    A coding-agent MCP process resolves exactly one bank when it starts, and none of the eight tools accepts a scope or bank argument. M6 therefore advertises Hindsight as one shared-bank backend rather than pretending it has Wiki's Global and per-Project isolation. Supporting several Hindsight scopes would require separately configured MCP processes and is outside MVP.

The package's automatic git and transcript ingestion belongs to supported coding-agent harnesses. GOAT's managed plugin process supplies the eight tools only: it does not seed git, retain GOAT conversations, or inject an automatic reflection. GOAT only promises actions it performs through the dedicated adapter. A separately configured ordinary Hindsight MCP server remains independent and is never mistaken for the memory provider.

The brain-icon Memory settings tab owns the master switch, provider health, the Global and new-Pen defaults, and per-Pen overrides. Provider changes never perform an automatic migration or union two stores. The switcher offers only two honest operations: resume a previously bound provider, or start fresh with a different provider while preserving the old data as inactive. Returning to an old provider resumes it. A future explicit export or copy can be one-way, but MVP does not claim round-trip fidelity between editable scoped Wiki notes and Hindsight's shared append-only bank. Selecting Hindsight is an explicit user-configured network choice under the Herd Guarantee; GOAT supplies no hidden default endpoint or credential.

The Wiki browser has List and Graph modes. Existing MarkdownUI remains the note-body renderer. The store's two-field frontmatter and generated index use a strict canonical codec rather than accepting general YAML or Markdown as authority. Graph mode uses a bounded recognizer for exact [[safe-note-slug]] extensions outside inline and fenced code; it does not attempt to implement a general Markdown parser. It performs layout away from MainActor and renders with native SwiftUI Canvas. M6 adds no graph or parser dependency. Providers without link or graph capability show the list without pretending to have a graph. Obsidian support is a future provider or explicit folder bridge, not another name for the default store; the default Wiki is already the flat-file memory style.

Consequences ​

  • Default memory honors pillar #1 with zero moving parts: no daemon, Docker, embeddings, or network.
  • Hindsight users get knowledge-page search and graph-backed reflection through the maintained coding-agent MCP contract. Service location and data handling remain their explicit configuration choice.
  • Disabling memory is reversible and non-destructive. Switching providers preserves old stores but does not merge, mirror, or migrate them automatically.
  • Native graph rendering keeps the dependency budget flat, but GOAT owns a small deterministic layout implementation and its performance tests.
  • Wiki recall is index+read (no semantic search). Accepted for MVP; the fix if it hurts is a local embedding pass (MLXEmbedders), parked.
  • Auto-reflect (a post-turn extraction into the active backend) is a prompt-quality risk. It is mitigated by a strict note schema, an opt-out toggle, bounded source text, visible writes, and full wiki editability. Hindsight's deep hindsight_reflect remains a distinct tool and is not treated as a write.
  • Capability differences are visible. Wiki notes can be edited and deleted; the current Hindsight MCP contract has no delete tool.

Alternatives considered ​

Vector DB in-app (rejected: complexity without the wiki's legibility), memory in SQLite (rejected: files-you-can-open is a feature, not an implementation detail), Hindsight-only (rejected: a service dependency as default violates "works out of the box"), a generic user-managed Hindsight MCP row (rejected: not first-class enough for provider identity and lifecycle guarantees), a bespoke Hindsight HTTP adapter (rejected: it would drift from the maintained coding-agent MCP contract), automatic cross-provider migration (rejected: lossy and unsafe), and a third-party graph package (rejected: native Canvas is sufficient for the bounded Wiki graph).