Memory: remember, cite, forget
In short Store the raw text, keep facts small, derive everything else, and make every belief point back to its source so it can be traced and forgotten. The design is built; two journeys pass, but a memory benchmark is still missing.Full verdict ↓
An agent that forgets is a chatbot, so memory was the first thing I built. It was also the thing I rebuilt most. Where I ended up is one rule: every belief the agent holds must point back to the words it came from.
The design
Four operations sit on top of that: retain (store a payload, extract facts), recall (find what is relevant), reflect (answer a question over what is stored) and mental models (a written answer about the user that is refreshed and cites its sources).
The journey
- A mining pipeline. Extract entities and rules from conversations, cluster them into a knowledge graph, consolidate in the background.
- Rebuilt on a published design. Retain, recall, reflect and mental models, following the Hindsight paper, running as durable workflows.
- The first pipeline is removed. Mining, clustering and consolidation were taken out in full once the rebuild replaced them.
- Raw text first. Store every payload as sent, give each fact a source, add a separate layer for conclusions.
- One provenance trail. Profiles, relations and observations all cite their sources in one table, and forget walks it.
- Realigned with the paper. Observations are recomputed per entity from all of its facts.
Decisions that held
| Decision | Why | The lesson behind it |
|---|---|---|
| Store the raw text as sent | Without it a belief cannot be traced to anything | Facts stored before this have no source, and there is no backfill because the text was discarded |
| Keep the base small, derive the rest | A summary is a conclusion, not a fact | A one-way rule: derived reads base, never the reverse, enforced by a test |
| One provenance table for every kind of record | Profiles, relations and observations all need the same trail | Two evidence tables turned out to be the same relation |
| The model proposes citations, code checks them | A model can cite a fact it never saw | Code accepts only ids that appeared in the evidence it was given |
| Three clocks: happened, told, stored | A fact told about 2019 and stored today is new to the system | I used the told time as a "new" marker in four places, so a fact told about 2019 would never have counted as new |
What already exists
I read the main systems before rebuilding. What each keeps of where a memory came from:
| System | What it stores | Source of a memory | History when it changes | Deletion |
|---|---|---|---|---|
| MemGPT / Letta | Evicted messages, a rolling summary, archival store | Only that messages are still stored | Summary overwritten | Not described |
| Mem0 | Facts the model adds, updates or deletes | No raw-text provenance described | Graph variant marks relations invalid | Model chooses DELETE |
| A-MEM | Linked notes that evolve | The original content stays on the note | Evolved note replaces the original | Not described |
| Zep | A temporal knowledge graph over raw episodes | Edges trace back to episodes | Edges are invalidated with timestamps | Not described |
| Hindsight | Four networks of facts and observations | Observations list their source fact ids (in the released code) | Facts can be invalidated | Deleting a document removes the observations built from it (in the released code) |
| Mine | Raw text, base facts, a derived layer | One table, for six kinds of record | Each conclusion supersedes with history | One operation walks the trail |
This is from my reading notes of the papers and, for Hindsight, its released code. I first claimed in my own draft that nobody had lineage or deletion. That was wrong: Hindsight's code has both for observations. What remains different is breadth, one trail covering profiles, relations and written answers, and history for conclusions.
Where it stands, and my verdict
| Question | Answer | |
|---|---|---|
| Does it remember across sessions? | Retain, recall, reflect and mental models run | yes |
| Can a belief be traced to its words? | Yes, for everything formed after the raw layer | yes |
| Can I forget something and what came from it? | One operation does it | yes |
| Does it help the agent decide better? | Two journeys pass (a preference recalled, a fact saved); no with/without comparison | thin |
| Do conclusions earn their cost? | No with/without comparison yet | unproven |
Verdict: the design is sound and still under-tested. The traceability and forgetting work as designed, and the journey suite now gives the memory layer a first signal: two scenarios, and the 35B passes all six samples (the thinking 8B misses one). That is not a memory benchmark. I still have no measurement that the derived layers improve what the agent does, which is the next thing I owe this component. Next: the toolbox, which is how that agent reaches anything outside itself.