> ## Content Index > Fetch the complete content index at: https://blog.dev32.io/llms.txt > Use this file to discover other available public pages before exploring further. # Building Long-Term Memory That Feels Natural - URL: https://blog.dev32.io/building-long-term-memory-that-feels-natural/ - Published: 2026-08-31T04:24:32.000Z - Updated: 2026-08-31T04:24:32.000Z - Description: I built long-term memory for a family voice assistant. The hard part was deciding what to trust, forget, retrieve, and leave unsaid. - Author: Kevin Ye - Tags: LLM Memory, AI > I built memory for a family voice assistant. Search was the easy part. Deciding what should survive, what could return, and when memory should stay quiet became the real system. I was not trying to make a coding agent more productive. I was building a voice assistant for my family. A tool can forget you when the session closes and still do its job. A family assistant cannot feel continuous if every conversation starts from nothing. I wanted a mention of skiing to bring back last week’s Tahoe plan. I wanted a dinner suggestion to account for an allergy without asking for the same information again. My first design was embarrassingly obvious: save the conversations and search them. Two later measurements broke that confidence. The real embedding model treated nonsense as relevant, and a full LoCoMo run showed plain BM25 beating my hybrid retriever. ## I wanted recall before the agent asked for it Most agent memory waits for the model to call a tool. That is useful, but it makes remembering an explicit action: the agent first has to notice a gap, then search for what it forgot. DeepMemory has three ways for the past to return: - A small `MEMORY.md` file enters every new session with durable facts worth their context cost. - **Spark** searches automatically from the opening utterance of each turn, before the first model call. - The model can deliberately search deeper history with `memory_recall`. In the skiing example, a new session already knows the small standing profile. When someone mentions renting skis, Spark gets one chance to find the Tahoe plan before the model answers. If the model needs more detail, it can deliberately search and open the source transcript. Spark adds no extra LLM call, although it still runs local embedding and index search. It returns at most three short snippets under a rough 250-token budget, and the turn continues without memory if search takes longer than 500 ms. ![DeepMemory keeps session history and Markdown as canonical sources, derives a rebuildable hybrid index, and returns bounded memory through automatic Spark or deliberate recall. Weak or unsafe recall stays out of the agent context.](https://blog.dev32.io/content/images/2026/08/deep-memory-architecture.png) I also did not want a vector database to own the family history. DeepMemory keeps Markdown and append-only sessions as the durable record. A local SQLite keyword-and-vector index is disposable. Delete it, change the embedding model, or rebuild the schema; the memory still exists in files a person can inspect and edit. That decision became more important every time the retrieval layer was wrong. ## The first threshold remembered almost everything Spark originally accepted results above `0.60`. The number looked reasonable, and unit tests with controlled fake vectors passed. The real multilingual E5 model did not share that intuition. Against a populated memory scope, unrelated and even nonsense queries often scored around `0.74–0.79`. Nearly every turn found something. The model sometimes ignored the irrelevant snippet, so the product looked less broken than the retrieval logs. We raised the threshold to `0.78` after a small live diagnostic. That reduced weak recall, but it was not a clean calibration: relevant and irrelevant scores still overlapped. Later, the benchmark showed that the threshold also cost recall. This was my first useful correction. “Return nothing” could not be a design principle only. It needed to be reachable in the real score distribution of the model and corpus we shipped. ## Old text becomes trusted surprisingly quickly Automatic recall creates a path from an old conversation into a future model prompt. Nightly consolidation makes that path longer, not safer. A session can contain user speech, assistant output, tool results, and text fetched from the web. If a malicious instruction enters through a tool result, gets summarized overnight, and returns later as memory, it can look like trusted background knowledge. Passing through another LLM does not clean its source. DeepMemory keeps provenance on recalled and consolidated material. Tool-derived text stays tool-derived after Dreamer—the nightly consolidator—turns a session into an episode or candidate fact. Notes that enter every prompt are scanned before they are written. Direct edits on disk are checked before rendering or indexing. Recalled snippets are treated as untrusted context before they can affect the model. Our first scanner policy was too blunt. It rejected suspicious and hostile text everywhere. Normal summaries often contain phrases such as “the user confirmed,” which can resemble context-manipulation language even when they are accurate. We changed the policy by destination. Standing notes remain strict because they enter the system prompt. Journals may retain suspicious text, but they are screened again if recalled later. The fix came from following the text back into model context instead of applying the harshest rule to every file. ## The most worrying bugs still returned success Dreamer began as “summarize new sessions every night.” It ended up behaving like a transaction. For each user, it snapshots a fixed window of new sessions and generates candidate facts. Before changing the notes, it archives the old version. It writes the new files, synchronizes the index, and advances its checkpoint last. If the process stops halfway through, the next run sees the same input. Deterministic IDs and idempotent writes let it converge instead of duplicating work. The ordering matters because the canonical file and the search index can disagree. Review found that a Dreamer rewrite could make `MEMORY.md` correct while leaving the previous section active in the index. Spark could retrieve a fact that no longer existed in the file. The repair was to route Dreamer through the same index-sync path as interactive writes. Another review found that the gateway sent time ranges as `from/to` while the Python service expected `start/end`. Searches and purges returned success while silently ignoring the range. The HTTP response still said success. A missing Dreamer checkpoint had a different cost. The first implementation treated it as overdue work, which could send an existing household’s entire history through a paid provider on first enable. Now a new checkpoint starts at the current session head. Only a previously valid but stale checkpoint triggers catch-up. These failures changed how I thought about forgetting. Dreamer can rewrite, supersede, or mark facts stale, but it cannot hard-delete history. Old index entries become inactive. Archives and source sessions remain available for audit and repair. Destructive purge stays an explicit operator action. ## Similarity cannot decide who is allowed to remember A family assistant also has a problem that most single-user memory demos can ignore: the closest semantic match may belong to the wrong person. The gateway authorizes scopes before retrieval. A signed-in person can search their private memory and, where allowed, a shared household scope. Shared entries carry authorship and audience, and child sessions filter adult-only material before it reaches either the standing prompt or search results. The index service sees opaque scope IDs, not arbitrary filesystem paths. That still does not solve the room. A phone identifies the person holding it. A kitchen speaker may say private information in front of anyone nearby. DeepMemory handles authenticated data boundaries; it does not pretend those boundaries identify a physical audience. ## Then the benchmark disagreed with us After the implementation worked across the local stack, I wanted a number I could defend. That became its own project. [VoiceMem](https://github.com/xzf-thu/VoiceMem?ref=blog.dev32.io) reports 91.2% on [LoCoMo](https://github.com/snap-research/locomo?ref=blog.dev32.io), but its public evaluator describes 152 questions while the current official dataset has 1,986\. Other systems mix retrieval with answer models, judges, different ingestion rules, and much larger context budgets. I could not put those scores beside DeepMemory and call the comparison fair. So the first run asked a narrower question: when each LoCoMo session is one indexed entry, does DeepMemory retrieve the session containing the annotated evidence? The harness used the real local search stack. It covered all 272 sessions and 1,986 questions. Four questions with no evidence were excluded, and nine malformed evidence citations were corrected under a reviewed policy. No answer model or judge was involved. Recall@5 asks how many annotated evidence sessions appeared among the first five results. MRR rewards putting the first relevant session earlier. | Index mode | Evidence-session Recall@5 | MRR | | ------------------------- | ------------------------- | ---------- | | Hybrid FTS + vector + RRF | 70.26% | 64.13% | | Vector only | 59.55% | 51.66% | | FTS5/BM25 only | **73.62%** | **71.33%** | Hybrid beat vector-only by 10.71 points. Then FTS-only beat hybrid by 3.36 points and produced a much better ranking. I expected the hybrid system to be the safe winner. It was not. LoCoMo questions often contain exact names, activities, and dates, so lexical search has strong signals. The result is also consistent with unweighted rank fusion weakening a decisive keyword match when the vector order disagrees. We have not isolated that cause yet. The closest open-source retrieval comparison I found was [Engram](https://github.com/Nitin-Gupta1109/engram?ref=blog.dev32.io). It reports a binary question-level Recall@5 of 93.9% with BGE-large embeddings, finer chunks, speaker-name injection, and a cross-encoder reranker. As a rough diagnostic, I recomputed the same binary “did any evidence session appear?” criterion from our raw run: 87.08% for FTS-only and 82.14% for hybrid. Those are not our primary metric, and I have not reproduced Engram under our evidence normalization or retrieval setup. Its repository-reported score is about 6.9 points above our best mode, but that is a directional gap, not an aligned system ranking. DeepMemory’s retrieval is entirely local. The current fusion still needs work. The benchmark has another large limitation: the product normally searches Dreamer-consolidated notes, while this run indexed raw sessions directly. It measures the index, not whether Dreamer kept the right facts, whether an answer model used them, or whether a conversation felt more natural. ## What I got from building it Focused tests exercised the real MLX, SQLite, FTS5, sqlite-vec, HTTP, authentication, search, and purge paths. Reviews found the ineffective threshold, stale index entries, unsafe first-run consolidation, and the time-range mismatch. The benchmark found a ranking weakness the implementation tests could not. I do not think DeepMemory is finished, and the benchmark makes that hard to pretend. A local reranker, better fusion, finer retrieval units, and a real end-to-end memory benchmark are all still open work. I enjoyed the project more after the failures started. Each one took some magic out of the word “memory.” What remained was old text crossing back into the present, carrying an owner, a source, a lifecycle, and a permission check. I still want the Tahoe plan to return when someone mentions skiing. I just no longer think the impressive part is retrieving it. The harder test is whether it returns for the right person, from a source we can trust, without reviving an obsolete plan—and whether the assistant stays quiet when the match is weak. DeepMemory can enforce much of that now. I still do not know whether my family will experience it as natural memory. A retrieval benchmark cannot answer that.