Giving Our AI Agent a Memory That Doesn't Grow Forever

Giving an AI agent memory sounds like a solved problem the moment you say the word "database." Save what the model says is worth keeping, pull it back in on a future turn, done. That version works fine for a demo and starts breaking down the moment the memory store actually grows past a handful of entries, which is exactly the point most write-ups on agent memory stop talking about it. The interesting engineering isn't storing a fact. It's what happens on the hundredth fact, and the thousandth.

The two tools: remember and recall

Every product built on our harness gets two memory tools automatically, not as something a given Skill has to opt into. Remember lets the model save something it's decided is worth keeping, a preference, a fact about the person it's talking to, the state of something ongoing. Recall lets it search that store later and pull relevant entries back into context. Because these are harness-level infrastructure rather than a per-product feature, every AI agent we build gets the same memory behavior, and any improvement to how memory works benefits every product at once, not just the one it happened to be built for.

We didn't just assume this works because the code looks right. We verified it end to end: a fact saved through remember in one conversation was later retrieved through recall in a completely separate, brand-new session that shared no prior history at all. That distinction matters more than it sounds like it should. A lot of what gets marketed as "AI memory" is really just a longer context window within a single conversation, which forgets everything the moment that conversation ends. Genuine cross-session memory, the kind that lets an agent pick up a relationship with someone it hasn't talked to in a week, is a different and harder thing to get right, and it's the thing we actually tested for.

The problem nobody mentions: memory that never stops growing

Here's what happens next, past the point most demos stop. An agent that's genuinely useful over weeks and months accumulates a lot of saved facts, and a naive memory system has exactly two ways to handle that growth, both bad. Keep everything forever, and either every recall search has to sift through an ever-larger haystack, or worse, memory content gets stuffed directly into the system prompt and grows the token cost of every single turn indefinitely, even the ones that have nothing to do with most of what's stored. Or prune aggressively on some fixed schedule, delete anything past a certain age or count, and you risk silently throwing away something a future conversation genuinely needed, with no way to know which discarded fact was going to matter until it's already gone.

Neither option is acceptable for something meant to be genuinely long-lived. An agent whose costs quietly creep upward every month as memory grows isn't sustainable. An agent that randomly forgets things isn't trustworthy. We needed a third option: memory that stays useful and bounded without losing information.

How we compact memory without losing it

The answer we built is a compactor that runs automatically, checking after every new memory is saved whether the total store has grown past a rough token budget. Below that budget, nothing happens, the store just grows normally. Cross it, and the compactor kicks in: it hands the model every fact currently saved, one per line, and asks it to merge, deduplicate, and summarize them into a smaller set of entries that together still contain every distinct piece of information. The instruction is explicit about the goal: say the same things more densely, don't drop anything a future conversation might actually need, and combine clearly related facts into one entry only where doing so loses nothing.

This is deliberately invoked inline, right after the memory-changing action that might have pushed the store over budget, rather than on some separate schedule running once a day or once a week. That matters because it means the store never drifts far past its intended budget between checks; it's kept close to the line continuously; rather than allowed to balloon for a day and then get yanked back down all at once.

The safety check: refusing a bad compaction

Handing memory compression to the same model that's being compressed introduces an obvious risk: what if it does a bad job? A summarization pass could come back malformed, empty, or, worse, technically valid but having silently dropped something that mattered. We treat a failed or suspicious compaction as strictly worse than no compaction at all, and built the safety check around that principle directly. If the model's response can't be parsed as the expected structure, the compaction is discarded and memory is left exactly as it was. If it comes back empty, discarded. And if the compacted result isn't actually smaller than what went in, meaning the summarization didn't accomplish anything, it's discarded too, since a compaction that doesn't compact anything isn't worth the risk of having asked at all.

The practical effect is a system that fails safe. On a good day, memory quietly gets denser and stays within budget without anyone noticing it happened. On a bad day, where the summarization pass genuinely goes wrong, memory just... doesn't change. Nothing is lost, which is the only acceptable behavior for a failure mode in something explicitly meant to preserve information over time.

Why compress with a model instead of a simpler algorithm

It would be simpler, and cheaper per call, to compact memory with a plain deduplication algorithm, strip exact repeats, maybe fuzzy-match near-identical strings. We use the model instead because the kind of redundancy that actually accumulates in a long-lived memory store isn't usually exact duplication, it's semantic overlap phrased differently across several separate entries: a preference stated once, restated with slightly different wording a month later, a fact updated rather than replaced so the old and new versions both linger. Catching that reliably needs something that understands meaning, not just string similarity, which is exactly what the model already sitting in the harness is good at, and it's a task well suited to a smaller, cheaper model than whatever's handling the agent's actual conversation, since compression doesn't need the same reasoning depth as a live response.

A concrete example of compaction in action

Picture an agent that's saved, across several separate conversations over a few weeks, entries like "prefers to be contacted in the morning," "mentioned they're usually free before 11am," and "asked not to be messaged during lunch." Three entries, one underlying preference, phrased three different ways because they were saved on three different occasions without the model cross-referencing what it had already stored. Once the store crosses its budget, the compactor hands all of it to the model and gets back something closer to a single entry: "prefers contact in the morning, before 11am; avoid messaging during lunch." Same information, a third of the storage, nothing lost.

That's a small example, but it's representative of what actually accumulates in a real memory store over time: the same underlying fact, restated slightly differently across separate conversations, rather than genuinely new information every time. Catching and merging that pattern is exactly what keeps the store's size roughly proportional to how much has actually been learned, rather than to how many times it's been mentioned.

What we'd still improve

It's worth being honest about the current limits rather than presenting this as a finished system. Recall today works by matching text against what's stored, which is straightforward and reliable for smaller memory stores but will eventually miss a relevant fact that's phrased very differently from how it was originally saved, a semantically related memory that doesn't share enough literal wording to surface in a plain text match. Moving that to something more semantic, comparing meaning rather than matching text, is the natural next step as any given agent's memory graph grows large enough for that gap to actually start costing something. We'd rather ship the simpler version that's honest about its limits than a more complex one we haven't verified actually works better in practice.

What this actually means for a business relying on it

The point of building this properly isn't abstract. An agent that's been genuinely useful for six months, remembering a client's preferences, the state of an ongoing project, prior decisions that shouldn't need re-explaining, is exactly the agent whose memory store has had six months to grow. Without compaction, that's either a slowly rising token bill built into every single turn, memory content or not relevant to what's being discussed right now, or a system someone eventually has to manually prune and risk losing something in the process. With it, the store stays lean on its own, the cost of memory stays roughly flat over time instead of climbing, and nobody has to notice it's happening for it to keep working correctly. That's the actual bar for "memory" being a feature worth trusting rather than a feature that looks good in month one and becomes a liability in month six.