Building the Harness: The Reusable Engine Behind Every AI Agent We Ship

Ask most vendors what's behind their "AI agent" and you'll get an answer that's really just the name of a model. That's not a complete answer, and it's not really an honest one either. A model is raw reasoning capacity, nothing more. What decides whether an AI agent is something you can actually trust with real work is everything built around that model: the rules about what it's allowed to do without asking first, the memory that lets it pick up where it left off, the checks that catch it when it starts behaving strangely, and the plumbing that makes all of that consistent across every product it powers. We call that surrounding system the harness, and this is the story of why we built our own instead of wrapping a chat window around whichever model happened to be popular that month.

The problem: every new AI idea meant rebuilding the same plumbing

We build AI features for clients and for ourselves, and for a while, each one started from a blank page. A new AI-powered feature meant a new way of calling a model, a new (or nonexistent) way of remembering anything across turns, and a new, usually thinner, way of handling what happens when a tool call fails or the model asks to do something it probably shouldn't do unsupervised. None of that is glamorous work, so it tended to get built just well enough to demo, which is a very different bar from built well enough to leave running unattended.

The actual symptoms showed up exactly where you'd expect: a chat session that couldn't be resumed after a page refresh, a model that silently lost track of tool calls it had made earlier in the same conversation, no consistent record of what an agent had actually done versus what it merely claimed to have done in its own reply. Individually, each of those is a small bug. Together, across several products each with their own version of the same missing plumbing, they're the reason a lot of "AI agents" work fine in a five-minute demo and quietly misbehave the first time someone actually depends on one. We wrote about that gap in general terms in a separate piece on why most AI agents break the moment you rely on them; building our own harness was us taking our own advice.

The core idea: an AI agent is not a model. It's AI Agent = LLM + Harness.

Once you separate those two things, a lot of decisions get simpler. The model is talent: it can reason about an ambiguous request, draft a reply, decide which tool would help. It has no memory of its own, no sense of what it's allowed to touch, and no way to tell you when something's gone wrong unless you built a system around it that watches for that. The harness is that system. It's the part that decides which tools the model can even see this turn, what happens when it asks for one it needs a human to approve first, what gets remembered after the conversation ends, and what a monitoring pass should flag as a warning sign after the fact.

The analogy we keep coming back to internally: think of the AI agent, the thing a customer or a teammate actually talks to, as a right hand man. Capable, always on, works from the same playbook every time, doesn't forget what you told it last week, and knows exactly which few things it's allowed to do without checking in first. None of that comes from the talent alone. A brilliant hire with no onboarding, no defined authority, and no manager checking in is a liability, not an asset, no matter how capable they are individually. What makes a right hand man trustworthy enough to actually hand real work to is the system they're operating inside, not the raw talent underneath it. Same logic applies to an AI agent, just with a harness standing in for the onboarding, the authority limits, and the manager.

That's also why we built the harness to be usable like software, not like a one-off integration you rebuild every time you have a new AI idea. You don't hand-roll a new operating system for every app you write; you write the app on top of one you trust. We wanted the same relationship between our AI products and the thing running underneath them: define a new Skill, a system prompt plus a scoped set of tools, and you have a new AI agent product, without touching the reasoning loop, the approval logic, or the memory system underneath it.

What we actually built

Structurally, it's one backend and many thin frontends. A single FastAPI service owns the reasoning loop, the tool and skill registry, and conversation history in a database. Every product we ship, a client-facing assistant, an internal ops tool, this website's own future AI surfaces, is a Next.js app that talks to that one backend over a standard streaming chat protocol. The frontend never talks to a model directly. It talks to the harness, and the harness talks to the model.

What actually differentiates one product from another isn't a forked copy of the backend, it's a Skill: a named bundle of a system prompt and a specific list of tools that skill is allowed to use. A customer-support agent and an internal scheduling assistant can run on the exact same reasoning loop, the exact same approval logic, the exact same memory system, and differ only in what their Skill tells them to do and what they're allowed to touch. That's the entire point of separating the two: the expensive, hard-to-get-right engineering lives in one place and gets reused, while the part that's genuinely specific to a given product, its job description, stays small and easy to change.

• A reasoning loop that calls the model, hands it a scoped set of tools, and keeps looping until it has a final answer or needs a human
• A tool interface with validated, typed inputs, so a tool either receives well-formed data or fails loudly before it ever runs, not silently on bad input
• A persistence layer that records every message and tool call incrementally, so a mid-stream crash never loses history that already reached the client
• A persona, one identity across every product, plus a memory graph the agent can write to and search across sessions, not just within one conversation
• A private, sandboxed database the agent can create its own tables in for a task, completely separate from the harness's own conversation storage
• A self-monitoring pass that reads the agent's own audit trail on a schedule and flags patterns that suggest something's actually wrong, not just whether a call technically succeeded

How a request actually moves through the system

Walk through what happens when someone sends a message. The frontend hands it to the harness along with which Skill is active for that product. The harness loads that session's history from the database, not from anything the client claims about it, since the database is the only source of truth for what actually happened. It builds a system prompt from three layers stacked together: the harness's own persistent identity, standing instructions about when and how to use memory, and the active Skill's own job description. Then it calls the model with that context and the Skill's scoped list of tools, and starts streaming the response back token by token as it's generated, the same live-typing feel you'd expect from any modern chat product.

If the model asks to use a tool, the harness doesn't just run it. It checks whether that specific tool is one that's allowed to execute on its own or one that needs a human to say yes first, a distinction we set per tool, not globally. Nearly everything defaults to requiring approval; a small, deliberate set of read-only tools, ones that can only look something up and can't change anything, are the exceptions, because requiring a human to approve every single lookup would make an agent unusable without actually making it safer. Once a call is approved and executed, the harness records both the call and its result, then loops back into the model with that result added to the conversation, and keeps going until the model has a final answer with nothing left pending.

A few details in that loop exist specifically because we found real failure modes while building it, not because they sounded like good ideas in the abstract. A session can only have one turn actually generating at a time, so a second message sent while the first is still in flight surfaces as "still working on it" instead of quietly starting a second, independent run that piles up alongside the first. Approving a tool call twice, say a retry after a flaky connection, replays the already-recorded result instead of executing the action again, so a network hiccup can never turn into a duplicate charge, a duplicate message, or a duplicate anything. And if a tool call is left waiting on approval and the conversation is picked back up later, possibly by a different request entirely, the harness reattaches to the exact run it paused, rebuilding the model's context straight from what's actually stored in the database rather than trusting anything the client sends about what supposedly happened.

Memory that doesn't grow without bound

Two tools, remember and recall, are available to every Skill automatically, not opted into per product. The model decides on its own what's worth keeping and goes looking for it later, and we've verified this genuinely works across sessions, not just within a single long conversation: a fact saved in one session was later retrieved from a brand-new session that shared no prior history at all.

The part that took more thought was what happens once that memory store actually grows. A naive version of long-term memory either grows forever, quietly eating more and more of the context budget on every single turn, or gets brutally pruned and loses things a future conversation genuinely needed. We built a compactor that runs automatically once the store crosses a rough token budget: it asks the model to merge, deduplicate, and summarize everything it's saved into a smaller set of entries that still contain every distinct fact, denser, not thinner. And because an LLM summarizing its own memory can, in principle, produce something broken or badly compressed, the compactor refuses to apply a result that came back empty, malformed, or that didn't actually shrink the store. A failed compaction just leaves memory exactly as it was rather than risking the alternative: an agent that quietly starts forgetting things because a summarization pass went wrong.

An agent with its own database, safely

Some tasks genuinely need the agent to keep structured state across a task, a running list, a set of records it's building up over several turns, not just prose memory. Rather than letting an agent anywhere near production data for that, we gave it its own private SQLite database, a completely separate file from the harness's own conversation and session storage, that it can create tables in and read from and write to freely. Reads are unrestricted and don't require approval, since a SELECT statement can't damage anything and requiring a human to bless every lookup would make the tool pointless. Writes, creating a table, inserting a row, dropping something, go through the same approval gate as any other consequential action. The tool's own instructions explicitly tell the agent to check what tables already exist before creating a new one, so it doesn't quietly accumulate five near-duplicate tables for the same data under slightly different names, and every query goes through parameterized values rather than string-built SQL, the same discipline you'd want from a human-written backend.

How we evaluate the harness before we trust it with anything

We take the same position here that we take in a separate piece about testing AI skills before they touch client work: an AI system that hasn't been checked isn't proven, it's just unrefuted. That shows up at two different levels for the harness itself.

At the code level, an automated test suite drives the entire reasoning loop and its streaming protocol against a fake, scripted model, no live API calls, no cost, no network dependency, so basic correctness is checked on every change. That suite specifically exercises the failure modes described above: that an approved tool call only ever executes once even if approval is submitted twice, that a run correctly reattaches to the same paused state after approval instead of starting fresh, that a run gets marked failed rather than hanging forever when something throws partway through, and that a session genuinely rejects a second message while the first is still generating instead of silently starting a parallel run.

At the behavior level, once the agent is actually running, we don't just assume it's staying on the rails, we check. A self-monitoring pass reads back the audit trail the harness already keeps, every tool call, every run and its outcome, and flags patterns that suggest something's actually wrong: tables that look like near-duplicates of each other under slightly different names, a tool that's been failing repeatedly, memory lookups that keep coming back empty, runs that failed outright or have been stuck mid-turn for far longer than a real response ever takes. None of that requires a human to comb through raw logs looking for trouble. It surfaces the trouble.

What we haven't built yet, on purpose

We'd rather say plainly what's still ahead than pretend a skeleton is more finished than it is. Every tool today runs with the same permissions as the backend process itself, fine while every tool is one we've written and reviewed ourselves, not fine the moment that stops being true, so sandboxing tool execution is next before anything less trusted runs through it. Authentication and rate limiting aren't wired in yet either, deliberately deferred while the focus has been proving the harness itself locally, but both are on the list before any product built on it goes fully public. And memory search today is straightforward keyword matching; upgrading that to something more semantic is the natural next step as the memory graph grows past what keyword matching can reliably find.

Why this matters beyond our own engineering

The practical payoff of building it this way is that a new AI product for a client stops being a new backend project. It's a new Skill, a job description and a scoped tool list, sitting on top of a reasoning loop, an approval system, a memory system, and a self-monitoring pass that already exist and are already proven. A bug fix or a safety improvement made once benefits every product running on the harness, instead of needing to be re-discovered and re-patched separately in each one. That's the actual difference between an AI feature that was bolted on for a demo and one that was built to be depended on, and it's the standard we're holding our own products to before we'd ever recommend it to a client's.