How We Watch Our Own AI Agent for Signs It's Gone Off the Rails

"It ran without crashing" and "it's actually behaving well" are different questions, and it's easy to only ever check the first one. A tool call that returns a result, no exception thrown, no error in the logs, looks identical to a tool call that succeeded whether the underlying behavior was actually sound or quietly going sideways. An agent can run for weeks without a single crash while slowly accumulating bad habits nobody's watching for: a tool that keeps failing in a way that's being silently swallowed, a memory search that never finds anything because nothing useful was ever actually saved, a handful of near-duplicate database tables from three separate attempts at the same task. None of that shows up as an error. All of it is worth catching. So we built a pass that specifically goes looking for it.

This is the piece we get asked about most directly, in plainer words: "how do you actually know it's working?" A demo answers that question for exactly one run, watched by exactly one person, on inputs chosen because they'll behave well. It says nothing about run four hundred, unwatched, on an input nobody thought to test. Answering the real question needs something that keeps checking long after the demo is over, which is what the rest of this piece actually describes.

Why "it didn't crash" is the wrong bar

Every part of the harness we've written about, the approval gate, memory, the agent's own database, already persists a detailed record of what actually happened: every tool call and its result, every run and how it ended, every memory operation. That data existing is necessary but not sufficient. Nobody reliably notices a pattern by occasionally skimming raw logs, and "nobody noticed anything wrong" is a weak substitute for "we checked and nothing's wrong." We wanted something that actively reads that record and surfaces a problem, rather than passively hoping a human happens to spot one.

What we actually check

The eval pass runs four distinct checks over the harness's own persisted history, each aimed at a specific way an agent can be technically functioning while actually going wrong.

• Duplicate tables: scans the names of every table the agent has created, in both its own private database and the harness's data layer, and flags groups of names that look like the same underlying thing written slightly differently. This is the direct check on the table-sprawl problem we've written about separately, catching it after the fact even when the agent's own instructions to check first didn't fully prevent it.
• Failed tool calls: looks back over recent tool calls for ones that came back with an error in their result, and groups them by which tool failed and how often. A tool that's failed once might just be a fluke. A tool that's failed five times recently is a pattern worth a human's attention, not something to keep quietly retrying.
• Empty recalls: checks how often a memory search has come back with nothing found recently. An occasional empty recall is normal, sometimes there's genuinely nothing relevant saved yet. A cluster of them can point to something worth investigating: memory that isn't being written the way it should be, or an agent repeatedly asking for context that was never actually captured in the first place.
• Stalled and failed runs: looks at recent runs and flags any that ended in an explicit failure, plus any that have been sitting in a "generating" or "waiting on approval" state for far longer than a real turn ever takes, the specific signature of a run that's silently hung rather than cleanly failed or finished.

Read-only, on demand, not on the hot path

This pass doesn't touch or change anything, it only reads back state that's already being persisted regardless of whether the eval pass exists at all. That's a deliberate design choice, not an accident: it means running it carries no risk of altering agent behavior or corrupting anything it's inspecting, and it's meant to run on demand or on a schedule, not injected into the hot path of every single turn. Checking for accumulated patterns is inherently a look-back-over-time operation; running it after every single message would add latency and cost to every turn for a check that's only meaningful in aggregate anyway. It belongs on a schedule or triggered when something already feels off, not baked into the critical path of a live response.

What a finding actually looks like

Each check produces a structured finding rather than a vague alert: which check flagged it, a severity of either info or warning, a one-line human-readable summary, and the specific detail behind it, the actual tool name and sample errors for a failed-tool finding, the actual table names for a duplicate-table finding, the actual run IDs for a stalled-run finding. That specificity is the entire point. "Something might be wrong with tool X" is a shrug. "'send_email' failed 5 times recently, here are 3 of the actual errors" is something a person can act on immediately, in the same afternoon it was flagged, not a mystery to go spelunking through raw logs to reconstruct.

A finding, start to finish

It helps to see how one of these checks plays out rather than just take the categories on faith. Say an agent has been running a scheduling Skill for a few weeks, and a webhook it depends on to check calendar availability starts intermittently timing out, something nobody's actively watching for because each individual failure just looks like one turn that took slightly longer or came back with a polite "I couldn't check that right now." Nothing crashes. Nothing pages anyone. The next time the eval pass runs, the failed-tool-calls check counts how many times that specific tool has errored recently, finds it's happened repeatedly, and produces a finding naming the exact tool, the count, and a sample of the actual error messages behind it.

That finding is the difference between a slow-building problem staying invisible until enough people notice missed bookings to complain, and someone on the team seeing a specific, actionable line item, this tool, this many failures, these error messages, well before it's cost anything visible. The check didn't need to know in advance that a webhook would be the failure point. It just needed to be watching the same signal, tool call outcomes, that already gets recorded regardless.

Why severity has just two levels, not five

Each finding is marked either info or warning, deliberately not a longer scale of low, medium, high, critical, and so on. A finer-grained scale sounds more sophisticated but tends to invite second-guessing about exactly where a given finding sits on it, time better spent deciding whether to act. Info means something worth being aware of that isn't necessarily broken, a handful of empty recalls, for instance, which can be entirely normal. Warning means something that's actually a plausible sign of a real problem, worth someone's attention this week rather than filed away. Keeping the distinction binary keeps the output usable at a glance instead of requiring its own interpretation layer on top.

This is also how the harness catches its own bugs

This kind of after-the-fact auditing isn't purely theoretical for us, it's exactly the category of thing that's already caught real bugs during development. Persisting a full, checkable record of what an agent actually did, not just what it claims in its own reply, is precisely what makes it possible to notice when something the harness itself should have handled correctly, quietly wasn't, instead of only finding out once a client or a teammate happens to notice a symptom downstream. An audit trail nobody reads is barely better than no audit trail at all. The value only shows up once something is actually reading it and knows what a problem looks like.

Why we don't treat this as optional

We've written before about a broader habit: testing AI "skills" before trusting them with client work, on the basis that an unproven skill is worse than no skill at all, since it's easy to assume something works because it reads reasonably and never actually check. Self-monitoring is the same principle applied to a live, running system instead of a piece of prompt configuration. A prompt can be reviewed once before launch and left alone. An agent that runs continuously accumulates behavior over time in a way a one-time review can't catch, which is exactly why the check needs to keep running for as long as the agent does, not just at launch.

Why this matters more than which model is underneath

It's tempting to treat "which model" as the main lever for AI reliability, upgrade to a smarter one and problems go away. In practice, a more capable model with nobody watching what it actually does over time is still a system nobody can vouch for with confidence past the first week. A modest model with a genuine self-monitoring layer catching duplicate tables, repeated tool failures, and stalled runs is a system you can actually stand behind months in, because you have a mechanism for finding out when something's off instead of waiting for it to surface as a customer complaint. That's the actual answer to "how do you evaluate your AI agent": not a one-time benchmark run before launch, but a standing, structural check that keeps running for as long as the agent does, and tells you exactly what to look at the moment something drifts.