Where Langfuse and Harbor Fit Into How We Evaluate Our Own AI Agents

We've written before about the self-monitoring pass we built to watch our own AI agent: a read-only check that scans everything the harness already persists, tool calls, run outcomes, memory recalls, looking for the specific patterns that mean something's quietly gone wrong even though nothing crashed. It's genuinely useful, and it caught real problems during development. It was also never built to answer two other questions we kept running into: was a given answer actually correct, and is it safe to let a change near a client before someone finds out the hard way. Those are different problems from operational drift, and rather than keep bolting custom checks onto our own eval pass, we're wiring in two open-source projects built specifically for them: Langfuse and Harbor.

What our own eval pass was never built to answer

The self-monitoring pass is deliberately narrow. It counts things: how many times a tool failed, how many memory searches came back empty, how many runs stalled instead of finishing. That's exactly the right shape for catching structural drift, but it has no opinion on content. A tool call that succeeded and returned a perfectly well-formed result can still have been the wrong tool call, or the right tool call followed by a reply that got the client's policy wrong, misread what they were asking, or answered a different question than the one in front of it. None of that shows up as a failure in any log. It shows up as a bad answer that happened to execute cleanly, and our structural checks have no way to see it, because they were never reading for meaning in the first place.

There's a second blind spot that's less about individual answers and more about timing. Everything the self-monitoring pass catches, it catches after the fact, on a run that already happened, potentially one a client already saw. It's an after-the-fact audit, not a gate. Nothing about it stops a bad change from shipping in the first place; it just makes sure we find out reasonably quickly once it has.

Two different questions, not one

It's worth being precise about the two gaps separately, because they call for different tools. "Was this specific answer, from this specific conversation, actually good" is a live-traffic quality question, and answering it means grading real output against real context, at the pace conversations actually happen. "Is this change to a Skill, a prompt, or the underlying model safe to ship" is a pre-deployment safety question, and answering it means running a candidate change against a fixed set of scenarios, repeatedly, before a client ever sees it. We looked at building both ourselves and decided against it, for the same reason we've written about not reinventing things that are already solved well: tracing infrastructure, LLM-as-judge scoring, and distributed benchmark running are each their own multi-month project, and none of it is the part of our harness that's actually specific to us. The part worth our own engineering time is the approval gate, the sandboxing, the Skill boundaries. The eval tooling around it doesn't need to be homegrown to be trustworthy.

Langfuse: putting a judge on the loop we're already recording

Langfuse is an open-source LLM engineering platform built around exactly the signal our harness already produces: a detailed, step-by-step record of what an agent did during a run. Rather than build our own tracing UI on top of that record, we're feeding it into Langfuse directly, which turns each run into a structured trace, the model calls, the tool calls, the intermediate steps, viewable end to end instead of reconstructed by hand from raw logs. That alone is useful for debugging. The part that actually closes our quality gap is what sits on top of it: LLM-as-judge evaluation, where a separate model scores a trace against a specific rubric, did the answer stay within the client's stated policy, did it use the retrieved context correctly, did it pick the right Skill for the request, rather than a human having to read every conversation to find the ones that went sideways.

The other half of Langfuse we're leaning on is datasets: a fixed, versioned set of real conversations pulled from a Skill's own history, which we can re-run against a new prompt or a swapped model and compare scores against the previous version before deciding anything ships. That's the same instinct behind "run it more than once before you trust it" from how we test skills generally, just formalized into something repeatable instead of a few manual spot-checks. Because Langfuse is genuinely self-hostable, a Docker Compose setup gets a working instance running locally, we're keeping client conversation data on infrastructure we control rather than sending transcripts to a third-party SaaS product by default, the same preference that's shaped other infrastructure decisions we've made.

Harbor: proving a change is safe before it goes near a client

Harbor comes from the Terminal-Bench team and describes itself plainly as a framework for evaluating and improving agents: point it at a containerized, reproducible task environment, run one or more agents through it, and get scored pass/fail results back instead of one anecdotal run someone happened to watch. It's built and best known for benchmarking coding agents like Claude Code, OpenHands, and Codex CLI against standardized terminal-based tasks, and it can scale that across thousands of environments in parallel using cloud providers like Daytona or Modal rather than running everything on one machine.

Our own agent isn't a coding agent, so we're not pointing it at Harbor's built-in benchmark suite. What we're actually adopting is the underlying pattern: define a task as a reproducible, containerized environment with a clear pass/fail condition, then run a candidate change against a whole set of those tasks, many times each, before it ships. We're building our own benchmark tasks this way, a fixed set of scheduling scenarios, a fixed set of the kinds of customer questions a given Skill actually gets, and running a new prompt or a new model against all of them in parallel rather than eyeballing a handful of manual runs. That's the pre-deployment gate our self-monitoring pass was never positioned to be, catching a regression before a client ever becomes the first person to find it.

How the three layers actually stack

None of these three replace each other, because each one is watching for a different failure at a different point in time. Harbor runs before a change ships, against fixed scenarios, and answers whether a candidate change is safe to release at all. Langfuse runs on live traffic and on regression datasets, scoring whether real answers to real questions were actually correct, not just structurally valid. Our own self-monitoring pass runs continuously in production after everything else, catching the operational drift that neither of the other two is looking for, duplicate tables, a tool quietly failing more than it used to, a run that stalled instead of finishing. A change can pass Harbor's benchmark, score well in Langfuse, and still, months later, start drifting in a way only the self-monitoring pass would catch. All three are necessary. None of them is sufficient alone.

Why this is worth wiring in rather than building further ourselves

We keep coming back to the same formula in how we think about this: AI Agent = LLM + Harness, and the harness is where the real engineering effort belongs, because it's the part that's actually specific to what we're building. Tracing infrastructure, judge-scoring pipelines, and distributed benchmark runners are exactly the kind of unglamorous, generic infrastructure that's easy to skip and expensive to have skipped, and none of it differentiates one agent from another the way its approval flow, its sandboxing, or its Skill boundaries do. Langfuse and Harbor are mature, open-source, and self-hostable, which means we get the eval layer without handing a client's data to a vendor or spending months rebuilding something that already exists. The part we're still building ourselves is the part that's ours to build: the specific benchmark scenarios, the specific judge rubrics, the specific thresholds that decide whether a change is actually ready. The tooling underneath just needs to be trustworthy, not novel.