Managing one AI agent is straightforward. Managing a growing fleet of them, each deployed across different clients and messaging platforms, turns into a coordination problem fast. Latitude, an open-source platform for agent monitoring and evaluation, aims to solve this by treating agent management like a software operations pipeline: automatic detection, structured investigation, and reporting that scales from five agents to fifty.

The Scattered-Feedback Problem

When you build agents for clients, their feedback arrives everywhere. A screenshot lands in WhatsApp. A failed task gets reported in Telegram. A Slack thread holds the only explanation of a bug you thought you fixed last week. Before you can investigate, you have to track down which agent produced the output, which version it ran, and whether anyone has seen this problem before.

That only covers failures someone bothers to report. The agent that quietly gives the wrong delivery time, or claims a task succeeded after a tool failed, goes unnoticed until the next check-in. As you take on more clients, keeping track of these signals becomes a job of its own. The work is not technical. It is the manual labor of reading transcripts, matching them to deployments, and deciding what matters.

Latitude brings both automatic and human feedback into one place. Flaggers inspect completed sessions and leave annotations on traces where failures appear. Client reviews arrive as scores or custom signals. Related findings get grouped so you can investigate a recurring issue with its examples attached. Through an MCP integration, a coding agent can query that evidence across the entire fleet, produce reports, and compare releases without opening each project individually.

Organizing the Fleet by Deployment

The recommended structure is one Latitude project per agent deployment. A consultancy managing a logistics client with three agents -- an operations copilot in Slack, an invoice chaser in Microsoft Teams, and a support triage agent in WhatsApp -- keeps each in its own project. Other clients get separate projects. Development traffic lives in its own space.

This keeps conversations, costs, and signals together. When a client reports a problem with the invoice chaser, you know where to look. When you need a fleet-wide report, you query across projects. Organization access controls who can see each client's data.

Each deployment gets a clear name and a recorded release version from the start. Those details save you from reconstructing which agent did what when a complaint arrives two weeks later. The telemetry plugin sends conversations and tool activity to the selected project automatically.

Flaggers as the First Line of Detection

The first job is finding problems without waiting for a client complaint. Flaggers inspect completed sessions and annotate traces where a failure appears. Those annotations feed into signal discovery, so a single finding becomes part of a recurring issue rather than sitting unnoticed in a transcript.

Two kinds of checks run in the background. Deterministic checks catch structural failures: empty responses, tool errors, malformed JSON, and low cache usage. These run on every eligible session without incurring LLM costs. LLM checks look for interpretive problems: user frustration, forgotten context, incomplete work, unnecessary refusals, or an agent pretending a failed tool call succeeded. These use sampling, defaulting to 10% of eligible traffic.

The presets provide a starting point, but different agent roles need different checks. A support triage agent and a structured extraction agent should not necessarily have identical flagger configurations. The checks run as conversations arrive, so taking on another client does not require assigning someone to manually review a sample of that client's chats every morning.

Client Feedback as Structured Evidence

Clients will keep reporting problems through WhatsApp, Telegram, and Slack. The monitoring process needs to accommodate that. When a client reports a failure, the feedback enters Latitude as a score or a custom signal, with the reviewer's note attached. Either route requires a way to transfer the feedback; Latitude does not automatically read every messaging channel.

One example from the guide's sample fleet includes a review about a reminder sent to a customer who had already paid. The reviewer wrote: "reminder sent to a customer who paid. again. this is the one thing that makes us look bad." That note gives the failure context a generic error rate cannot. Keeping it with the evidence means your team can investigate the issue even if the original report came through a different channel.

Custom signals let you record an issue without first turning it into a score. The distinction matters: custom describes how a score entered Latitude, not whether a human or model produced it. An imported client review is still human feedback; an automated score is not.

Signals Turn Repetition Into Investigation

Signals group related findings into named patterns. They show occurrence counts, affected sessions and users, trends, and the underlying evidence. A recurring tracking failure becomes one investigation with several examples to inspect, rather than five separate complaints to track down.

In the support triage project, the findings included tracking lookups that lost leading zeros, delivery windows quoted in UTC but labeled as local time, and escalations that never received a human follow-up. These require different owners and different fixes. The signal list makes them visible together.

The guide emphasizes reading the actual conversations before accepting a signal title as the diagnosis. One signal in the example said customers were not told what evidence to provide for damage claims. The conversations revealed a more specific problem: the agent repeated the same evidence instructions regardless of claim type, even asking a customer to photograph damage when the pallet was missing entirely. Adding another instruction to request photos would have made it worse. The fix needed to distinguish a missing shipment from a damaged one.

Signals help you choose where to spend attention. The traces help you decide what to change. A useful request to a coding agent via MCP is: review the open signals across this fleet, prioritize by severity and affected users, and for each show the project, representative conversations, and what evidence to review before deciding on a fix.

Fleet Reports and Client Updates

A consultancy needs to know where attention is required across its customers. A client needs to know what their agents handled and what is being fixed. Both reports use the same data.

Through MCP, a coding agent can query Latitude and produce an HTML report. The example guide generated a fleet report covering 845 conversations across a six-week window, with a 4.66% tool-call error rate and 15 open signals. Of those, 831 conversations belonged to five client deployments and 14 to development projects.

Client-facing reports are separate. A service report for a logistics client with three agents handling 763 conversations explains verified improvements and outstanding problems in language the client can share with colleagues. It separates the consultancy's next steps from decisions or action needed from the client. The support agent was creating escalation tickets that nobody was picking up, for instance. The client needs to know who should fix that handoff, not just that an escalation metric changed.

The reporting request can be reused with different clients and periods. You still owe each customer a considered update, but you do not have to assemble every report from dashboards and old chat threads.

Measuring Whether Fixes Actually Work

Once you ship a fix, Latitude supports experiment-based comparisons of release-tagged traffic. The example compared v1.6.0 with v1.7.0 on a tracking failure caused by converting IDs to integers and losing leading zeros.

The results showed meaningful improvement: tool-call error rates dropped from 11.23% to 2.63%, conversation error rates from 9.42% to 2.17%, and human handoffs from 15.25% to 7.25%. The targeted signal fell from five occurrences to one. But the share of conversations affected by other signals rose from 5.83% to 10.87%, driven by timezone and damage claim problems that still needed work.

The lesson is that headline numbers alone are not enough. Compare rates and inspect examples of the specific failure you intended to address. An attractive reduction in one metric may mask deterioration elsewhere.

Turning Known Failures Into Ongoing Evaluations

Flaggers cover common failure categories. When you find a problem specific to one deployment, you can convert its signal into an evaluation that keeps checking new traffic. For the invoice chaser, that means continuously looking for reminders sent after payment.

Latitude requires a human annotation on at least one of the signal's traces before generating the evaluation. Annotating a clear failure gives the evaluator an example of what counts as wrong. The evaluation then runs on eligible traffic, in this case at 10% sampling.

This saves you from repeatedly searching for the same known behavior by hand. The evaluation still needs maintenance: an agent's tools, instructions, and client requirements change over time. But it converts a one-time investigation into a standing check, which is the kind of leverage that matters when the fleet is growing faster than your team.