AI agent evaluation: when the agent's tool is your data

AI agent definition, for evaluation purposes

An analytics agent takes a question in plain language, decides which data to look at, runs one or more queries, and returns an answer with a chart or table. Three vendors describe the pattern in their own material.

Amplitude's Global Agent runs on every page of Amplitude, queries the project, builds and edits charts, creates cohorts and runs multi-step investigations; each response is a short answer with links to the charts it used. It runs in one of three modes: Fast, 2 to 4 tool calls and minimal taxonomy exploration; Default, 4 to 6 tool calls; Investigate, 10 or more tool calls with extensive taxonomy exploration and hypothesis testing.

Adobe's Data Insights Agent sits behind the AI Assistant in Customer Journey Analytics. A reasoning engine routes a data question to it; it scans a knowledge base built from the component names of the data view, picks a dimension, a metric and a time range, and builds the visualization in Analysis Workspace.

Shopify's Sidekick generates ShopifyQL queries, displays visualizations and exports reports from the store's database; it uses the page the merchant is on as context.

For evaluation, the definition that matters is this: an analytics agent's tools read a table the agent did not make. Its ceiling is the table.

Agent evaluation criteria

Five criteria, in the order they fail.

• Answer correctness. Does the number match the measured truth, within a tolerance set per question? The truth is measured on the site, not read from the table, because the table is what is being tested. Method on /tracking-audit.
• Refusal correctness. When the data cannot support the question, does the agent say so? On one national retailer's site the consent field in the history is a constant, because closing the mandatory notice left every category on. "How did consented visitors convert" has no answer there. An agent that gives one fails this criterion, however plausible the number.
• Evidence fidelity. Does the evidence the agent shows actually support the answer? Amplitude documents that every Global Agent claim links to supporting data. The link is a pointer, not a proof; the evaluation follows it and checks.
• Trajectory. Which components did the agent choose, and were they the right ones? An agent that picks the metric named "orders" when the plan's order count lives under another name is wrong at step one. Adobe's knowledge base is component names, so the taxonomy is the trajectory.
• Stability. Does the same question, asked again in the same mode, give the same class of answer? Modes matter: Amplitude documents different depths of exploration per mode. A row records the mode.

Agent evals as a set you keep

Generic agent evals are run once against a benchmark. Analytics agent evals have to be run again, because both sides move. The vendor ships modes, skills and flags: Amplitude introduced an isGenerated flag for AI-created content in November 2025, and content created before that date does not carry it. The site ships releases: the retailer's audited tag container was superseded ten days after the extract. So the graded rows are kept as a set, with the agent version and the container build recorded on each, and the set is re-run after either side changes. A row that moves class between runs names what moved it.

Why LLM agent evaluation is different here

Generic agent evals test the model's tool use on a benchmark environment the evaluator controls. The environment is clean by construction, so the score measures the model.

In analytics the environment is the client's data, and it is not clean. A checkout progress event fired six times in one pass on the retailer's site. 279 of 280 tag rules carried no consent gate. Analytics requests marked as denied still reached the vendor as cookieless pings. Every one of those changes the table, and the agent reads the table.

So the score measures two things at once, the model and the data, and only one of them is yours to fix. The evaluation has to separate them. That is why every failed row is traced to a cause: collection, taxonomy or consent. A failure caused by the data is fixed upstream and re-graded. A failure the data cannot explain is the model's, and the vendor's.

Agent evaluation tutorial

Six steps for the first ten questions.

1. Write ten questions your analysts answer every week.Plain language, the way a leader asks. Note for each one whether you believe the data can support it.
2. Measure the truth for each.Query the tables directly. For any question about a page event, open the site on a fresh profile with the network recorder on and count what fires. Check frames: on the retailer's checkout the entire analytics stack ran inside a child frame and a top-window reading saw none of it.
3. Ask the agent, once per question, in one mode.Record the answer verbatim, the chart or query, and the mode. Do not rephrase to help it. The leader will not.
4. Follow the evidence.Open every chart the agent linked. Does it show what the answer claims?
5. Class each row.One of six, from correct and supported to answered when it should have refused. Rubric on /how-we-grade.
6. Trace each failure.Collection, taxonomy or consent. Write the owner next to it.

Ten rows will show you the pattern. Forty will show you its shape. That is the answer audit on /answer-audit.

The three agents, one page each

What the evaluation does not do

It does not tune, prompt or replace the agent. It does not score the vendor's model against another vendor's. It grades the answers your leaders will act on, against the truth of your site, and hands the causes to the people who can fix them. The wider discipline is on /ai-audit; the one case we can show is on /cases.