The row
Every graded row has five fields.
- The question, as a business user would ask it. "Revenue by channel last month." "Conversion on mobile after the release." "How did consented visitors convert."
- The answer as given. Verbatim, with the chart or query the assistant produced, at the time it produced it. Assistants change: Adobe shipped its Data Insights Agent with one skill and announced three more. The row records which version answered.
- The truth. The correct answer derived from measured data, or the honest "not answerable from this data" when the collection cannot support the question.
- The evidence. What decided between answer and truth: a decoded beacon, a parsed container rule, a recorded session in a named consent state, a warehouse query. Evidence a second engineer can re-run.
- The class. One of six, below.
Six classes
| Class | Meaning |
|---|---|
| 1. Correct and supported | The number matches the measured truth and the assistant shows evidence that actually supports it. |
| 2. Correct, not supported | The number is right; the evidence shown is missing, wrong or points at a different chart. Right by accident is still a risk. |
| 3. Correct refusal | The data cannot support the question and the assistant said so. A pass. |
| 4. Incomplete | The answer is right as far as it goes and silently drops a dimension, a segment or a date range the question needed. |
| 5. Wrong | The number differs from the measured truth beyond the tolerance set for the question. |
| 6. Answered when it should have refused | The data cannot support the question and the assistant gave a confident number anyway. The most expensive class, because it is the one nobody checks. |
Why the truth comes from the site
The assistant reads the table. The table is downstream of the site. When the two disagree, the table is wrong, and the assistant is wrong with it.
On one national retailer's site, closing the mandatory cookie notice left every consent category on, so the consent field in the history is a constant. Any assistant asked about consented visitors answers from a field that never varied. The truth for that question is "not answerable from this data". A confident number is class six.
A checkout progress event fired six times in one pass and begin checkout fired twice. An assistant that counts checkout starts is wrong before it starts reasoning. Class five, cause: collection.
The whole analytics stack on that checkout ran inside a child frame, invisible to a top-window reading. Two of our own readings that evening were wrong for the same reason and were retracted once the frame scan ran. The method has to be harder on us than on the assistant.
Tracing the cause
Every row in classes two, four, five and six is traced to one cause.
- Collection. The event never fired, fired twice, or fired where the collector did not look.
- Taxonomy. The name, the key or the metric definition differs between the plan, the container and the page. Adobe's Data Insights Agent builds its knowledge base from the component names in a data view, not from the data points. A misnamed metric is a wrong answer with a correct-looking chart.
- Consent. The consent field and the network disagree. The gated tags obeyed a refusal on the retailer's site; the analytics pings and the retail-media stack did not.
The cause names the owner of the fix: the container owner, the taxonomy owner, the consent platform owner.
Humans in the loop, code on the counts
LLM validation done by another LLM inherits the first one's blind spot: neither can see what never arrived. So the truth is measured by people on real devices and derived by code, and the grading is done by analysts who read the evidence, not by a model that reads the answer.
Counts are derived two ways, from the container and from the network, and refuse to publish on mismatch. When a live run contradicts the tool, the tool is fixed and the case pinned before the result is used.
Where an assistant offers its own evidence, we use it as a starting point and not as proof. Amplitude's Global Agent links every claim to the chart it used and its documentation tells users to review AI-created analyses before important decisions. The link tells us which chart to check. It does not tell us the chart is right.
What a delivered set looks like
Forty to one hundred rows. A class distribution. A cause distribution. A fix list grouped by owner. The question set itself, so the client can run it again after the next release. The one case we can show is on /cases; the engagement is on /answer-audit.
The measurement behind the truth column: website tracking audit. The wider discipline: AI audit and AI agent evaluation, with one page each for Adobe's Data Insights Agent and Amplitude's agents.