What is an AI audit
An AI audit, as we practise it, is a graded comparison between what an AI answered and what is true, with the cause of every gap. For an analytics assistant, "true" has a precise meaning: what actually happened on the site, measured, not what the table says happened.
That definition sets it apart from two neighbours.
- An AI model audit examines the model: its training, its bias, its behaviour on benchmarks. Useful, and not ours. Adobe, Amplitude and Shopify run their own models; Amplitude documents that its agents combine models from several providers behind its own orchestration. You cannot audit that model. You can audit its answers.
- An AI governance audit examines the organization: who approved the AI, what policy covers it, who is accountable. Also useful, also not ours.
- An AI data audit is closest. It asks whether the data the AI reads is fit for the questions asked of it. Our audit contains that step and adds the one that matters to the buyer: the answers themselves, graded.
Why the answers are the thing to audit
The assistant is confident by construction. Amplitude's documentation describes its Global Agent as reporting what the data shows and linking every claim to the data. Adobe describes its Data Insights Agent as building the visualization in Analysis Workspace so the user can verify the data. Both are honest descriptions of a design. Neither can reach past the table.
On one national retailer's site, closing the mandatory cookie notice left every consent category on, so the consent field in the history is a constant. Ask either assistant how consented visitors converted and it will draw a chart. The chart will be beautifully supported by a field that never varied.
On the same site, 279 of 280 tag rules carried no consent gate; a checkout progress event fired six times in one pass; the whole analytics stack on checkout ran inside a child frame, invisible to a reading taken in the top window. None of this is visible from inside the data. All of it changes the answers.
How to audit AI that reads analytics
Five steps. Each one produces an artifact the next one uses.
- Fix the question set.Forty to one hundred business questions, written the way a leader asks them, agreed with the client before anyone touches the assistant. This is the audit's contract. Questions added afterwards are graded in the re-audit, not slipped in.
- Measure the truth.Real sessions on real devices, per consent state, network recorded. The tag container parsed rule by rule. The warehouse queried directly. Every question gets a correct answer or the honest "not answerable from this data". Method on /tracking-audit and /data-quality-audit.
- Ask the assistant.Each question, once, as written, with the answer recorded verbatim together with the chart or query it produced and the version of the assistant that produced it. Assistants change; Adobe shipped its Data Insights Agent with one skill and named three more to come.
- Grade each row.One of six classes, from correct and supported to answered when it should have refused. A confident answer to an unanswerable question fails, however plausible the number.
- Trace each failure.Collection, taxonomy or consent. The cause names the owner of the fix. Full rubric on /how-we-grade.
AI audit checklist
Before the audit starts
- The question set is written and agreed.
- Read access exists to the assistant, the analytics tool, the tag container and the tables, all named seats, time-boxed.
- The consent states the site offers are listed, per region.
- The assistant's current version or skill set is recorded.
During the audit
- Every session runs on a fresh profile, on a real device, in a named consent state.
- Every request is decoded, not counted.
- Frames are scanned; a top-window reading is never the evidence. Two of our own readings were wrong on one evening for exactly that reason and were retracted the same night.
- Counts are derived two ways and published only when they agree.
After the audit
- Every row has a class and every failed row has a cause and an owner.
- The question set is handed over for re-runs.
- Anything the audit could not determine is written as undetermined, not guessed. On the retailer's site, which stage a second tag environment belonged to was unread; the report says unread.
AI audit tools
The tools are ordinary and the discipline is not. A real browser with a network recorder. A parser for the tag container. Read access to the warehouse. The assistant's own evidence links, which tell you which chart to check and not whether the chart is right. Amplitude's documentation says its Global Agent links every claim to supporting data and asks users to review AI-created analyses before important decisions. That instruction is the audit in one sentence. We do the reviewing.
AI audit software that reads only the table can find nulls, drift and duplicates. It cannot find a constant that should have varied, or an event that fired in a frame nobody recorded. Those need the site.
AI audit best practices
- Grade the refusal. A correct "not answerable" is a pass. An answer where a refusal was due is the most expensive failure, because nobody checks a confident number.
- Record the version. Vendors ship skills, modes and new agents continuously. Amplitude documents three response modes for its Global Agent with different depths of taxonomy exploration. The same question in a different mode is a different row.
- Be harder on yourself than on the assistant. Our first published finding on the retailer's consent controls was wrong. We corrected it on the record the same night. An audit that cannot correct itself cannot grade anything.
- Never let an LLM grade an LLM alone. Both read the same table. Neither can see what never arrived. People measure the truth; code derives the counts; analysts assign the class.
- Trace to an owner. A finding without a fix and an owner is a complaint.
LLM audit versus answer audit
An LLM audit tests the language model: prompts in, outputs out, scored against a reference set. It is the right tool for a model you built. For an assistant a vendor built and you cannot open, the model is a fixed part and the data is the variable. Audit the variable.
That is the AI answer audit. One AI, one data source, forty questions graded, every wrong answer traced to its cause. Method on /how-we-grade; the engagement on /answer-audit; the agent-level view on /ai-agent-evaluation; the one case we can show on /cases.