What an analytics AI agent does with a question
The three agents we grade describe their own mechanics.
Adobe's Data Insights Agent, behind the AI Assistant in Customer Journey Analytics, takes the question, scans a knowledge base built from the component names of the data view, picks a dimension, a metric and a time range, and builds a freeform table and a visualization in Analysis Workspace.
Amplitude's Global Agent queries the project, builds or edits charts, and returns a short answer with links to the charts it used; in Default mode it makes 4 to 6 tool calls and searches existing work before creating new charts.
Shopify's Sidekick generates a ShopifyQL query against the store's database, shows a visualization and can export the result; every default Shopify report is itself a ShopifyQL query.
Three different engines. One shared limit: each reads what was collected.
Worked example one: "How did visitors who accepted cookies convert last quarter?"
- Agent. Any of the three; the question is about a consent field.
- What the site did. On one national retailer's site, the cookie notice offered no choice, blocked navigation until closed, and closing it left every consent category on. The only control that turned a category off sat three pages away, below a newsletter form. The consent field in the history is therefore a constant.
- What the agent would do. Find the consent dimension by name, filter to the accepted value, compute conversion, draw the chart. The chart would be well-formed. Every visitor is in it.
- The truth. Not answerable from this data. The field never varied.
- The class. Answered when it should have refused, if the agent gives a number. Correct refusal, if it says the segment cannot be separated. Cause: consent.
- The fix. A notice that offers the choice, and a consent field that records what was chosen. Method on /cookie-audit.
Worked example two: "How many checkouts started last week, by step?"
- Agent. Adobe's Data Insights Agent or Amplitude's Global Agent, on the retailer's event stream.
- What the site did. A checkout progress event fired six times in one pass and begin checkout fired twice. The whole analytics stack on checkout ran inside a child frame; a reading taken in the top window saw no Adobe at all.
- What the agent would do. Find the checkout events by name, count by step, trend by day. Adobe's agent picks the metric from component names; Amplitude's explores the taxonomy to find the right events. Both would count what the table holds.
- The truth. The measured number of real checkout starts, from a session walked on a real device with frames scanned.
- The class. Wrong, by the duplication factor. Cause: collection.
- The fix. One event per step, and a firing table that the next release is tested against. Method on /tracking-audit.
Worked example three: "Which products drove revenue last month, and how does that compare with Google Analytics?"
- Agent. Shopify's Sidekick.
- What Shopify documents. Sidekick answers from the store database through ShopifyQL. Shopify also documents why its numbers and Google's differ: page reloads counted differently, session definitions, visitors without JavaScript or cookies, blocking extensions, time zones, and tracking mechanisms that are proprietary and never shared. A cookie banner reduces session counts and the metrics built on them, conversion rate included.
- What the agent would do. Answer the first half from orders. The second half depends on a Google property that Sidekick does not read; how it handles that half is a row to grade, not a fact to assume.
- The truth. Revenue by product from orders is a database fact. The comparison with Google is answerable only if the Google collection is measured: which events fire from the storefront pixel, in which consent state, with what product identifier. A product identifier that differs between the storefront and the pixel is the kind of fault that makes the two systems disagree by product, not just in total.
- The class. Correct and supported for the first half is likely; the second half is graded on whether the agent refuses, qualifies or invents the comparison. Cause, if the comparison is wrong: taxonomy.
- The fix. One product key across storefront, pixel and analytics. Method on /data-layer.
What the three examples share
In each one the agent behaves as documented. In each one the answer is decided before the agent runs, by what the site collected. Grading the agent alone would blame the model. Measuring the site first tells you which failures are yours to fix and which are the vendor's. That separation is the audit.
AI analytics without the audit
Conversational analytics removes the analyst from the loop. The analyst was also the person who knew that the consent field was a constant and that checkout was in a frame. That knowledge has to go somewhere. The answer audit puts it into a graded question set the agent is tested against after every release.
Forty questions, graded, causes traced, fixes listed. One AI, one data source, three weeks. The engagement is on /answer-audit.
One page per agent: Shopify Sidekick, Amplitude AI agents, Adobe Data Insights Agent. The method behind the category: AI agent evaluation.