See how the AI features in your product are used, what they cost, and how people rate them. Prompts, conversations, models, latency, errors, tool calls and adoption, one date range across eight tabs.
What AI Analytics is
AI Analytics is where you see how the AI features in your product are used, what they cost, and how people rate them. It covers any LLM-backed feature your product tracks, whether an assistant, a copilot or a single generate button, and reads each interaction with its model, tokens, cost and feedback.
Overview summarizes the period. Adoption compares the profiles that use the AI features with those that do not. Satisfaction is the thumbs-up and thumbs-down feedback. Cost breaks spend down by model and by component. Conversations lists every threaded exchange and opens one in full. Models shows which models and providers handled the prompts. Performance is latency and errors. Tools and agents covers tool calls and agent routing. Every tab reads the same date range from the picker beside the page title, so switching tabs never changes the period.
A project that has never recorded an AI interaction sees an introduction instead of the tabs, with a link to the tracking guide. The tabs appear once the first interaction arrives.
When a total is a lower bound
Not every prompt carries every number: a provider may report no token usage for a call, a model may have no price on file, and a call may have no recorded latency. Those gaps are shown as gaps, never filled in with zero, because a zero would read as a free or instant call.
Three markers carry this. A dash means nothing was measured. unpriced means the tokens were spent but no price is known for the model, so the cost exists and is unknown rather than zero. A ≥ in front of a total means some of the rows it sums had no number, so the figure is a lower bound and the real total is at least that much.
Where a total leaves rows out, the card's caption says how many, for example 12 prompts not costed, and its hint says why: 10 prompts unpriced · 2 prompts without usage. Both kinds make the total a lower bound. Averages behave differently from sums: cost per prompt and cost per conversation are taken over the prompts and conversations that had a price, so they are shown without the ≥ sign.
The exports carry the same honesty in a form a spreadsheet can hold. A cell cannot carry a ≥ sign, so the number of prompts with no price and the number with no reported usage travel as their own columns beside the totals.
Use Cases
Check how much your AI feature is used and what it costs
A weekly look at whether the AI feature is being used, what it cost, and whether anything moved.
- Open AI Analytics from the Analytics section of the sidebar. It lands on the Overview tab.
- Set the period with the date picker beside the page title. Turn on comparison and each KPI shows its change against the period of the same length immediately before the one you selected.
- Read the five KPIs across the top: Prompts, Cost, Satisfaction, AI adopters and Avg latency.
- Scan the six charts below. Each is a preview of another tab, and View all in its header opens that tab.
- If the Cost card carries a ≥ sign, read its caption: some prompts had no price or reported no usage, and the total is a lower bound.
Find the conversations that went wrong
Find the exchanges where the assistant failed or the person gave a thumbs down, and read them.
- Open the Conversations tab. The Trouble rate KPI is the share of conversations in the period with at least one failed prompt or one thumbs down, which tells you how big the pile is before you open it.
- In the table toolbar, pick the Has errors or Has thumbs-down filter. Multi-model narrows to conversations answered by more than one model, which is worth a look when routing is new.
- Sort by Last active to read the most recent first, or by Cost to find the expensive ones.
- Click a row to open the conversation.
- In the transcript, a failed turn carries an error status and the note that the call failed before completion.
Read one conversation end to end
A support ticket or a comment names a conversation, and you want to see exactly what was asked and answered.
- Open the Conversations tab and paste the conversation id into the search box. Search also matches the agent name and the top model.
- Click the row. The page moves to the conversation's own address, so you can send the link to a colleague and they land on the same conversation.
- Read the Conversation stats panel for the totals: prompts, cost, tokens, duration, average latency, the models and tools involved, and thumbs up and down when there was feedback.
- Read the transcript under it. Each turn is a card with the prompt, the response, the model, the status and any tools the model called.
- Use the Conversation detail table under the transcript to compare turns by tokens, cost and latency, and its download button to export them.
- Back to conversations returns you to the list.
FAQs
Why does the Overview show far fewer conversations than prompts?
A prompt is one model call and a conversation is a run of prompts sharing a conversation id, so one conversation usually holds several prompts. A prompt sent without a conversation id never becomes a conversation.
Why is Cost shown as ≥$12.40?
Some prompts in the period had no price on file or reported no usage, so the figure is what the costed prompts cost and the real total is at least that. The caption says how many prompts were left out, and the hint says why.
Why don't the per-model satisfaction rows add up to the Satisfaction KPI?
Ratings that could not be matched to a prompt count in the KPI but cannot be attributed to a model or an agent, so the rows can add up to less; the Feedback card states how many. They can also add up to more: a rating on a turn that used two models counts once for each of them.
Can I share a conversation?
Yes. Each conversation has its own address, and opening the link lands on that conversation.
Why does the page show an introduction instead of the tabs?
Because the project has not recorded an AI interaction yet. The tabs appear once the first one arrives.
Why is the date picker missing on a conversation?
Because the page is scoped to that one conversation, so a period does not apply to it. It returns when you go back to the list.
Why do some turns say the text was not captured?
The SDK was running below observability level 2 when the prompt was sent. The turn's model, status, cost, tokens and latency are still recorded.
Why do some turns say the text is hidden by data masking?
The project masks that text in Data Manager: prompts, responses, reasoning, tool argument values or feedback comments. It was captured, and AI Analytics leaves it out. If a tab needs a masked field for its numbers, such as the model, it shows a notice naming the field instead of loading; the tabs that do not need it still load.