# How to measure AI conversations: what Insights can and cannot tell you

- URL: https://plandora.com/blog/measure-ai-conversations/
- Authors: Jason Chong (Founding AI Engineer)
- Published: 2026-10-22
- Updated: 2026-10-22
- Facts checked: 2026-10-22
- Category: Growth

> Insights shows volume, answer coverage, engagement and audience, but its figures are counts and system assessments, not proof of satisfaction or sales. Measure handoff rate, response time and outcomes through Inbox, Stages and your own CRM or bookings data. Keep containment separate from resolution, sample conversations before trusting an Answered label, and review the same few numbers every week.

## Key takeaways

- Insights covers volume, answer coverage, engagement and audience. It does not report a handoff rate, reply time, purchases or time saved.
- Answered is a system assessment, Quick Chat counts entries rather than unique people, and country is approximate from IP.
- Containment (no handoff) is not resolution (problem solved). A visitor who gives up still counts as contained.
- Revenue attribution needs your own CRM, bookings or order data matched to the source of each contact.
- Review weekly: read a sample of conversations, fix recurring gaps, check handoffs, and change one thing at a time.

Insights shows how much activity your Digital Mind gets, which questions it answers poorly and where visitors come from. It does not show whether visitors were satisfied, whether a conversation led to a sale, or how much time you saved. Use it to decide what to fix next, and measure outcomes in your own CRM or bookings data.

## What does Insights show, and over what period?

Insights has up to five tabs. Start on Overview, then open the tab that answers your question.

| Tab | What it shows |
|---|---|
| Overview | Conversations, visitors, answer rate (paid plans), product opens (with a store), active times and items needing attention |
| Conversations | Answer coverage, top questions and knowledge in use (paid plans), activity over time and chat engagement |
| Products | Recommendations and product opens, only when a store is enabled |
| Quick Chat | Opened, Replied and Reply rate for each Quick Chat, plus Shopper signals quoting what visitors said |
| Audience | New and returning visitors, first channels and countries |

Choose 7 days, 30 days, 90 days or All time; the default is 30 days. Each period is compared with the previous one of the same length, and All time has no comparison. Trend charts label days and weeks in UTC and today is partial, so the newest day looks short. The Active times card is the exception: it shows visitor messages by weekday and time in your own time zone. When the earlier period is small (under 30, according to Plandora's reference notes), Insights shows the raw change instead of a percentage, and rightly: five conversations becoming eight is a 60% rise that means very little.

## Which figures need a second look before you act on them?

**Answered is a system assessment.** Plandora's documentation calls it that, says it is not proof the visitor was happy, and advises reading the conversation before judging quality. The coverage bar splits every recorded question into Answered, Partial, No answer and Unclassified. The answer rate divides answered questions by the three classified states only, so the two figures use different denominators and will not match.

Plandora's Insights documentation does not describe how the assessment is made, and this article makes no claim about it. Research on automated grading in general shows why a spot check is sensible. A 2023 study found GPT-4 as a judge agreed with human votes 85% of the time on non-tie MT-bench comparisons, against 81% between humans, and documented position, verbosity and self-enhancement biases ([Zheng et al.](https://arxiv.org/html/2306.05685)). It tested which of two replies people preferred, not whether a reply answered a question, so it says nothing about Plandora's accuracy. The habit holds either way: sample what was marked Answered.

**Quick Chat counts entries, not unique people.** Opened can include repeat visits, and Replied needs a visitor reply, which the automatic opening message does not provide. Reply rate is Replied divided by Opened, not a conversion rate.

**Country is approximate.** Audience derives country from IP addresses and never shows a city. MaxMind, which sells IP geolocation data, calls the method "an approximate location, not a precise one" and says country-level accuracy is generally very high, falling at city level ([MaxMind](https://www.maxmind.com/en/geoip/ip-geolocation-accuracy)). Visitors on a VPN can appear elsewhere.

**Answer analysis depends on your plan.** Plandora's documentation says it needs a paid plan, while activity, engagement and audience reports work without one. If Conversations looks sparse, check that first.

## Which metrics belong in a measurement framework?

Five measures cover most of what an owner wants to know: volume, coverage, handoff rate, response time and intent signals. Insights reports the first two. Inbox and Stages supply most of the rest, and your own systems supply the outcome.

**Volume** is conversations, visitors and messages in a period. It shows demand and whether a link or placement is working, not quality: a spike can come from one shared link or a mismatched audience.

**Coverage** is the share of questions the Mind could answer. Treat it as a to-do list, not a score: the recurring No answer and Partial answer questions are worth fixing, and the Add answer button beside a question opens Knowledge › Drive with the question filled in. It cannot tell you whether an answer was good.

**Handoff rate** is the share of conversations in which the Mind asked for a person. A handoff request notifies your team and the Mind keeps replying until someone takes over, so a request and a takeover are separate events. Plandora's Insights documentation points you to Inbox when requests for human help rise, and no Insights tab shows a handoff count, so count requests in Inbox. Neither a high nor a low rate is good by default. A rate near zero can mean the Mind has no way to offer a person, while a high rate can mean the boundaries you set around consequential topics are working. [Human handoff for AI conversations](/blog/human-handoff-for-ai-conversations) covers designing the handoff itself.

**Response time** has two halves. How fast the Mind replies is not a figure Insights reports: Typical duration on the Conversations tab is how long chats run, not how quickly replies arrive. How fast a person replies after a handoff is the half the visitor feels. The Unreplied filter in Inbox shows who is waiting now; for a trend, note request and reply times yourself. [Speed to lead with a Digital Mind](/blog/speed-to-lead-with-a-digital-mind) explains why that half matters commercially.

**Intent signals** come from Stages. You define stages such as New contact, Qualified and Customer, each with "Move here when" criteria. The Mind can move a contact into a stage set to Automatic when a reply gives enough evidence, and the won stage is always manual. Counting contacts by stage shows where your criteria place them, not who will buy. Moving a contact yourself pauses automatic updates for it, so a count mixes machine and human decisions. In Quick Chat, the Shopper signals card can quote the visitor statement behind labels such as Buying intent or Unmet request, so you can check the evidence. [From conversation to qualified audience](/blog/from-conversation-to-qualified-audience) goes deeper on Stages.

## Which metric answers which question?

| Question you want answered | Metric | Caveat |
|---|---|---|
| Is interest growing? | Conversations and visitors on Overview, against the previous period | Counts activity, not quality. Small bases swing and today is partial |
| What can't the Mind answer? | Answer coverage and recurring No answer or Partial questions in Conversations | System-assessed and plan-dependent. Read the conversation before judging |
| Do people stay engaged? | Messages per chat, typical duration and Quick Chat reply rate | A long chat can mean interest or confusion. Entries are not unique people |
| Is the Mind handing over at the right moments? | Handoff requests in Inbox, divided by conversations | Not a headline Insights figure. A low rate can mean there is no way to reach a person |
| How long do people wait for a person? | Time from handoff request to first human reply, read from Inbox | Insights does not report reply time. Typical duration is chat length |
| Who looks ready to buy? | Contacts by Stage, and quoted Shopper signals in Quick Chat | Stages follow your criteria. A stage is not a purchase |
| Where do visitors come from? | Audience countries and first channels | Country is approximate, from IP address |
| Did conversations produce revenue? | Your CRM, bookings or orders, matched to the source of each contact | Insights counts opens, not purchases. Needs your own data |

## What is the difference between containment and resolution?

Containment is the share of automated conversations that end without a handoff to a person. Resolution is the share in which the visitor's problem was actually solved. They diverge whenever someone gives up. One vendor's glossary defines containment as "the share of automated conversations that end without escalating to a human", counts an abandoned chat as contained, and advises reading it beside resolution rate ([Fin](https://fin.ai/glossary/containment-rate)).

Research suggests the gap can be wide, though none measures it for your sector. In a Gartner survey of 5,728 customers, reported by [CX Today](https://cxtoday.com/customer-experience/only-1-in-7-customer-service-queries-resolved-with-self-service-gartner-study-finds/) in August 2024 and run in December 2023 according to [Digit](https://www.digit.fyi/?p=103910), customers fully resolved 14% of their customer service and support issues in self-service. For issues they called very simple, the figure was 36%. The survey covers self-service in general, not AI chat, and the coverage does not define "fully resolved".

The US Consumer Financial Protection Bureau described the other side in June 2023, after reviewing consumer complaints and the market for chatbots in consumer finance. It said "doom loops" are "often caused when a customer's issue falls outside the chatbot's limited capabilities" ([CFPB](https://www.consumerfinance.gov/data-research/research-reports/chatbots-in-consumer-finance/chatbots-in-consumer-finance/)). A visitor stuck in such a loop never reaches a person, so the conversation would count as contained. The context is US consumer finance, so take the failure pattern, not any numbers.

Insights publishes neither a containment nor a resolution figure, and Plandora's reference notes state that system signals are not customer-confirmed resolutions. To estimate resolution yourself, sample conversations that ended without a handoff and mark each resolved, abandoned or unclear. Twenty gives a rough read, not a precise rate.

## Can Insights tell you what a conversation was worth?

No, and it says so. Product opens are not purchases or revenue. Quick Chat reporting counts engagement only, and Plandora's notes say no physical sale or staff-time saving is inferred. Plandora does not claim that Insights improves conversion, saves time or raises revenue: its reference notes say improvements in owner outcomes have not been measured with real users. Purchase attribution needs an actual matched order event, which has to come from the system that holds the order.

Attribution is also a modelling choice. Google Analytics Help defines it as "the act of assigning credit for important user actions to different ads, clicks, and factors", and its reports offer a data-driven model and last-click models that can credit the same conversion differently ([Google Analytics Help](https://support.google.com/analytics/answer/10596866)).

The practical route is to carry the source into your own system. Record which Quick Chat link, placement or channel a person came through, then match that contact to bookings, orders or CRM deals. For a pilot, Plandora's Quick Chat notes suggest one store and website, a few placements, and a comparison of effort, engagement, handoffs and usage costs over comparable periods or against a website holdout before claiming incremental value. [Quick Chat for in-store conversations](/blog/qr-codes-and-quick-chat-for-in-store-conversations) covers the entry points.

## What does a weekly review look like?

1. **Set 7 days on Overview and read the comparison.** Note conversations and visitors, and ignore percentages on small bases. If you use Quick Chat links, check which Quick Chats get replies, beyond how many get opened.
2. **Open Conversations and list the recurring No answer and Partial answer questions.** Use Add answer for anything asked more than once. Leave one-off questions alone.
3. **Read ten recent conversations in Inbox, including some on topics that Top questions shows as Answered.** Mark each right, wrong or unclear, then fix any wrong answer at its source in Knowledge or the Playbook.
4. **Open Inbox.** Count handoff requests, check Unreplied for anyone waiting and note the longest wait. Then glance at contacts by Stage.
5. **Write the week's numbers in your own sheet, beside your CRM or bookings data.** Record conversations, the share that needed a person, qualified contacts, and bookings or orders from the same source. Make one change, note it, and compare next week.

## Frequently asked questions

### What does "Answered" mean in Plandora Insights?

Answered is a system assessment of whether the Mind answered a visitor's question. It is not proof the visitor was satisfied. Plandora's documentation advises reading the conversation before judging quality, and the answer rate only counts questions the system could classify.

### Can Insights show how many sales your Digital Mind produced?

No. Insights counts conversations, visitors, product opens and Quick Chat entries, and product opens are not purchases or revenue. Purchase attribution needs a matched order event from the system that holds your orders. Record the source of each contact and compare it with your CRM, bookings or order data.

### What is the difference between containment and resolution?

Containment is the share of automated conversations that end without a handoff to a person. Resolution is the share where the visitor's problem was actually solved. A visitor who gives up, or who cannot find a way to reach a person, still counts as contained, so read the two figures side by side.

### Do Quick Chat numbers count unique people?

No. Quick Chat counts entries, and Opened can include repeat visits. Replied needs a visitor reply, which the automatic opening message does not provide. Reply rate is Replied divided by Opened, so treat it as a comparison between Quick Chats rather than a count of customers.

### How accurate is the country shown in Audience?

It is approximate. Insights infers country from IP addresses and never shows a city or exact position. MaxMind, an IP geolocation vendor, says country-level accuracy is generally very high but the method gives an approximate location. Visitors on a VPN can appear in a different country.

### How often should you review Insights?

A weekly pass on the 7 days view is a workable rhythm: read the comparison, list recurring unanswered questions, sample conversations, check Inbox for handoffs and record a few numbers beside your CRM. Use 30 or 90 days for trends, because small weekly bases make percentages swing.

## Sources

- [Chatbots in consumer finance](https://www.consumerfinance.gov/data-research/research-reports/chatbots-in-consumer-finance/chatbots-in-consumer-finance/), Consumer Financial Protection Bureau, accessed 2026-10-22
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/html/2306.05685), arXiv (Zheng et al.), accessed 2026-10-22
- [Only 1 in 7 customer service queries resolved with self-service, Gartner study finds](https://cxtoday.com/customer-experience/only-1-in-7-customer-service-queries-resolved-with-self-service-gartner-study-finds/), CX Today, accessed 2026-10-22
- [Coverage of the Gartner self-service resolution survey (August 2024)](https://www.digit.fyi/?p=103910), Digit, accessed 2026-10-22
- [Chatbot Containment Rate: Formula and Why It Misleads](https://fin.ai/glossary/containment-rate), Fin, accessed 2026-10-22
- [IP geolocation accuracy](https://www.maxmind.com/en/geoip/ip-geolocation-accuracy), MaxMind, accessed 2026-10-22
- [Get started with attribution](https://support.google.com/analytics/answer/10596866), Google Analytics Help, accessed 2026-10-22