How to Measure Your Website Chatbot's Performance
Putting a chatbot on your website is the easy part. The hard part is knowing whether it helps: does it answer correctly, does it say so when it does not know, and does it take real work off your team? Numbers like "answered 90% of questions" are easy to produce and easy to misread. This guide shows how to measure a website chatbot in five steps, with figures that mean what they say, before you go live and every week after.
Step 1: Test before launch with a readiness score
Before visitors see the chat, check how it does on your own content. OneType writes test questions from your pages in the words a visitor would use, plus questions your content does not cover, and gives the agent a score from 0 to 100. Finding the right passage counts for three quarters of the score, and holding back instead of guessing counts for the rest. A score of 85 or more is Ready, 70 to 84 is Good and anything lower needs work. If none of the tests are questions your content does not cover, the score never reads Ready, because the chat's ability to hold back was never checked. Each failing question comes with a button to add the missing answer or open the source.
Step 2: Replay your real questions
A score on generated questions is a start; your own tickets are better. If you have past tickets or chats, upload them as a CSV file and OneType answers each question as it would today. You see how many it would answer with sources, how many it would send to a person and how many your content does not cover. Ticket replay is not available on the Free plan, and on paid plans your plan sets how many replays you can run a month. The questions it cannot cover are the first things to fix.
Step 3: Read the weekly figures, and know what they count
Open Analytics once a week. It shows conversations, resolution rate, response time, leads and your top topics, each compared with the period before. Read the resolution rate carefully: a conversation counts as resolved by the AI only if nobody on your team stepped in, it was not handed off and the visitor did not rate an answer down; it is confirmed when the visitor says so or rates an answer up, and counted as resolved automatically when it closes quietly after a helpful answer. It is a useful guide, not proof that every problem was solved. Test chats from previews never count, and when nothing has ended yet you see no rate at all rather than a zero.
Step 4: Follow the questions it missed
The most useful number is the list of questions the chat could not answer. OneType collects the questions it held back on and the answers that visitors rated as not answering, wrong or outdated, groups them by meaning, with a count, and shows the most asked first. Each group is one answer away from covered: write a saved answer or update a page, and the chat knows it from the next conversation. If a group is off-topic, dismiss it. Watch the list get shorter over a few weeks.
Step 5: Estimate time saved, and label it as an estimate
The value report shows conversations resolved by AI, handoffs avoided, leads captured and hours saved. Hours saved is an estimate: conversations resolved by AI multiplied by the minutes a person would spend on one. It starts at 6 minutes, and owners and admins can change it to match your team. Use it to compare with your own records, such as the number of support emails per week before and after, and report it as an estimate.
See what your chatbot is worth
Learn more: analytics, read about the readiness score, or start free.