In conversation design, a key skill is analysing conversation data to improve the user experience.

At HelloDone, we had just launched a new conversational AI bot, powered by a proprietary machine learning model. Initial recall was 78% and in the early days we didn’t have figures for precision.

Data was available in a Kibana dashboard, but processing was initially manual. Even looking at 500+ utterances as a sample was time consuming to consider, categorise and ultimately refine for model training.

To increase efficiency, I worked with our head of AI to design analysis tools to pinpoint when people were not being understood. We refined clustering and categorisation tools and I ran these in a developer environment.

By rewriting the training data and iterating the conversation design, I improved the chatbot’s recall rate (how it understands users) from 78% to 89%. This data-driven approach improved the user experience and enhanced the content and interaction quality.

Manual review time and subsequent training improvements reduced from 2-3 days to just one afternoon.