Chatbots are exciting territory because they offer a scalable way for businesses and brands to have meaningful, one-on-one conversations with people.
And as a bot developer (or soon to be 😊), you play the important part of defining how these conversations unfold for each user. From designing the conversation flow to crafting a bot personality, there’s been lots to do. But before you know it, your awesome chatbot is out in the wild, ready to deliver experiences that are both relevant and delightful.
Still, chatbots aren’t foolproof
Despite our best intentions, sometimes chatbots don’t say the right thing. They may misunderstand or forget contextual information. These shortcomings will soon be irrelevant with the quick improvement of building technologies, but they can still sometimes be frustrating.
To better chatbots, and beyond
It’s important to establish a rock-solid foundation for your chatbot (for best practices, visit here and here), but just as important is what comes afterwards. To have your bot provide a service that people actually want to use, you’ll want to continually improve and build upon it in the longer term.
In other words, aside from launching your chatbot, the single most important process to ensure success is measuring, learning, and tweaking your bot.
Why metrics matter
You may be thinking: “Bah, humbug! My chatbot is already perfection.” And for what it’s worth, you may be totally right.
For most developers, however, perfecting a bot and reaping its rewards (both extrinsic and intrinsic) can feel like you’re going ’round in circles. Subjective in nature, it requires some strategic thought, a handful of decisions that could impact your bot in the long term, and a bit of alignment.
To have a clear compass throughout this whole process, metrics are essential. They help you understand if your chatbot is doing a good job or not, and whether you’re focusing on the right priorities to make it better.
Iteration is perfection
If you’ve ever worked in tech, then you’re probably very familiar with this three-letter acronym: MVP, or Minimum Viable Product.
The concept of an MVP isn’t simply about releasing a crappy, unfinished version of your product. Rather, it’s about boiling your initial product down to its most important features, instead of trying to focus on every little thing from the very start. By shipping a ready-to-use version of your product with minimal time and effort, you’re able to prove or disprove ideas fast.
Chances are, you’ve done something similar for your chatbot.
However, a lot of people miss the most important part. They build stuff quickly, release them to others, and ask if they like it. But as soon as they get a “yes” or see a bit of traction, they think they’re done. They stop talking to people, bury their heads in the sand, and just keep building, until…
D’oh! What a monstrosity!
How to iterate using metrics
As you iterate, you’ll need both quantitative data (metrics) and qualitative data (user feedback) to validate your assumptions.
The dynamic between metrics and user feedback can be pretty complicated. It’s not always as simple as “If I change feature A per someone’s feedback, then metric B will improve.”
For example, let’s say that you built a simple chatbot that tells the weather ⛅️.
And let’s say that you also just added Slack a new channel based on some users specifically requesting Slack. By adding this new channel to your chatbot, you’re expecting usage numbers to go up.
After a few weeks, however, you see no improvement in total number of conversations and users. Your metrics still look the same, signaling there may be a more complex problem to tackle in the next iteration.
Coupling user feedback with metrics allows you to better assess your response to customer feedback. By repeatedly circling back to these two types of data, you’re given the most accurate picture of how well (or poorly) your new feature and overall product is performing.
More importantly, with a clearer view of how past iterations went, your intuition about your chatbot and how users get value out of it improves, helping you make more fine-tuned predictions in the future!
But there’s another way we can use metrics to create better chatbots.
Diagnosing your chatbot
When it comes to chatbots, you don’t need an M.D. to be a doctor. You just need the right metrics to understand how people are using your chatbot.
Using our Usage Metrics panel, your weather bot might look something like this:
An intent represents an idea that your bot is able to understand. The main feature, the core purpose of your bot, should translate into your most important intents. However, if these intents aren’t being used enough, then there is probably a lack of alignment between your vision of the bot’s core use case and how it’s actually being used.
Here, based on your chatbot’s popular intents, it’s possible that users start out fine with your chatbot (“greeting”), only to quickly get annoyed by the experience (“isnt-happy”) and throw insults at your chatbot (“insult”) ☹ It’s also possible that your chatbot isn’t correctly detecting when users ask about the weather (“get-weather”), which would explain why your users are so frustrated.
A skill is a block of conversation that has a clear purpose and that your bot can execute to achieve a goal. It’s not unusual for floating skills (“small-talk”) to occur more frequently than business skills related to the core purpose of your bot (“weather”). However, you should definitely be concerned they’re occurring significantly more often, so much so that your chatbot is all fluff and no substance.
For your bad weather chatbot, it seems that your chatbot is frequently resorting to the fallback skill (“fallback-skill”), which is triggered when no other skills are. Based on what you gleaned from the intents usage, you have a strong suspicion that your chatbot hasn’t been trained enough to understand user inputs regarding the weather (“get-weather”).
Let’s take a look under the hood…
Seems this intent is still a work-in-progress.
A parting thought
Chatbots open up a radically new world where services are easy, personalized, and fun. They’re built to be helpful 😇.
But just as simply, with the wrong priorities, chatbots can turn from delightful to frustrating, desired to undesired 😈.
The biggest challenge here isn’t technical; it’s conversational. As you continue to build upon your awesome bot, you must learn to leverage the conversational interface in a way that maximizes impact while minimizing fluff.
And to this end…
We’ve got your back! Ta-da!
Check out our brand-new Usage Metrics panel. To learn more, click here.