We’ve all heard that about 80 % of communication between human beings is non-verbal. Other types of communication such as: smiling, eye movement, body posture, hand gestures when we are talking or even the flow of our words are important to understand a conversation at a deeper level. These weak signals can tell us if someone is disappointed, for instance, despite the actual words they use. This valuable information can help us, for instance, detect and rectify a misunderstanding in communication.
As human beings, we are naturally capable of reading these weak signals, often this is by instinct and we use it all the time to inform our understanding. But what about the communication between a human and a chatbot? Can chatbots today feel these weak signals too?
And what would the most significant weak signals be for achieve an accurate conversation? In this post we aim to provide some answers to this. But first, let’s talk a little bit more about weak signals in human to human conversation.
Weak signals in a human to human conversation
Imagine you are calling your insurance company to find out if you can get a refund on repairs to your car after an accident. You are talking to customer service, but cannot get the right information. You desperately need the money to buy a new car to keep your job. You are struggling to find out whether you will be refunded partially, or not at all, due to the circumstances of the accident. Little by little, you start showing showing signs of impatience and anger :
- your politeness is diminishing
- you sentences are turning negative,
- you loose your smile
- your eyes are fixed
- the flow of your words is speeding up
- you’re marching up and down
- you finally decides to press the off button in a surge of anger, determined to change of insurance contractor.
As you can see, the weak signals may built up over time reaching a dangerous point where no communication at all is possible: the person stop the call and probably decide to look for a new company. The customer service agent should have noticed this, and changed the way he communicated to solve the situation peacefully and in order not to lose a customer…
Similarly, we may feel very frustrated when talking to chatbots. To overcome this the chatbot could be capable of detecting frustration and improve the user experience. As more and more companies are willing to delegate a part of their after-sale service to chatbots, detecting weak signals seems primordial. Note that, negative emotion detection is used as an indicator of conversation quality in call centers <fn>Automatic Detection of Emotions in Text, Precognox, September 30 2015</fn>
But what signals would a chatbot need to detect?
Weak signals in a human to bot conversation
At first it would seem impossible to detect non-verbal signals from a text conversation with a chatbot. However, shortly after SMS came into being, people developed a means to express non-verbal signals: emoticons and, later, emojis. Nice, isn’t it? 🙂 Emojis are more descriptive than emoticons and can express more complex emotions (Figure 1).
Figure 1 : Examples of emojis
Other non-verbal signals might be extracted as well, such as: speed of communication, sentence length, repetition and, obviously, inactivity, and length of inactivity.
Observing the content of the text also gives interesting insights: how negative, positive or neutral are the words used. The level of politeness of the phrase is also an interesting indicator, as well as the presence of rude words or even the high occurrence of strongly emotional words, etc.
It is interesting to consider the evolution of these weak signals and indicators during a conversation. As you will see below, this can be used as a filter.
Now that we know what weak signals to look out for, how can we put this into practice?
Concrete uses of weak signals in a human to bot conversation
Emoticons and emojis permit to modulate the degree of emotion conveyed by a sentence. You can typically give emphasis to a specific emotion, ex : the emoticon « 🙂 » in « I’m happy 🙂 » gives more emphasis to happiness. Emoticons and emojis can also be used to express irony: when the emotion convoyed by the emoticon (or emoji) is contradictory to the emotion convoyed by the sentence, ex : «:-( » added to « what an awesome party » : « what an awesome party 🙁 » totally changes the meaning of the sentence …
The speed at which the user interacts with the bot mean different things, it can be: user frustration or an inappropriate bot design (the user trying different ways to interact). This time, if the user stop talking in the middle of a conversation, it is a total failure: the bot has not being capable of doing its job until the end. The fact that the user sends too small « robotic » sentences to the bot may mean the bot does not fake well enough the human aspect of a conversation. Repetition may mean that the chatbot is malfunctioning at its core. And so on.
Computing the polarity of a sentence gives an idea of the dominating emotion convoyed by the sentence. However, it does not specify a specific emotion. Although polarity is quite a low-level information, it is a very interesting indicator to watch. For instance, if the polarity of the input sentences starts by being either positive or neutral and fall down to negative or very negative in a short amount of time, there is high probability that something inappropriate happened during the conversation.
Examples of polarities:
« A really bad, horrible book. » : negative polarity
« This was a pleasant day » : positive polarity
« The TV is turned on » : neutral polarity
The evolution in time of the politeness gives an insight about the success of the bot conversation. If the bot conversation starts polite and finishes not-polite, something inappropriate probably happened. The high occurrence of words conveying strong emotions (such as: « painful », « hard », « difficult », « impossible » and so one) helps find out the underlying emotion.
Note that, the context in which occurs weak signals is important, indeed:
1) An emotionally good or neutral and stable bot conversation means, most of the time, a successful bot conversation. The user has got the information he needed (or triggered the action(s) he wanted). This is a good indication the bot is well designed.
2) One emotionally bad sentence occurring abnormally, right in the middle of an emotionally good bot conversation, might be black humour from the user. The user might just be simply attempting to tease the bot.
3) A conversation starting with insults and continuing, or not, with insults is clearly an attempt of the user to test the limit of the bot or troll it. This case is clearly rubbish and means nothing.
4) A continuously degrading conversation means that the user is becoming more and more upset or frustrated by his conversation with the bot (Figure 2). This is typically the kind of behaviour we are interested to detect to counterbalance the user-experience by changing the behaviour of the bot accordingly.
Figure 2 : Degradation curve followed by counterbalancing
In practice, emoticons and emojis can be detected using good old regular expressions. Note that, you can filter out emojis by their unicode definitions <fn>Emoji, Wikipedia, Frequently Updated</fn>. Guys from Slovenia had a good time ranking emojis by their weight, so enjoy yourself here.
Example of code in Python:
import re str=”I’m in a neutral mood 😐” re.findall(u”\U0001F610”, str) [😐]
According to this, the sentiment score conveyed by 😐 is -0.388 (Not even 0! That’s because people use this emoji to express more or less negative emotions).
The low-level emotionally-related polarity of a sentence might be extracted using what is called a sentiment analyzer, there are several algorithms available for doing that: from the simple arithmetic of word vectors to the training of deep-learning models<fn>Sentiment Analysis of Citations Using Word2vec, Haixia Liu, School Of Computer Science, University of Nottingham Malaysia Campus, 1 April 2017</fn> <fn>Sentiment analysis using RNNs(LSTM), Manish Chablani, June 21 2017</fn> .
For a code example of the sentiment analysis, please have a look to this page.
Some emotion algorithms directly infer emotional states from text, many of them are based on the Ekman’s basic emotion set (anger, disgust, fear, happiness, sadness and surprise) <fn>Emotion Detection and Recognition from Text Using Deep Learning, Microsoft Developer Blog, November 29 2015</fn> <fn>Emotion Detection and Analysis Using Machine Learning and Deep Learning, Sylvester Kaczmarek</fn>. Some emotion detector algorithms use dictionaries of emotional words as feature sets. Note that, this is quite a higher-level signal, thus it may takes more computation times to get it :-).
FoxType designed a politeness checker, telling if a sentence is polite, impolite or neutral<fn>FoxType Politeness Checker Helps You Write More Polite Messages, Patrick Allan, LifeHacker, October 14 2015</fn>. Note that, this might be possible to implement such a politeness detector based on the LSTM sentiment analyser implementation given above, by changing the target labels, having the good training dataset and tweaking some parameters 🙂
Let’s dream a little …
Detecting when the conversation is irritating and/or frustrating the user and making up for it at a certain point is nice but what if the chatbot itself was capable of doing it smoothly, what if it had a real emotional intelligence? With this level of awareness, the chatbot would be capable of adapting itself to the mood of the user, and directly from the beginning of the conversation, for a much better user experience. According to some, it’s going to be the major differentiator separating great from rather good AI products<fn>Emotional Intelligence is the Future of Artifical Intelligence, Asha McLean, Fjord, February 21 2017</fn> <fn>How higher emotional intelligence will help chatbots, Jonh Brandon, VentureBeats, August 21 2016</fn> [8, 9]. Some researchers already started working on that subject: Emotional Chatting Machine.<fn>Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory, Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, Bing Liu, September 14 2017</fn>
We saw that, despite the fact bots only deal with textual content, it is possible to extract non-verbal-like weak signals from a human to bot conversation. These weak signals may clearly help us understand the bot conversation at a deeper level, and detect if and when it starts getting bad. This way, we can adapt the conversation later. Nowadays, many tools exist to extract these weak signals or indicators. What is currently missing is a general approach to decide whether or not the conversation is going well based on these signals.
The real future of chatbots may be an improvement of this: an emotional intelligence. Thus, the bot would be capable of adapting itself instantaneously, and not only deal with the « surges of anger » some users might have. For instance they could adapt themselves to the mood of the user, for a much better user experience.