Have you ever spoken to a customer service agent who has really got on your nerves?

Perhaps, they are too informal and chatty when you'd prefer something more transactional. Or perhaps you are from a culture where you are accustomed to informality and friendliness.

Maybe you just encountered someone who seemed rude or hostile?

It can be hard to control the quality of customer support interactions. Companies want to give agents freedom to be human and do their job with empathy. But sometimes personal or cultural differences lead to messaging that is off-brand or tonally wrong.

Global outsourcing of customer support means customers could be chatting with someone with a very different perception of tone. What might seem empathetic in one culture can be jarring in another.

Even with extensive training, agents are human and make mistakes. What if we could help them by offering in-product guidance to pick up potentially annoying messaging and offer guidance to them before they hit the send button?

Could large language models power this? Are they sophisticated enough to pick up nuances in tone or breaches of quality and then rephrase messages accordingly?

That's what this set of experiments aimed to find out.

The 50-60 utterances were labelled to show what was wrong and then offered a corrected version, to use in 4 prompt experiments.

Both models could usually detect when a phrase wasn't right. However, they didn't always correctly identify exactly what rules had been broken.

Rewrites were, more often than not, an improvement on the original version.

The most successful prompts specified the tone of voice desired e.g. "friendly but professional" or specified that these were rules for "customer service". When either of these were specified in the prompt, the responses given were usually close to the mark, with acceptable rewrites even if the models' reason for the original being wrong was off the mark.

One of the surprises from this experiment for me, was that the models could handle quite complex, multi-step instructions. 

Another surprise was that the 70B model struggled with more black and white rules. An example of one that I tested was "never say sorry or apologise". The model failed on multiple occasions to get this right, either failing to identify the apologetic agent, or going too far in the other direction and identifying this as the rule that was broken every time, even when it wasn't.

For this prompt experiment, the models were given a personality type and asked to rewrite phrases that were tonally wrong.

You are a customer service agent for {brand + characteristics}. Rewrite the following phrases {list of phrases}.

I experimented with various personae, including "a hip technology company employee" and named various brands, (Apple, BT, gov.uk for example) and characteristics (including "hip", "trendy", "progressive" and "polite").

Interestingly, this form of prompt tended to result in high quality rewritten phrases, particularly for the 70B model, despite being much less prescriptive.

I was interested in getting to an informal but professional voice. The most effective persona for this was the 'hip technology company customer support agent'.

For this prompt listed out varying numbers of customer service messages and asked the models to identify the tone of each message.

The models often correctly identified the tone. However it did not label the tones consistently (especially the 70B model). Thanks to the wonderfully huge number of synonyms we have in the English language, the same phrase could be labelled completely differently each time, without being wrong. 

And let’s face it, tone is also quite subjective. In that sense, the model accurately reflected humanity: one person's description of the same phrase could be completely different to another's. Each time, the response could be valid.

So was this tone identification experiment successful? Well, although the labels applied to different tones varied considerably, they were consistent in terms of where you might put them on a positive to negative spectrum. The model might label the same phrase as ‘hostile’, ‘rude’ or ‘uncaring’, but you’d class all of these as being undesirable and something to correct.

I also observed that the model made a distinction between professional and unprofessional tones with reassuring consistency.

This was a very quick experiment to start to look at whether LLMs can act on simple linguistic instructions.

The application for this would be to help customer support agents stick to a brand's style guide.

For example, would an LLM be able to prompt an agent to reword their sentence from the passive voice to the active voice?