According to a recent study, call centre agents’ spend approximately 82 percent of their total time looking at step-by-step guides, customer data, and knowledge base articles.
Traditionally, dialogue state tracking (DST) has served as a way to determine what a caller wants at a given point in a conversation. Unfortunately, these aspects are not accounted for in popular DST benchmarks. DST is the core part of a spoken dialogue system. It estimates the beliefs of possible user’s goals at every dialogue turn.
To reduce the burden on call centre agents and improve the SOTA of task-oriented dialogue systems, AI-powered customer service company ASAPP recently launched an action-based conversations dataset (ABCD). The dataset is designed to help develop task-oriented dialogue systems for customer service applications. ABCD consists of a fully labelled dataset with over 10,000 human dialogues containing 55 distinct user intents requiring sequences of actions constrained by company policies to accomplish tasks.
The dataset is currently available on GitHub.
How is ASAPP’s dataset different?
Call centres are increasingly tapping AI-powered chat assistant tools. For training the model, the system needs a significant amount of question-answering datasets.
Notable question-answering datasets include SQuAD, Natural Questions (NQ), Question Answering in Context (QuAC), Conversational Questions Answering (CoQA, pronounced as Coca), HOTPOTQA, ELI5, ShARC, MS MARCO, TWEETQA, NewsQA etc.
“The major difference between our and other datasets is that it asks the agent to adhere to a set of policies that call centre agents often face while simultaneously dealing with customer requests,” said Derek Chen, a research scientist at ASAPP.
ASAPP focused on increasing the count and diversity of actions and text within the customer service domain, unlike other large open-domain dialogue datasets, often built for more general chatbot entertainment purposes.
In terms of labeling, dataset participants were additionally incentivized through financial bonuses when properly adhering to policy guidelines in handling customer requests, mimicking customer service environments and realistic agent behavior. The training process to annotate the dataset at times felt like training for a real call centre role, said the participants. “I feel like I’m back at my previous job as a customer care agent in a call centre,” said an MTurk agent involved in the study.
Further, ASAPP said its SOTA dataset offers two new tasks — action state tracking (AST) and cascading dialogue success (CDS). While AST keeps track of the dialogue when an action has taken place during that turn, CDS understands actions in context as a whole, which includes the context from other utterances.
ASAPP said the empirical results showed while more sophisticated networks outperform simpler models, a considerable gap (50.8 percent absolute accuracy) still exists to reach human-level performance on ABCD.
Conclusion
ASAPP believes it is time for both the research community and industry to do better in customer service and call centre applications. Today, models relying on DST to measure success have little indication of performance, and discerning CX leaders should consider the ground reality that agents face, said ASAPP.
“Rather than relying on general datasets which expand upon an obtuse array of knowledge base lookup actions, ABCD offers a corpus for building more in-depth task-oriented dialogue systems,” said Chen. The new tasks create a new opportunity for researchers and developers to explore more reliable models for task-oriented dialogue applications. “We can’t wait to see what the community creates from this dataset,” concluded Chen.
Recently, Google launched Agent Assist for chat–available for public preview. The demand for AI/ML platforms supporting call centre agents is on the rise. The tech giant recently unveiled LaMDA, a Language Model for Dialogue Applications.
According to an Accenture survey, nearly 56 percent of companies said conversational bots are disrupting the industry. As per Grand View Research, the global contact centre software market is expected to reach a valuation of $90.6 billion, growing at a CAGR of 21.1% from 2021 to 2028. Meanwhile, Gartner predicts that customers would prefer to use speech interfaces to initiate 70 percent of self-service customer conversations by 2023.
