How QBox (Now Cyara) Improved TOBi across 15 Markets
Customer Success Story:
About Vodafone
Vodafone, known primarily as a global telecommunications company, has expanded its digital strategy into many other technologies.
One of the main pillars of its strategy was automation through a chatbot framework. TOBi, a digital assistant, helps customers with their queries in multiple languages across 15 markets. TOBi leverages Microsoft LUIS and IBM Watson Assistant as its main Natural Language Understanding (NLU) platforms.
Over the past three years, Vodafone has seen the volume of customer interactions directed to TOBi increase by more than 400% (to 45 million conversations per month in 2022). With this in mind, it became critical to increase TOBi’s level of understanding and accuracy when supporting Vodafone customers.
Across all Vodafone markets, the five main challenges were:
Understanding how best to train the NLU models
Manually testing and trying to improve the models with limited conversational AI resources
Improving and delivering a higher-accuracy TOBi across all markets
Reducing the fallback rate to live agents
Building confidence in the quality of new versions of TOBi
The Challenge
Vodafone started its journey with Cyara in 2019. TOBi trainers across 15 markets currently use Cyara Botium as their conversational AI testing tool, helping them test, understand and fix TOBi’s understanding and behaviors.
With Botium becoming part of the weekly TOBi model-improvement and validation process, the Vodafone teams have tested over 6,000 versions of their local TOBi models, using Cyara Botium's unique insights to see where improvements are needed and learn how to extract the most automation out of their chosen NLU provider.
Botium adoption across markets has been so successful that its three KPIs are now the reference for the global teams to review individual market performance.
The Solution
QBox [now Cyara] is a pivotal partner in Vodafone’s goal in improving TOBi across our global markets, and our partnership enables us to increase the accuracy of responding to our customers through our digital and voice platforms. As technologies evolve and with the ever-changing nature of the conversational AI market, our TOBi trainers have learned how to optimize and reach high-performing levels of TOBi models, thus increasing our domain and increasing our NPS scores.
- Leonid Mishkind Principal Manager AI Platforms, Vodafone
Over 6,000 versions of TOBi tested annually. Cyara Botium is adopted across 15 markets.
Outcomes
With over half a billion conversations going through TOBi annually, Botium is critical to Vodafone achieving a 53% end-to-end containment rate on these. Markets have seen an increase in their TOBi accuracy, with some of them even reaching over 50% improvement. Consequently, this has reduced the fallback rate to live agents and is saving millions in operational costs.
Secured Conversations by Reducing Fallback:
Through Botium and Vodafone’s partnership, Vodafone teams are included in Cyara's Botium vision and ambitions. Vodafone’s TOBi trainers are part of the development of the new Botium features and modules, by testing Beta features and giving Botium product owners valuable feedback for the product roadmap.
Beta Feedback Partnership:
Additionally, Botium has enabled TOBi trainers to drastically reduce the time spent testing their NLU models, with a Botium automated test only taking 3-5 minutes. Over 500 versions of TOBi are tested each month, saving Vodafone almost 2,000 manual testing hours, and allowing TOBi trainers to focus on expanding TOBi’s domain and to introduce new topics relevant to the seasonality of the business and the needs of the customers.
Reduced Manual Testing Time:
Botium works with all key NLU providers on the market to ensure its technology is updated regularly. One of the most recent updates in the conversational AI market is the introduction of Microsoft’s newest NLP engine, Cognitive Language Understanding (CLU). In line with this new development, Vodafone’s teams now have peace of mind that Botium will support the engine and the transition will be made smoothly. Teams will be able to continuously leverage Botium insights when training TOBi models.
Botium is a valued Vodafone partner, sharing best practices with TOBi trainers when it comes to improving the NLU, crafting quality training data, and working closely with relevant markets in order to maintain a high-performing TOBi model.
The Journey Continues
manual testing hours saved per month
Over
2,000
conversations automated end-to-end
265m
improvement in TOBi’s accuracy
Over
50%
Vodafone's Success with Cyara
Learn More about Botium
Learn More about Botium