In the spring of 2023, we launched twoday AI Agent, an AI application that utilizes OpenAI's GPT language models but is customized to meet the needs of businesses in terms of security and confidentiality. As a ready-to-use product, twoday AI Agent is suitable for companies wanting to immediately implement generative AI into their operations and quickly derive business value. Common use cases include enhancing customer service, automating internal processes, and facilitating data processing and utilization.
Case in short
In a recent project, experts at a large organization used a wide range of realistic and complex customer support questions to test both a freely available AI application and twoday AI Agent, which uses a built-in Retrieval Augmented Generation (RAG) framework. They evaluated whether they could provide the AI-generated response directly to the customer.
The AI Agent performed significantly better in the test than the compared AI, namely the free version of ChatGPT.
This blog will discuss how much better the AI Agent performed and why.
twoday AI Agent in a Large Customer Service Center
The use of the twoday AI Agent was evaluated in a project with a customer service center employing hundreds and handling over two million calls and half a million chat messages annually.
When dealing with large amounts of data, the AI solution must be high-quality and reliable, as even a small error rate can significantly affect business, staff, and customers. Therefore, the project also adhered to responsible AI principles and monitored their implementation.
Read more about responsible AI in our blog: "Where AI Projects Fail and How to Ensure Your AI Doesn't Spout Nonsense," where Artificial intelligence expert Tony Shepherd discusses how companies can steer AI projects from hype to production, and how to do so responsibly.
The Significant Impact of the RAG Framework
The project combined twoday's technology and expertise with the customer service center's knowledge. The Retrieval Augmented Generation (RAG) framework was utilized in development. This means that the language model receives not just the user's question but also enriches the input with data based on the question and possible other additions, using so-called prompt engineering.
A significant advantage of twoday AI Agent is that intelligence and all necessary features are built into the product, making configuration and customization easy and quick. This means the product is ready to use immediately, and methods like RAG can be utilized directly without the extensive development work typically required.
In tests, the AI Agent's performance was compared to the free version of ChatGPT, which does not use the RAG framework. All the information needed for responses is available from the public internet, so ChatGPT's language model should also have access to this information when forming responses. The test results show that the AI application using the framework clearly outperforms compared applications, offering more useful and higher-quality answers.
The project consisted of several experiments where experts from the customer service center used different AI applications, asked questions, and provided feedback for agent analysis and improvement. The AI application performance was evaluated using jointly developed metrics. These metrics covered accuracy and ease of use, with users rating each agent's response speed and quality on a scale of 1–4, where 4 means a professional could directly offer the response to a customer.
The project consisted of several experiments where experts from the customer service center used different AI applications, asked questions, and provided feedback for agent analysis and improvement. The AI application performance was evaluated using jointly developed metrics. These metrics covered accuracy and ease of use, with users rating each agent's response speed and quality on a scale of 1–4, where 4 means a professional could directly offer the response to a customer.
Results
The AI Agent performed significantly better in the test than the compared AI. Users compared responses from both agents using the same guidelines and questions. According to the experts’ assessment, most of the AI Agent's responses were suitable to be presented directly to end customers.
"The AI Agent's ability to precisely pick industry-specific information from the training data resulted in it performing significantly better in the test than the compared AI application, namely the free version of ChatGPT," comments Tony Shepherd, Senior Consultant in twoday's Data & Analytics team.
When comparing the success rate of ChatGPT and twoday AI Agent (i.e., when a user gave a response a rating of 3 or 4), the AI Agent achieved an overall success rate of 83%, while the AI application not utilizing the RAG framework had a success rate of only 36%.
When comparing the success rate of ChatGPT and twoday AI Agent (i.e., when a user gave a response a rating of 3 or 4), the AI Agent achieved an overall success rate of 83%, while the AI application not utilizing the RAG framework had a success rate of only 36%.
During the test, experts also evaluated the future features of the AI Agent and their impact on the success rate.
Experiments showed that future improvements of the AI Agent, such as adding more language models for information retrieval and more effectively utilizing user feedback, could potentially achieve a success rate of over 95%.
twoday AI Agent's Performance According to User Evaluations. The AI Agent, using the RAG method, scored higher than ChatGPT in user evaluations, particularly when considering the direct applicability of AI responses to customer questions.
AI Brings a New Dimension to Customer Service
As a result, twoday's AI application not only provided high-quality support to customer service centers but also significantly surpassed the performance of the compared AI solution not utilizing the RAG framework. The project ensured the AI solution was optimized and that users could continuously monitor and improve it.
"twoday AI Agent not only offers customized and secure service to our corporate clients but also clearly demonstrates how AI's potential can be further utilized. By using the Retrieval Augmented Generation method and combining different large language models, we have succeeded in creating a solution that meets the high demands of accuracy and reliability in customer service centers," says Janne Sipilä, Business Development Director at twoday's Data & Analytics division.
"The project results clearly show that the twoday AI Agent can provide excellent support and perform significantly better than currently available solutions on challenging questions," Sipilä continues.
"twoday AI Agent not only offers customized and secure service to our corporate clients but also clearly demonstrates how AI's potential can be further utilized. By using the Retrieval Augmented Generation method and combining different large language models, we have succeeded in creating a solution that meets the high demands of accuracy and reliability in customer service centers."
Janne Sipilä, Business Development Director, Data & Analytics
twoday's client base is continuously growing, and twoday AI Agent is already in use in about 40 organizations in Finland.
Want to learn more about the AI Agent or other AI solutions from twoday?
Contact:
Janne Sipilä
Director of Business Development
janne.sipila@twoday.com