Autodesk Uses AWS to Develop Models for Routing Support Cases between Customer and Product Support Teams

Customers at software provider Autodesk have support problems that range from the typical, straightforward task of finding and installing software to complex queries about the nature of the advanced modeling within the company’s software packages. These queries often have a significant financial impact for customers, making the need for support critical.

But when customers want to contact Autodesk about an issue, they aren’t thinking about its multiple divisions or the various support departments that handle a wide range of questions—they just know they need help solving a problem quickly and effectively. Often customers aren’t certain how to describe their issue; in the past, this led to customers being directed to the wrong support team, creating frustration and longer resolution times. Product support group specialized assistance requires deep engagement whereas many other problems can be solved in minutes. A customer with a simple request, such as locating a download link, might be stuck for hours because they have inadvertently joined a highly specialized, technical queue rather than getting help from the customer support team.

To solve this problem, Autodesk built machine learning (ML) skills models using Amazon SageMaker, a fully managed service that provides developers and data scientists with the ability to build, train, and deploy ML models quickly. The skills models differ from typical classification models because they attempt to choose the correct team responsible for supporting the user. This is in contrast to many models that attempt to predict the topic of a case and then route from that topic.

To train the model, Autodesk drew historical datasets from its data lake in Amazon Simple Storage Service (Amazon S3), an object storage service that offers industry-leading scalability, data availability, security, and performance. This model has led to a better customer experience and a simpler support experience with reduced business costs and increased Autodesk support staff productivity.

Autodesk creates software solutions for architecture, engineering, construction, media and entertainment, and manufacturing industries—using emerging technologies such as 3D printing, artificial intelligence, generative design, and robotics. In 2017, Autodesk moved its data science development practice from local machines to Amazon Web Services (AWS) as the first step in a broader strategy to use AWS for its product development process. So when the company had a problem with its customer support system, it immediately turned to AWS.

Autodesk’s previous rule-based routing system for customer support led to wasted time in redirecting support calls that depended solely on customer input for routing. Customers struggled to navigate the support system because it was challenging to classify their questions, and a misrouted query could raise the response time from minutes to hours or longer. “Customers shouldn’t need to understand the implications of saying they have a technical problem versus saying they have a download problem and how that impacts where they will be routed,” says Alex O’Connor, lead data scientist at Autodesk. Yet when a customer became lost in the system, the support teams had to pass information between each other to figure out where to route each incoming request internally.

Autodesk sought to create flexible, customizable ML models that would use natural language processing that looks at both words and how words go together to more accurately automate routing its customers to the right solutions. “For these sorts of problems, where customers are describing their issue in their own words, you want to try different combinations of models and data to account for the huge variation in detail and language,” says O’Connor, “and the ability to try all those things was part of the merit of AWS.”

To build the ML skills models, the Autodesk data science team’s business analysts met with each of the support teams to understand their specialties and map who does what and how. Using this knowledge, the team curated the training datasets for building the skills model. The data science team pulled historical data from thousands of customer service requests and their resolutions from its data lake in Amazon S3. Then Autodesk used Amazon SageMaker notebooks to pinpoint which model to use and how much data it needed. “The Amazon SageMaker notebooks are attractive because you can explore the data, understand the dynamics of different features in the data, and even train toy models that help you understand what the behavior of an ML model might look like trained on different parts of the data,” explains O’Connor. Once the team tentatively had the right pairing of model and data to suit a support team, it could move on to the next step: using increasingly feature-rich models that are trained on larger datasets with additional checks and balances. These richer models often generalize better and are more robust in terms of variation of customer input.

After using Amazon SageMaker notebooks to perform the analysis and initial exploration, Autodesk built exploratory models with Scikit-learn, a classical ML library of choice for most Python projects, using the many helper functions and shallow models to gain insight into the solution. Autodesk then trained its skills models in several deep learning frameworks using script mode, which facilitates rapid code reuse and model iteration. The company also uses PyTorch to run fast.ai and Hugging Face transformers for natural language processing.

To deploy the models for testing, Autodesk used Amazon Elastic Container Service (Amazon ECS) for the first API hosting versions. “We then moved to Amazon SageMaker endpoint hosting in the subsequent versions and in production, as this gave us enhanced flexibility and lowered complexity,” says Yathaarth Bhansali, principal engineer for the data science team at Autodesk. Developing and deploying the initial skills models, a process that included scaling and automating the entire script, took about 2–3 weeks.

The models that Autodesk created using AWS services have removed complexity from the customer experience and driven a more than 30 percent reduction in case misdirection in key support channels, which has helped Autodesk customers get their answers up to three times faster. “Any reduction in time that the customer spends waiting for a solution can have a huge business impact for them and their employers, as often the problems that they encounter can block progress on business-critical projects,” says O’Connor. What’s more, consistent positive customer feedback has shown that the automated routing is correctly matching the customer to a support agent.

“Moving away from our old rule-based system was difficult but very worthwhile,” says James Bradley, director of data science at Autodesk. “It led to a huge simplification of the way that we route support cases and removed some historical technical debt that was entrenched in using our former system.” Autodesk has been able to improve its development practices as AWS updates its services. For example, the company is planning to A/B test different models, and Amazon SageMaker endpoints make doing so easy. “Every time we run into something that we would like to do, we find that AWS has anticipated that need,” says O’Connor.

Overall, using Amazon SageMaker and Amazon ECS lets Autodesk’s data science teams focus on improving their algorithms instead of maintaining infrastructure. “I’m very grateful not to have to manage my own servers or update drivers anymore,” says Bradley. “I wasted a lot of time in the past having to deal with things that are below the level of the problem that I was trying to solve. We think of ourselves as reasonably expert. So when we run into problems, we would like to skip over the basics.”

The Autodesk team will continue to improve the routing system, including by adding the ability to monitor queues and alert Autodesk staff to customers who are potentially in the wrong queue, as well as using data to predict what customers need, which will improve the time to answer and increase first-touch resolution. “When the customer does something, we could derive information from those behaviors and then make a recommendation,” says Bradley. The company has initiatives to expand to other languages in its routing system and is exploring additional environments and modalities in which customers can engage with support.

On AWS, Autodesk has removed the guesswork for customers navigating the support system. “The skills models and the support environment are something that should fit the users’ needs and be the best choice for them rather than forcing them into one channel or another just because that’s the only place they can go,” says O’Connor. Now, Autodesk’s customer support system more accurately does exactly what it’s intended to do: provide customers the resources and knowledge they need to solve problems efficiently.