Autodesk Builds Reliable, Cost-Effective Big Data Analytics Using Amazon EMR

Founded in 1982, Autodesk produces 3D design, engineering, and entertainment software used by more than 100 million customers worldwide. Its AutoCAD software helps architects, engineers, and construction professionals create precise digital drawings. As a data-driven company, Autodesk uses analytics and insights to bring better solutions and features to its customers. However, its third-party big data solution couldn’t keep up with its growth.

“Our previous solution had a lot of reliability issues,” says Nick Wilson, senior engineering manager at Autodesk. “Data is a fundamental unit of Autodesk’s business. We needed a new provider that could deliver the reliability that we required and could scale with our growth trajectory.”

After evaluating different solutions for reliability and performance, Autodesk chose to migrate to Amazon EMR. “Autodesk is an AWS shop, and we have a great history of using AWS services,” says Wilson. “AWS also offered many more service offerings for the cost that we were paying. We could identify a path to migrate every workload from our third-party provider to Amazon EMR and other AWS services.”

Migrating Autodesk’s big data workloads to Amazon EMR would not be a simple task. This effort would involve migrating workloads from 30 different teams, each of which used different coding languages and ran different types of queries. Additionally, all Autodesk’s business-critical workloads ran on the third-party solution, and the company needed to avoid unnecessary interruptions during the process. To facilitate the complex migration, Autodesk participated in the AWS Migration Acceleration Program (AWS MAP), which helps accelerate cloud migration and modernization journeys with an outcome-driven methodology. Not only did AWS MAP support Autodesk in the technical aspects of the migration, but the program provided credits to offset some of the costs.

“Whenever you migrate to a new service, compute is not going to be optimized until you get some experience with it,” says Wilson. “With the AWS MAP pricing, we minimized the migration’s impact on our bottom line and made a seamless transition.”

Autodesk runs Apache Spark, an open-source unified analytics engine, on Amazon EMR to serve as its primary compute engine. Spark clusters support all the data processing and extract, load, transform pipelines. To handle its data load, Autodesk attaches volumes powered by Amazon Elastic Block Store (Amazon EBS)—a scalable, high-performance block-storage service—to Spark instances. These services scale seamlessly to process Autodesk’s data, which amounts to 1 PB and grows every month. To power dashboards and serve analytics, Autodesk runs Presto, an open-source distributed SQL query engine, on Amazon EMR.

“Presto on Amazon EMR is a fundamental part of the developer flow,” says Wilson. “It helps our team interact with and learn from data and provides general business insights to help develop new products.”

By adopting Amazon EMR, Autodesk has vastly improved the reliability of its big data service. For workloads that rely on Apache Spark and Presto, the company achieved 99.8 percent availability and has not experienced any significant outages or service downtime. Autodesk has also reduced the hours spent troubleshooting by 25 percent.

The development team greatly appreciates this consistent reliability and can now focus its resources on other areas of the business. “I used to not want to look at company communications on Sunday. I would see that business-critical processes were broken because our third-party solution was broken,” says Wilson. “Now, our pipelines are running stably, clusters are running as expected, and we are meeting our service-level agreements. We have a much more stable system running on Amazon EMR today than we had before the migration.”

Figure 1. Job submission and migration