Intuit Migrates Apache Cassandra Workloads to Amazon Keyspaces with High Availability and Reliability

Intuit is a global technology platform that helps consumers and small businesses overcome their most important financial challenges. At Intuit, the IDX team builds and delivers foundational platform capabilities for all the company’s products, such as TurboTax, QuickBooks, and Mint. Since 2017, the IDX team has relied on Apache Cassandra to store data for financial services and entities. Previously, this team relied on full-time engineers with expertise around Apache Cassandra administration and cluster operations to manage the underlying infrastructure. The IDX team was keen to remove effort around managing database infrastructure both for productivity gains and to focus more on delivering business value from its data.

The team at Intuit chose Amazon Keyspaces so that it could meet its operational efficiency goals while retaining its Apache Cassandra–based functionality. “We were keen to implement an innovative, cloud-native managed service for persistence and so that we can seamlessly scale up or down for variance in traffic using features that are on par with Apache Cassandra,” says Manoj Mohan, software engineering leader at Intuit. “That’s where using Amazon Keyspaces made sense.”

The Intuit team put together a robust plan for functional and performance evaluation followed by a complete implementation plan for migrating their Apache Cassandra workloads. The team came up with innovative strategies including implementing a dual-write approach: it continued persisting data to Apache Cassandra while also persisting to Amazon Keyspaces asynchronously in the background. Using this approach, the team was able to simulate its actual production workloads and optimize its serverless capacity needs on the new solution without affecting the customer experience. “To be able to pull off such a complex initiative involving large-scale data migration as well as database overhaul with zero downtime and without any adverse customer impact has been truly phenomenal,” says Mohan. “We were able to solve issues proactively and encapsulate all of this complexity under the hood with the engine running!”

During the entire life cycle of the project, Intuit’s team worked with AWS engineers and the Amazon Keyspaces team from AWS to make the migration successful. Working alongside AWS, the Intuit team was able to develop feature parity, plan onboarding, and receive prompt support when needed—as well as receive two performance-related enhancements delivered by the Amazon Keyspaces team. “From beginning to end, the support from AWS was great,” says Jason Lashmet, staff software engineer at Intuit.

With a small team of engineers, Intuit seamlessly migrated more than 120 TB of data distributed across a 66-node Apache Cassandra cluster. In March 2021, Intuit’s solution on Amazon Keyspaces went live. On Amazon Keyspaces, Intuit was able to expand reliability and data availability for its customers, whereas the legacy system had experienced occasional time-outs. On Amazon Keyspaces, Intuit has delivered users a consistent experience without any production incidents. Furthermore, Intuit’s team members no longer need to dedicate time for Apache Cassandra maintenance and operations (including handling lower-level database management tasks, data compaction, read repair, and tombstone removal). 

On Amazon Keyspaces, the IDX team has enhanced the elasticity of its workloads, facilitating the ability to bring on additional workloads and get them to market quickly. “In our prior state, if we had to scale out our cluster for more capacity, we would need a lead time of a few weeks,” says Mohan. “Now, using Amazon Keyspaces, we can accomplish this in 1 day.” Intuit’s expanded scaling capabilities also make adding workloads simpler for the team, and it extends those growth benefits to users. 

The Intuit team also relies on features of Amazon Keyspaces to deliver security and to enhance its data recovery process. By using the service, which encrypts data by default, Intuit can facilitate data encryption at rest and in transit. Also, to improve its recovery times, the data exchange team uses point-in-time recovery from Amazon Keyspaces. Using this feature, which provides continuous backups, the team can immediately restore its Amazon Keyspaces data to the second.

Now that it has boosted its operational efficiency and scaling capabilities, Intuit plans to continue using Amazon Keyspaces and adopt it for similar use cases. For data entities that are better suited to a relational database service, Intuit is exploring other solutions on AWS. “From our experience, this went about as well as a project could possibly go in terms of the migration itself and the outcomes,” says Lashmet. “For any suitable use case, I would absolutely recommend Amazon Keyspaces.”