Saving Millions without Sacrificing Performance of 100+ PB Data Lake Using Amazon S3 Intelligent-Tiering with Salesforce

Salesforce offers businesses cloud-based software so that all teams—from sales and marketing to IT—can share a single view of customer data and better connect to their customers.

Previously, Salesforce used on-premises infrastructure for UIP, ingesting over 100 TB of data per day, representing over 1 trillion events. Staff often needed to spend time resolving local machine disk failures and maintaining the physical hardware. They struggled to accommodate requests to increase capacity due to the additional hardware required.

In 2020, the UIP team began migrating the data lake from an on-premises infrastructure to AWS, becoming one of the over 700,000 data lakes that is built on Amazon S3. With an Amazon S3 data lake, users in Salesforce’s organization can discover, access, and analyze all their data, regardless of where it lives, in a secure and governed way.

For compute, the UIP relies on Amazon EMR, a cloud big data solution for petabyte-scale data processing, interactive analytics, and machine learning using open-source frameworks. To shorten the migration timeline, Salesforce used the AWS Migration Acceleration Program (AWS MAP), a comprehensive cloud migration program that uses outcome-driven methodology developed by migrating thousands of enterprise customers to the cloud.

Shortly after completing the migration, the UIP team began enhancing the infrastructure. To reduce Amazon S3 costs while maintaining optimal performance, the UIP team started to use the Amazon S3 Intelligent-Tiering (S3 Intelligent-Tiering) storage class, which delivers automatic storage cost savings by moving data to the most cost-effective access tier when access patterns change. Throughout the migration, the team worked alongside AWS representatives to optimize the S3 Intelligent-Tiering storage class to fit Salesforce’s needs. “AWS has supported us from the beginning and is always there when we face a challenge. The AWS team has offered continuous support over the years,” says Eric Legault, principal engineer at Salesforce.

Using S3 Intelligent-Tiering automatic access tiers, frequently accessed data stays in the Frequent Access tier, data that hasn’t been accessed in the last 30 days moves to the Infrequent Access tier, and data that hasn’t been accessed in the last 90 days moves to the Archive Instant Access tier. About 50 percent of data from the data lake is stored in the Archive Instant Access tier, a very-low-cost access tier that offers significant cost savings without compromising on performance. All three access tiers have the same low-latency and high-throughput performance. If Salesforce’s users need to retrieve data from the last several years, there is no retrieval fee, and the data is available right away. “We’re saving millions of dollars per year by using S3 Intelligent-Tiering,” says Legault.

By migrating the infrastructure of the data lake to the cloud, Salesforce lowered its operational overhead, further saving on costs and reducing maintenance for staff. “We can better control our costs for data storage and the associated maintenance when we delegate to AWS,” says Legault. “The amount of work we need to do is a small percentage of what it used to be when everything was our responsibility on premises.”

Salesforce’s data lake has experienced significant growth, scaling as it accumulates internal logs from multiple applications. This centralized repository empowers users to harness the vast potential of this data, extracting profound insights and intelligence. The migration to AWS has notably enhanced both the performance and elasticity of the data lake for a better user experience. UIP uses up to 30 instance types on Amazon Elastic Compute Cloud (Amazon EC2), which provides secure and resizable compute capacity for virtually any workload. To address changes in performance needs, UIP uses the managed scaling feature for Amazon EMR, which automatically increases or decreases the number of instances or units in a cluster based on workload. Amazon S3 provides elasticity at the storage level, scaling to accommodate growing amounts of data. “With our infrastructure built on AWS, it’s simple to scale up and down,” says Legault. “We can increase the storage we use or decrease the storage if data is no longer needed.”

For users, the data lake is more reliable, efficient, and scalable using a cloud infrastructure. Because the UIP team spends less time managing the infrastructure, it can also spend more time adding tools to enhance the customer experience. For example, the UIP team added tools for users to ingest data using predefined patterns so that onboarding is more efficient. Users also have access to Amazon SageMaker—fully managed infrastructure, tools, and workflows for building, training, and deploying machine learning models for any use case—to build machine learning models on top of the data lake for advanced analytics. “When we save on costs using technology like S3 Intelligent-Tiering,” says Legault, “we can invest money into added-value solutions such as Amazon SageMaker projects and make a bigger impact in the business.”