Airbnb Uses Amazon EFS to Scale CI/CD Pipeline for Expanding Online Marketplace

Airbnb turned to Amazon Web Services (AWS) to help achieve its goals. “We worked with AWS for many months to consider different options that could solve the challenges with our source control infrastructure,” says Joel Snyder, software engineer for Airbnb. “They took time to listen to what we required and gave us advice about how we could best integrate with their services.”

While exploring how to solve its challenges, Airbnb realized it could utilize Amazon Elastic File System (Amazon EFS), a simple, scalable system for Linux-based workloads for use with AWS Cloud services and on-premises resources. “Through our research, we discovered Amazon EFS, which we could use to share a single file system mounted on all Git mirror instances,” explains Snyder. “When one Git mirror changed, every other Git mirror was guaranteed to have the same update.” In this way, Airbnb used Amazon EFS to back the real-time image of its GitHub repository data. Mirrors were continuously in sync with the production repository at the scale required for the CI automation pipeline.

Airbnb then needed a solution that could keep the GitHub Enterprise repository's file system in sync with the mirrored repository's file system. The team chose an event-driven approach, meaning only actual code changes to a repository prompted the repository syncing process. Airbnb turned to Amazon Simple Queue Service (Amazon SQS), a fully managed message queuing service, to avoid malformed or bad-data messages being committed to the production GitHub repository. “We used Amazon SQS as a queuing mechanism to buffer events from the GitHub primary to our syncing service,” says Daniel Low, software engineer for Airbnb.

Most of the time, repositories are automatically synced successfully. However, if a message can’t be handled automatically, Amazon SQS ensures no data is lost by redelivering messages until the repository is successfully synced. Each day, the system processes approximately 10,000 messages about code changes and repository syncing with a 99.5 percent success rate.

“When a bad message enters our system, we don’t constantly retry,” explains Low. “After five attempts, we move messages that can’t be successfully processed into a dead-letter queue.” Airbnb sets alerts for the dead-letter queue, so it can quickly discover the reason for the failed message and find a solution. “Once the issue is fixed, we replay the contents of the dead-letter queue back to Amazon SQS,” he says. “We use that process to verify that the issue was resolved.”

“Before this migration, we had serious conversations about how to engineer and scale our source control infrastructure,” says Low. “Now, using Amazon EFS and Amazon SQS, we no longer worry about that. We know we can scale to match our growth.”

By using managed services such as Amazon EFS and Amazon SQS, Airbnb has significantly reduced the operational load required to maintain its source control infrastructure as its workload scales to tens of thousands of commits per day. “Our team can now focus on building new features and bringing value to our internal customers, the other engineers at Airbnb,” says Snyder. “Reliable source control is a crucial foundation to support Airbnb’s infrastructure and to enable Airbnb’s engineers to be productive.”