Plume Meets Demands of Explosive Growth with Migration to Amazon DocumentDB

When Plume, a leader in the smart home services industry, reached database scalability limits in MongoDB, it faced complicated application redesigns that would require onboarding new infrastructure experts. Plume determined that moving to a managed service from Amazon Web Services (AWS) would help the company meet its scalability needs while avoiding the costs associated with self-management. Migrating to Amazon DocumentDB (with MongoDB compatibility), a fast, scalable, and fully managed document database service, proved to be the right move for the company.

“We’ve had explosive growth through the years,” says Todd Grantham, chief marketing officer at Plume. “Now we are deployed to nearly 20 million homes globally, have over half a billion devices managed in our cloud, and are growing very, very quickly. We’re on almost every continent and very broadly deployed across all sizes of internet service providers.” Even with Plume operating at this scale, the migration to Amazon DocumentDB proved efficient and easy from an application standpoint. The managed service increased scalability potential and eliminated the need for in-house database management, saving the cost of three to five approximately $150,000 Silicon Valley salaries (more than offsetting the managed service fees) and enabling Plume’s data science team to focus more energy on fulfilling Plume’s mission of creating a flawless wireless internet experience for its customers. 

Founded in 2015, Plume sells its services platform through some of the largest internet service providers in the world, including Comcast, Liberty Global, Bell, and J:COM. It also works with smaller local-level providers and even offers direct-to-customer services. Headquartered in Palo Alto, California, and with operations based out of Switzerland, Slovenia, Poland, and Taiwan, Plume employs more than 200 people worldwide. “All of our services are run in a cloud environment on AWS, and that’s really the unique differentiation for us,” says Grantham. “Instead of approaching the wireless internet environment from a hardware perspective, we solve problems through software-based solutions.”

In 2018 Plume used MongoDB for all database needs. “We had just been using basic replica set functionality—three-node replica sets with a backup,” says Brad Fino, director of DevOps at Plume. “It became clear after a while that if we wanted to scale to meet the demands of almost a billion devices, the replica set model wasn’t going to work.” If Plume were to stick with MongoDB, it would need to break up large tables into multiple shards to achieve horizontal scaling. Unfortunately, this would also require bringing in new infrastructure experts. “Our controller—the one that interfaces with the pods—was writing a state update every 3 seconds for every location in the Plume universe,” says Robert Miller, software development lead for public APIs at Plume. “This was starting to stress out MongoDB. And at the time, we couldn’t change the software architecture significantly.” 

The team identified Amazon DocumentDB as a solution that would eliminate the challenges associated with self-managing MongoDB—including latency issues. The new solution also had the potential to drastically increase scalability and functionality without increasing the workload of Plume staff. And from an application standpoint, Plume would experience virtually no feature loss when migrating from MongoDB to Amazon DocumentDB.

When it came to integrating Amazon DocumentDB into Plume’s architecture, the modifications that were necessary to refactor code were fast and painless. “We only had to change around 20 lines of code,” says Miller. “One of our API team members was able to spin up an Amazon DocumentDB server and point our code at it. It only took about a week to work through things.” After the migration, Plume worked alongside the Amazon DocumentDB engineering team to enable everything to work smoothly. “The team has given us best practice insights for how things work so that we can understand it and make decisions about how we architect and implement things,” says Miller. “It’s been a collaborative process.” 

The improvements in scalability became readily apparent. “On one of our clouds, users are making like 50,000 API calls a minute, and Amazon DocumentDB is not really blinking an eye,” says Miller. “On day one, we were able to scale far beyond what we were capable of doing with MongoDB,” adds Fino. Plume also saw a reduction in overall cost: though the migration resulted in a cost increase for managed services, the team more than made up for this by avoiding costs associated with onboarding additional team members, adding additional infrastructure, and devoting the time of current employees to database management.  

The migration to Amazon DocumentDB led to improvements in security as well. Plume completed several rounds of information security threat modeling and is confident that Amazon DocumentDB is securely encrypting data. “Because we were doing encryption at rest and for Transport Layer Security since we started the whole project, breaking MongoDB into shards would have made it more difficult: we would have to spend time doing it for each deployment,” says Murugesan Guruswamy, principal data lead at Plume. But on Amazon DocumentDB, encryption only needed to be done once and then was easily replicated elsewhere with the click of a button. With more than half a dozen production clouds to juggle, the resource savings are significant. 

The development and the refinement of database architecture in Amazon DocumentDB are ongoing processes for Plume. By continuing to work alongside Amazon DocumentDB engineers, however, the company sees progress and overcomes issues on a daily basis. “We’ve gotten great customer service from the Amazon DocumentDB service team,” says Fino. “Even the head of the service team has told me, ‘Here’s my cell phone number. If you guys ever have an issue, I’m here.’” What’s more, the close relationship between the teams has proved to be mutually beneficial for AWS and Plume. For example, Amazon DocumentDB recently increased its connection limits by 10 times, in part because of Plume’s request; now all Amazon DocumentDB customers benefit from the limit increase.

The savings in terms of time, personnel needs, overhead, and overall cost, as well as the increased scalability and security, have made the move to Amazon DocumentDB well worth it for Plume. “From my point of view regarding infrastructure, infrastructure management, and operations, I’d say it’s a win,” says Fino. “We’ve absolutely come out on top.”