Docebo Improves System Observability Using Amazon Data Firehose and AWS Glue

Founded in 2005 as a startup, Docebo quickly accelerated to become a global supplier of e-learning technology. The company’s current client list spans the globe and includes AWS itself. Docebo successfully launched its initial public offering on the Toronto Stock Exchange in 2019 and on the Nasdaq in 2020.

Before adopting AWS solutions to create a new unified log stream, the Docebo team was spending valuable time reconstructing its customers’ actions each time a bug arose, trying to identify the problem. “I could spend at least 2 or 3 days on one bug,” says Michele Acierno, senior backend developer at Docebo. It took days to go through the onerous tasks of searching logs individually, and sometimes the team was still unable to fully understand the problem. “We needed a solution to avoid the huge, tedious, daily effort that we were facing,” says Alberto Roli, software development manager at Docebo.

Docebo needed a single source of truth for its notification components for the Docebo log stream. Rather than having disparate log streams for each of the different microservices that the company uses, Docebo required something that could provide a snapshot of each state of the notification flow from its creation to the job. “Our goal was to gather normalized data such that we could understand the flow for each solution on a specific logging group and be able to debug as necessary,” says Acierno.

Docebo began a straightforward implementation in August 2021 and spent approximately 1 month putting the solution into practice. The company’s solution uses a container on Fluent Bit—a logging and metrics processor and forwarder—to handle the tracing log requests. Fluent Bit sends the requests to Amazon Data Firehose—an extract, transform, load service that reliably captures, transforms, and delivers streaming data to data lakes, data stores, and analytics services. “We use Fluent Bit to send the data with specific information, like logging type and platform domain,” says Acierno. “We use that information as partition keys for Amazon Data Firehose.” Docebo organizes the data using Amazon Simple Storage Service (Amazon S3), an object storage service offering industry-leading scalability, data availability, security, and performance. It then extrapolates that data using Amazon Athena, an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL.

Docebo’s solution also uses AWS Glue, a serverless data integration service that makes it easy to discover, prepare, and combine data for analytics, machine learning, and application development. Using AWS Glue as a data catalog helped Docebo to collect the new static fields—such as logging type, platform domain, and timestamp—for logging messages. “We’re using AWS Glue to normalize the data that we have already structured with files for Amazon Athena,” says Acierno.

The team began to benefit from the solution’s precision right away, and it can now use agnostic trace logs to discover misconfigurations or lack of configurations and resolve the root issue. Before, combing through more than 100,000 notifications using different log streams made this kind of discovery nearly impossible. By deploying one Fluent Bit daemon in every region and using dynamic partitioning from Data Firehose, the team can avoid a bottleneck. And if the team needs to implement new workloads, it can do so by using an infrastructure-as-code approach, which makes it possible to install a PHP module or node within the microservice. The module wraps the logs and provides them to Fluent Bit, creating a consistent data structure.

Roli’s team gathered metrics on notification-handling performance by setting up queries on the traced data. It discovered that the system was unnecessarily processing notification jobs due to nonoptimized logics on the management of initiated events. “Millions of jobs resulted from just a few hundred actual submissions,” says Roli. With trace logs, Docebo became aware of this problem and was able to optimize code to avoid useless jobs.

Saving time is a huge benefit for Roli’s team. Docebo has reduced the time that it takes to detect and resolve issues from days to minutes, decreasing troubleshooting time by up to 90 percent. “The only thing I have to do now is insert a specific tracing into an Amazon Athena query, and I have details on when the post was created, if it was delivered, and if there was an error. I have everything, and I can understand everything,” says Acierno. “That was something we couldn’t do before. What used to take 3 days can now take as little as 5 minutes.” In the past, 70–80 percent of the team’s time was spent resolving bugs. With the trace logs in place, the team now spends less than 15 percent of its time on resolving bugs.

Using AWS Glue along with other AWS services and Fluent Bit, the company now has a consistent data structure, offering new benefits to clients. “We can improve our code, and we can improve the future development of the input for the new notification stream,” says Francesco Marchesini, senior backend developer at Docebo. “Using Fluent Bit alongside AWS services, including AWS Glue, is a great solution. Our customers will benefit, too, because we can solve their issues and offer a better, faster product.”

On AWS, Docebo is improving its business and better supporting its product. Docebo benefits from the agnostic trace logs provided by its combined solution of AWS services, including AWS Glue, and Fluent Bit. And using solutions, such as Amazon Athena, supports the company in querying data for specific payload information, something it couldn’t do before. The developer teams now have a consistent data structure and can integrate its new solution to provide logs. The infrastructure can then be deployed using an infrastructure-as-code approach. 

In the future, the Docebo team plans to use AWS solutions to implement unified tracing along with automatic alerts and notifications. “Using AWS, we can improve our product in many ways because we created a stand-alone library that each of our present or future components or microservices can use,” says Marchesini. “We have a more stable, maintainable, extendable, and performable solution.”