AWS Public Sector Blog

How Skillshare increased their click-through rate by 63% with Amazon Personalize

A user's Skillshare homepage featuring personalized class recommendationson.

Skillshare is the largest global online learning community for creativity. They offer thousands of inspiring classes for creative and curious people on topics including illustration, design, photography, video, freelancing, and more. Skillshare has a two-sided marketplace in which teachers create video classes for members to further their career or fuel their creativity.

Teachers are working creatives and industry experts in their field, such as Dylan Mierzwinski, Jake Bartlett, Ana Victoria Calderón, Peggy Dean, and Gia Graham. Skillshare also produces original classes with leading creatives like lifestyle guru Jonathan Van Ness; graphic designer Aaron Draplin, R&B singer Kelis; interior design star Emily Henderson; and YouTuber Marques Brownlee.

Skillshare wanted their members to easily discover relevant content. As their library grew, they wanted to personalize the learning experience for each member with a seamless discovery process. In 2020, Skillshare had a proprietary, in-house recommendations service called Porch to help streamline the class discovery process. However, Porch was unable to meet the surge of member growth and the resulting exponential increase in member data fueling the recommendations.

Skillshare decided to test Amazon Personalize from Amazon Web Services (AWS) to make these data-fueled recommendations for members. Amazon Personalize is a machine learning (ML) service that makes it simple for developers to provide individualized recommendations to customers who use their applications. You can use Amazon Personalize in a variety of scenarios, such as giving users recommendations based on their preferences and behavior, personalized re-ranking of results, and personalizing content for emails and notifications.

After testing Amazon Personalize for their recommendation system, Skillshare achieved a 63% increase in the click-through rate to recommended classes compared to the performance of their existing in-house recommendations system. This helped Skillshare’s members to find the best-matched classes for their interests.

Experiment: Powering recommendations with Amazon Personalize

During the peak of the 2020 lockdowns, Skillshare saw daily new user sign ups rise by three to four times at the peak, and user engagement doubled or tripled in most markets. To handle member and data growth, Skillshare looked for a quick and simple deployable machine learning training pipeline that used state-of-the-art neural network models so they could feed the data in and get great recommendations for members out. They needed a solution that could learn from a member’s journey on the platform using session-based sequential models.

Skillshare conducted an A/B experiment to compare the recommendation strength between Porch and Amazon Personalize on a single row of classes on their homepage. Amazon Personalize showed a 63% increase on the click-through rate to recommended classes on the website, and a 15% increase on the Skillshare mobile apps.

Skillshare built their Amazon Personalize architecture to support both core application recommendations and also the distribution of the recommendations by email. This allowed Skillshare to experiment with comparing Amazon Personalize recommendations through email against the existing in-house model to test the relative strengths of the recommendations. The outcome of the test revealed a 14% improvement in the email click-through rate to the recommended classes over the in-house model.

Skillshare also experimented with class descriptions as a data point to provide the machine learning model with richer comprehension about class subject matter. Amazon Personalize is now able to take in unstructured metadata and apply natural language processing (NLP) to extract features that improve the quality of the recommendations. Skillshare then added unstructured text metadata like class descriptions to improve the quality and relevance of the recommendations.

Skillshare developed a simple A/B testing framework for running two parallel Amazon Personalize models to compare against one another. The model that was trained with class descriptions — using the unstructured text metadata feature — improved the click-through rate by 7% for some segments.

Skillshare’s recommendation solution architecture on AWS

 Figure 1: Overall architecture diagram of Skillshare's Amazon Personalize solution.

Figure 1: Overall architecture diagram of Skillshare’s Amazon Personalize solution.

Skillshare collects interaction events using Amazon Kinesis Data Streams, such as when a member watches or completes a class. This information is transformed and stored in Amazon Redshift, which acts as the main data repository. Amazon Redshift also stores additional information on features about the members and classes, like the teacher’s name and class duration. By having all necessary data for training in a single place, Amazon Redshift acts as a single source of truth.

Skillshare wanted to generate fresh recommendations in real-time for members without full machine learning model training. To do this, a data crawler crawls Amazon Redshift for incremental data changes, like new events, new members, and new classes, and posts the data to Amazon Personalize using the PutEvents operation. Amazon Personalize automatically updates the latest solution version and recommendation filters with the latest data. Recording events in real time helps Amazon Personalize learn from a user’s most recent activity and update recommendations as they use the application.

Skillshare uses AWS Step Functions to automate and orchestrate the bulk model training and campaign management in Amazon Personalize. Finally, the batch inference job output is persisted to Amazon Simple Storage Service (Amazon S3). An internal microservice then fetches the recommendations for each member from Amazon Personalize.

Skillshare built a horizontally scalable machine learning operations (MLOps) solution based on the Amazon Personalize MLOps reference solution that allows spinning up multiple end-to-end training pipelines on Step Functions. Using Amazon Personalize with MLOps helped Skillshare to train the models in parallel with real-time events and to streamline, accelerate, and enforce architecture best practices for their machine learning models. This also helped Skillshare automate the training pipeline and set up multiple A/B testing pipelines — which resulted in time savings of multiple hours per week. The Step Functions pipeline shown in Figure 1 consists of a number of steps executed by the workflow, as elaborated in Figure 2.

Skillshare’s AWS Step Functions pipeline for bulk model training

Figure 2: The AWS Step Functions pipeline, described in the following steps: 1) Data extraction, 2) Data transformation, 3) Model training, and 4) Model serving.

Figure 2: The AWS Step Functions pipeline, described in the following steps.

Step 1: Data extraction

Skillshare uses this step to gather each member’s watch history and metadata about the members and classes. For example, members select skills they’d like to learn when they onboard and teachers select the skills they teach in each class. The data extraction step has two parts:

    1. Bulk data regarding members’ watch history and metadata (features) about the members and classes is extracted from Amazon Redshift in Parquet format and persisted in Amazon S3 buckets.
    2. A Skillshare-specific data crawler called Event Tracker App crawls Amazon Redshift for incremental data changes (new events, new members, and new classes) and posts the data to Amazon Personalize.

Step 2: Data transformation

Skillshare uses this step to map the watch history information, member features, and class features to the schema defined in an Amazon Personalize dataset group. An AWS Glue ETL (extract, transform, and load) job transforms the extracted data and persists the transformed data into an Amazon S3 bucket.

Step 3: Model training

This step imports the dataset into Amazon Personalize before training. The pipeline creates dataset import jobs to ingest transformed datasets from Amazon S3. It then creates a solution version by training a solution created from the User-Personalization recipe.

Step 4: Model serving

Skillshare uses this step to update the Amazon Personalize campaign to the latest solution version and make the trained model available for inference. The same trained model is used to create batch inference jobs for personalized email campaigns.

Learn more about Amazon Personalize for EdTechs

Using AWS has allowed Skillshare to focus on delivering great classes to its members instead of focusing on maintaining their infrastructure. With Amazon Personalize, Skillshare achieved a 63% increase on the click-through rate to recommended classes on web and a 15% increase on the mobile apps. Skillshare also saw a 14% improvement in their email click-through rate to classes recommended by Amazon Personalize. Skillshare can keep running experiments to further iterate on the quality of their recommendation system, and AWS Step Functions helps Skillshare keep their model training and recommendation output up-to-date automatically.

Are you interested in how Amazon Personalize can help transform your education technology (EdTech) company or organization? Get started with Amazon Personalize by visiting the product page, or refer to hands-on resources like the Amazon Personalize Samples GitHub repo and Amazon Personalize workshops.

For more questions about how Amazon Web Services can help digitally transform your EdTech, organization, or agency, reach out to us directly.

Read more stories about how EdTechs use AWS to innovate in education:

Subscribe to the AWS Public Sector Blog newsletter to get the latest in AWS tools, solutions, and innovations from the public sector delivered to your inbox, or contact us.

Please take a few minutes to share insights regarding your experience with the AWS Public Sector Blog in this survey, and we’ll use feedback from the survey to create more content aligned with the preferences of our readers.

TAGS: , , , , , , , , , , , ,
Parvez Rafi

Parvez Rafi

Parvez Rafi is the engineering lead of the Personalization Team at Skillshare. He works with his team on exciting engineering problems involving Recommender Systems, NLP, MLOps and other data-centric applications. Outside work, Parvez enjoys watching movies. Connect with Parvez on LinkedIn if you would like to learn more about Skillshare Engineering.

Franklin Horn

Franklin Horn

Franklin Horn is the lead product manager for the Personalization Team at Skillshare. He focuses on building products that delight members and that harness the power of machine learning. He is always willing to chat about product management and machine learning — connect with Franklin on LinkedIn.

Reagan Rosario

Reagan Rosario

Reagan Rosario works as a solutions architect at Amazon Web Services (AWS) focusing on education technology companies (EdTechs). He loves helping customers build scalable, highly available, and secure solutions in the AWS Cloud. He has more than a decade of experience working in a variety of technology roles with focus on software engineering and architecture.