Prepare Training Data for Machine Learning with Minimal Code

Open the SageMaker Studio interface. On the navigation bar, choose Data Wrangler on the left-hand side, and then choose the Import Data button.

Note that you can change the Flow’s compute instance type using the upper right button showing the current Compute instance.  You may decide to change the compute instance type based on your scenario’s dataset size and can scale it up or down when your requirements change. For the purposes of this tutorial, you can use the default ml.m5.4xlarge.

In the S3 URI Path field, enter s3://sagemaker-sample-files/datasets/tabular/brazil_houses/kaggle_brazil_houses_rental_data.csv, and then choose Go. Under Object name, select kaggle_brazil_houses_rental_data.csv.

In the S3 import details panel, note that you can change the default delimiter and the sampling method when necessary. For the purposes of this tutorial, you can use the default comma delimiter and First K sampling method. Then choose Import.

When exploring your dataset, begin by using the Data Quality and Insights report to help you quickly understand your dataset, identify possible issues, and focus your attention on the most important areas to improve the data. On the Data flow tab, in the data flow diagram, choose the + icon, then choose Add analysis. Then choose Get data insights. 

From the Data Insights pane, choose rent amount as the Target column. Then choose Regression as the Problem type. Then choose Create.

You may need to wait a minute while the report is generated. Once completed, review the Data Quality and Insights report sections to improve the dataset further before building the ML model. For this specific dataset, the Data Insights report has highlighted two possible issues: the first is related to duplicate rows in the dataset and the second is related to possible target leakage such that one feature is highly correlated with the output and may indicate a duplicate of the target rent column. The report can also be downloaded to a PDF file and shared with colleagues on your team.

For further data analysis and exploration, you can create additional analytical artifacts including correlation matrices, histograms, scatter plots, and summary statistics as well as custom visualizations. For example, choose the + icon, then choose Add analysis.

Under the Create analysis panel, for Analysis type, select Histogram and name it RentHistogramByRooms. For X axis, select rooms.

For Color by, select Rent amount.

Choose Preview to generate a histogram of the rent amount field, color-coded by the rooms variable.

Choose Save to save this analysis to the data flow.

Next, to gain higher confidence that the underlying data has some predictive power, we are going to create a Quick Model. Under the Create analysis pane, for Analysis type, choose Quick Model and name it RentQuickModel.

Then for Label, select rental amount and then choose Preview.  

The Quick Model may take several minutes to complete, then the pane shows a brief overview of the Random Cut Forest model built and trained with default hyperparameters.  The model generated also displays some statistics, including the Mean Square Error (MSE) score and feature importance to help you evaluate the quality of the dataset. Choose Save.

To navigate to the data flow diagram, choose Data flow. On the data flow diagram, choose the + icon, then Add transform

The first step is following the Data Insights Report recommendations regarding high risk items and removing the duplicate rows. So as the first transform step, choose Manage Rows, and then select the Drop duplicates operation. Then choose Preview and Save.

Second, we are going to remove the dataset features highlighted as possible sources of target leakage and not appropriate for a machine learning model predicting the rental amount. From the ADD TRANSFORM list, choose Manage columns. Then choose Drop column and choose property tax and fire insurance. Choose Preview then Save.

Then rename several columns to improve the readability of the input data set and later analysis.

From the ADD TRANSFORM list, choose Manage columns. Then choose Rename column.  Then choose bathroom as the input column and bathrooms as the output column. Choose Preview then Save.

Repeat this renaming column process for hoa [originally from hoa (R$)], rent [originally from rent amount (R$)], and total [originally from total (R$)].

In the dataset, the rental property’s animal and furniture classification is represented by various strings. In this step, you convert these string values to a binary representation, 0 or 1.

Under the ALL STEPS pane, choose + Add step. From the ADD TRANSFORM list, choose Encode categorical. SageMaker Data Wrangler provides three transformation types: Ordinal encode, One hot encode, and Similarity encode. Under the ENCODE CATEGORICAL pane, for Transform, use the default Ordinal encode. For Input columns, select animal and furniture. Ignore the Invalid handling strategy box for this tutorial. Choose Preview, then Add.

To scale the numerical columns area and floor, apply a scaler transformation to normalize the distribution of the data in these columns: Under the ALL STEPS pane, Choose + Add step. From the ADD TRANSFORM list, choose Process numeric. For Scaler, select the default option Standard scaler. For Input columns, select area and floor. Choose Preview, and then Add.

Finally, we will follow another recommendation from the Data Insight report and replace the 0s in the Home Owner Association (hoa) feature with NaN because they indicate missing data and should not be treated as valid inputs that might skew the model. Under the ALL STEPS pane, choose + Add step. From the ADD TRANSFORM list, choose Search and edit and then Convert regex to missing. Choose hoa as the Input column, specify 0 as the Pattern. Click Preview, and then choose Add.

From the Data Flow tab, review your end-to-end data flow graph including the data source, analytical artifacts, and data transformations. You can easily navigate, view, modify, and delete data flow steps iteratively.

Data Wrangler further streamlines the automation process of exporting the output of the data flow to a persistent destination and can orchestrate the schedule of the flow’s execution. First, set the storage destination to Amazon S3. 

Then specify the output dataset name (kaggle_brazil_houses_rental_data_dw_processed.csv) and the Amazon S3 location as your preferred S3 bucket. Then choose Add destination.

Lastly, create the scheduled job that will export the data flow output to Amazon S3 by choosing the Create job button from the Data Flow diagram pane, and then choosing Configure job.

Then you can decide on the job instance type, instance count, the job’s IAM security role, and the job schedule.

You can also integrate your data flow with SageMaker Autopilot which automates key tasks of training and deploying a machine learning model. From the Data Flow tab, choose the + icon and then choose Train model.

Specify the Autopilot Training method. You can choose Ensembling, Hyperparameter Optimization, or Auto. For the purposes of this tutorial, choose Auto.

Confirm the Autopilot Experiment deployment settings and then choose Create experiment.  This action launches a SageMaker Autopilot job that inspects the input data, generates and evaluates multiple ML models, and then selects the best model for subsequent deployment according to the desired performance metric (such as MSE in this tutorial). The Autopilot job may take several minutes to run and complete. Autopilot provides full visibility into how the models were selected, trained, and tuned through a visual leaderboard and programmatic APIs. Finally, Autopilot explains how models make predictions using feature attribution and explainability statistics using SageMaker Clarify. 

Data Wrangler can also be integrated with SageMaker Inference Pipelines to process data at the time of inference, thereby streamlining the steps between data processing and model inference.  When you export one or more steps from the data flow to an inference endpoint, Data Wrangler creates a Jupyter notebook that you can use to define, instantiate, customize, run, and manage the inference pipeline. To create the inference endpoint, choose the + next to the final transformation step (Convert regex to missing) and choose Export to, and then choose SageMaker Inference Pipeline (via Jupyter Notebook). Then inspect and run that Jupyter notebook.

You can optionally export your Data Wrangler data flow to a Jupyter notebook to run the flow steps as a SageMaker Processing job.