Below is the synopsis of the data science initiative completed by Joo Ann Lee, a senior data scientist with Witmer Group. Joo Ann walks through the business case for development of the model, the anticipated benefits, the tool used (Microsoft Machine Learning Studio), the data science techniques, and an example of the outcome. VERSO now offers this predictive model as a component of the VERSO subscription. Since VERSO runs on the Azure cloud, using the Microsoft Machine Learning Studio is a natural and seamless approach to modeling valuable predictive behavior.

Product Return Predictions in the VERSO Portal: Which, How Many and When.

By: Joo Ann Lee – Senior Date Scientist  – Witmer Group

Introduction

Product warranties and returns are a part of everyday business operations. Within the VERSO reverse logistics module, when a product is having some type of issue, a Maintenance Trouble Report (MTR) is created to determine if a service call or product return is required. Therefore, the ability to predict which product may trigger MTRs and estimate the number of potential MTRs is useful.

However, a business opportunity for the VERSO users lies in knowing not only which product and how often the product may trigger MTRs, but when the MTRs may be triggered. For example, an opportunity to increase revenue can come from implementing premium repair fees if an extended warranty for a product is not purchased. This business opportunity requires the assistance of a statistical model trained to predict products that are likely to trigger MTRs after the expiration of its warranty. Using the same statistical model, another revenue opportunity can come from the sale of extended warranties for products not predicted to trigger MTRs after the expiration of its warranty. This report details the project based on the abovementioned statistical model.

More specifically, a Boosted Decision Tree Regression model was trained to predict 1) which products are likely to trigger MTRs after the expiration of its warranty, and 2) estimate the number of MTRs triggered, within the VERSO system.

Model Building and Evaluation

Three different regression models were trained in Microsoft Azure machine learning studio (https://azure.microsoft.com/en-us/services/machine-learning-studio/)  –  where 1) the name of the company, 2) the name of the individual that created the MTR, 3) the current step the product evaluation is in, 4) whether or not the product has a warranty, 5) the total RMA incidents for the product and 6) the shipping company used by the company, were used to predict products that are likely to trigger MTRs after the expiration of its warranty and estimate the potential number of MTRs the product may trigger.

70% (n = 3567534) of the dataset (N = 5096477) were used to train the models while the other 30% (n = 1528943) were reserved to test the models. The Boosted Decision Tree Regression model was chosen because it produced the most accurate prediction when compared to the Multiple Linear Regression and Neural Network Regression models (see Table1).

Table1. RMSE values associated with each of the three statistical models.

Statistical Model	RMSE*
Multiple Linear Regression	0.020796
Boosted Decision Tree Regression	0.004429
Neural Network Regression	1.176622

*The Root Mean Square Error (RMSE) is a metric summarizing how close the actual data points are to the predicted values. Thus, the smaller the RMSE, the more accurate the statistical model.

Boosted Decision Tree Regression Overview

At a high level, the Boosted Decision Tree Regression model uses a series of recursive binary splitting rules that start at the top of the “tree” (where the dataset belongs to a single region) to split the predictor space into smaller cohesive rectangular regions called nodes or leaves. The decision to split is based on minimizing the sum of squares error of a potential region. After a split, the average value of the observations within the region is taken to be the predicted value of that region.

The Boosted Decision Tree Regression model belongs to the family of ensemble statistical model. Specifically, many different models are built using the same recursive algorithm and averaged into a final regression model. Due to the nature of ensemble models, the Boosted Decision Tree Regression model is expected to outperform the Multiple Linear Regression model, and due to its relative simplicity, compute faster than the Neural Network Regression model. However, the accuracy of the Boosted Decision Tree Regression model may or may not be better than the Neural Network Regression model. The accuracy will depend on the landscape of the dataset.

The evaluation of the current project however, suggests that the Boosted Decision Tree Regression model is more accurate than the Neural Network Regression model for the current dataset, making it the best model to predict and estimate product MTRs when compared to Multiple Linear Regression and Neural Network Regression.

Business Results – Prediction Model Example Use Case

Example: Using the above models with a VERSO client data, a simple example of an outcome follows: Using the Boosted Decision Tree Regression model, the out of warranty MTR predictions for the first 5 products in no specific order are shown below:

Table1. Out of warranty MTR predictions for the first 5 products.

Product	Predicted Number of MTRs	Predicted Number of MTRs (Rounded to the nearest whole number)
AAAAAAAA	0.000091	0
BBBBBBBBB	0.000445	0
CCCCCCCCC	4.996464	5
DDDDDDDD	105.997841	106
EEEEEEEEEE	119.545486	120

Since an estimation of the number of MTRs that may be triggered when the product is out of warranty is of interest, the chosen Boosted Decision Tree Regression model’s output are real values, instead of a classification output such as “yes” or “no”, or the probabilities associated with the classification. It is safe to round the predicted values to the nearest whole number, where zero means no MTRs triggered after the expiration of the warranty for that product.

Improvement to model

Improvements to the model can be accomplished by gathering more and diverse data to be used as predictors of the model. Specifically, predictors that are important to earlier phases of the business should be explored. For example, point of sale predictors are of value so the model can be trained to predict a product’s MTRs before an MTR request entry is created for that product within the VERSO logistics system. Nevertheless, the current model can serve as a useful starting point to which each product can be compared to.