The decision tree algorithm is effective for balanced classification, although it does not perform well on imbalanced datasets.<\/p>\n
The split points of the tree are chosen to best separate examples into two groups with minimum mixing. When both groups are dominated by examples from one class, the criterion used to select a split point will see good separation, when in fact, the examples from the minority class are being ignored.<\/p>\n
This problem can be overcome by modifying the criterion used to evaluate split points to take the importance of each class into account, referred to generally as the weighted split-point or weighted decision tree.<\/p>\n
In this tutorial, you will discover the weighted decision tree for imbalanced classification.<\/p>\n
After completing this tutorial, you will know:<\/p>\n
- \n
- How the standard decision tree algorithm does not support imbalanced classification.<\/li>\n
- How the decision tree algorithm can be modified to weight model error by class weight when selecting splits.<\/li>\n
- How to configure class weight for the decision tree algorithm and how to grid search different class weight configurations.<\/li>\n<\/ul>\n
Discover SMOTE, one-class classification, cost-sensitive learning, threshold moving, and much more in my new book<\/a>, with 30 step-by-step tutorials and full Python source code.<\/p>\n
Let’s get started.<\/p>\n
![\"How](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/01\/How-to-Implement-Weighted-Decision-Trees-for-Imbalanced-Classification.jpg\")
<\/p>\nHow to Implement Weighted Decision Trees for Imbalanced Classification
Photo by Bonnie Moreland<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
This tutorial is divided into four parts; they are:<\/p>\n
- \n
- Imbalanced Classification Dataset<\/li>\n
- Decision Trees for Imbalanced Classification<\/li>\n
- Weighted Decision Trees With Scikit-Learn<\/li>\n
- Grid Search Weighted Decision Trees<\/li>\n<\/ol>\n
Imbalanced Classification Dataset<\/h2>\n
Before we dive into the modification of decision for imbalanced classification, let’s first define an imbalanced classification dataset.<\/p>\n
We can use the make_classification() function<\/a> to define a synthetic imbalanced two-class classification dataset. We will generate 10,000 examples with an approximate 1:100 minority to majority class ratio.<\/p>\n
...\r\n# define dataset\r\nX, y = make_classification(n_samples=10000, n_features=2, n_redundant=0,\r\n\tn_clusters_per_class=1, weights=[0.99], flip_y=0, random_state=3)<\/pre>\n
Once generated, we can summarize the class distribution to confirm that the dataset was created as we expected.<\/p>\n
...\r\n# summarize class distribution\r\ncounter = Counter(y)\r\nprint(counter)<\/pre>\n
Finally, we can create a scatter plot of the examples and color them by class label to help understand the challenge of classifying examples from this dataset.<\/p>\n
...\r\n# scatter plot of examples by class label\r\nfor label, _ in counter.items():\r\n\trow_ix = where(y == label)[0]\r\n\tpyplot.scatter(X[row_ix, 0], X[row_ix, 1], label=str(label))\r\npyplot.legend()\r\npyplot.show()<\/pre>\n
Tying this together, the complete example of generating the synthetic dataset and plotting the examples is listed below.<\/p>\n
# Generate and plot a synthetic imbalanced classification dataset\r\nfrom collections import Counter\r\nfrom sklearn.datasets import make_classification\r\nfrom matplotlib import pyplot\r\nfrom numpy import where\r\n# define dataset\r\nX, y = make_classification(n_samples=10000, n_features=2, n_redundant=0,\r\n\tn_clusters_per_class=1, weights=[0.99], flip_y=0, random_state=3)\r\n# summarize class distribution\r\ncounter = Counter(y)\r\nprint(counter)\r\n# scatter plot of examples by class label\r\nfor label, _ in counter.items():\r\n\trow_ix = where(y == label)[0]\r\n\tpyplot.scatter(X[row_ix, 0], X[row_ix, 1], label=str(label))\r\npyplot.legend()\r\npyplot.show()<\/pre>\n
Running the example first creates the dataset and summarizes the class distribution.<\/p>\n
We can see that the dataset has an approximate 1:100 class distribution with a little less than 10,000 examples in the majority class and 100 in the minority class.<\/p>\n
Counter({0: 9900, 1: 100})<\/pre>\nNext, a scatter plot of the dataset is created showing the large mass of examples for the majority class (blue) and a small number of examples for the minority class (orange), with some modest class overlap.<\/p>\n
![\"Scatter](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/11\/Scatter-Plot-of-Binary-Classification-Dataset-with-1-to-100-Class-Imbalance-1.png\")
<\/p>\nScatter Plot of Binary Classification Dataset With 1 to 100 Class Imbalance<\/p>\n<\/div>\n
Next, we can fit a standard decision tree model on the dataset.<\/p>\n
A decision tree can be defined using the DecisionTreeClassifier<\/a> class in the scikit-learn library.<\/p>\n
...\r\n# define model\r\nmodel = DecisionTreeClassifier()<\/pre>\n