Resampling methods are designed to add or remove examples from the training dataset in order to change the class distribution.<\/p>\n
Once the class distributions are more balanced, the suite of standard machine learning classification algorithms can be fit successfully on the transformed datasets.<\/p>\n
Oversampling methods duplicate or create new synthetic examples in the minority class, whereas undersampling methods delete or merge examples in the majority class. Both types of resampling can be effective when used in isolation, although can be more effective when both types of methods are used together.<\/p>\n
In this tutorial, you will discover how to combine oversampling and undersampling techniques for imbalanced classification.<\/p>\n
After completing this tutorial, you will know:<\/p>\n
- \n
- How to define a sequence of oversampling and undersampling methods to be applied to a training dataset or when evaluating a classifier model.<\/li>\n
- How to manually combine oversampling and undersampling methods for imbalanced classification.<\/li>\n
- How to use pre-defined and well-performing combinations of resampling methods for imbalanced classification.<\/li>\n<\/ul>\n
Discover SMOTE, one-class classification, cost-sensitive learning, threshold moving, and much more in my new book<\/a>, with 30 step-by-step tutorials and full Python source code.<\/p>\n
Let’s get started.<\/p>\n
![\"Combine](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/01\/Combine-Oversampling-and-Undersampling-for-Imbalanced-Classification.jpg\")
<\/p>\nCombine Oversampling and Undersampling for Imbalanced Classification
Photo by Radek Kucharski<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
This tutorial is divided into four parts; they are:<\/p>\n
- \n
- Binary Test Problem and Decision Tree Model<\/li>\n
- Imbalanced-Learn Library<\/li>\n
- Manually Combine Over- and Undersampling Methods\n
- \n
- Manually Combine Random Oversampling and Undersampling<\/li>\n
- Manually Combine SMOTE and Random Undersampling<\/li>\n<\/ol>\n<\/li>\n
- Use Predefined Combinations of Resampling Methods\n
- \n
- Combination of SMOTE and Tomek Links Undersampling<\/li>\n
- Combination of SMOTE and Edited Nearest Neighbors Undersampling<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n
Binary Test Problem and Decision Tree Model<\/h2>\n
Before we dive into combinations of oversampling and undersampling methods, let’s define a synthetic dataset and model.<\/p>\n
We can define a synthetic binary classification dataset using the make_classification() function<\/a> from the scikit-learn library.<\/p>\n
For example, we can create 10,000 examples with two input variables and a 1:100 class distribution as follows:<\/p>\n
...\r\n# define dataset\r\nX, y = make_classification(n_samples=10000, n_features=2, n_redundant=0,\r\n\tn_clusters_per_class=1, weights=[0.99], flip_y=0, random_state=1)<\/pre>\n
We can then create a scatter plot of the dataset via the scatter() Matplotlib function<\/a> to understand the spatial relationship of the examples in each class and their imbalance.<\/p>\n
...\r\n# scatter plot of examples by class label\r\nfor label, _ in counter.items():\r\n\trow_ix = where(y == label)[0]\r\n\tpyplot.scatter(X[row_ix, 0], X[row_ix, 1], label=str(label))\r\npyplot.legend()\r\npyplot.show()<\/pre>\n
Tying this together, the complete example of creating an imbalanced classification dataset and plotting the examples is listed below.<\/p>\n
# Generate and plot a synthetic imbalanced classification dataset\r\nfrom collections import Counter\r\nfrom sklearn.datasets import make_classification\r\nfrom matplotlib import pyplot\r\nfrom numpy import where\r\n# define dataset\r\nX, y = make_classification(n_samples=10000, n_features=2, n_redundant=0,\r\n\tn_clusters_per_class=1, weights=[0.99], flip_y=0, random_state=1)\r\n# summarize class distribution\r\ncounter = Counter(y)\r\nprint(counter)\r\n# scatter plot of examples by class label\r\nfor label, _ in counter.items():\r\n\trow_ix = where(y == label)[0]\r\n\tpyplot.scatter(X[row_ix, 0], X[row_ix, 1], label=str(label))\r\npyplot.legend()\r\npyplot.show()<\/pre>\n
Running the example first summarizes the class distribution, showing an approximate 1:100 class distribution with about 10,000 examples with class 0 and 100 with class 1.<\/p>\n
Counter({0: 9900, 1: 100})<\/pre>\nNext, a scatter plot is created showing all of the examples in the dataset. We can see a large mass of examples for class 0 (blue) and a small number of examples for class 1 (orange).<\/p>\n
We can also see that the classes overlap with some examples from class 1 clearly within the part of the feature space that belongs to class 0.<\/p>\n
![\"Scatter](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/10\/Scatter-Plot-of-Imbalanced-Classification-Dataset-1.png\")
<\/p>\nScatter Plot of Imbalanced Classification Dataset<\/p>\n<\/div>\n
We can fit a DecisionTreeClassifier model<\/a> on this dataset. It is a good model to test because it is sensitive to the class distribution in the training dataset.<\/p>\n
...\r\n# define model\r\nmodel = DecisionTreeClassifier()<\/pre>\n
We can evaluate the model using repeated stratified k-fold cross-validation<\/a> with three repeats and 10 folds.<\/p>\n