Resampling methods are designed to change the composition of a training dataset for an imbalanced classification task.<\/p>\n
Most of the attention of resampling methods for imbalanced classification is put on oversampling the minority class. Nevertheless, a suite of techniques has been developed for undersampling the majority class that can be used in conjunction with effective oversampling methods.<\/p>\n
There are many different types of undersampling techniques, although most can be grouped into those that select examples to keep in the transformed dataset, those that select examples to delete, and hybrids that combine both types of methods.<\/p>\n
In this tutorial, you will discover undersampling methods for imbalanced classification.<\/p>\n
After completing this tutorial, you will know:<\/p>\n
- \n
- How to use the Near-Miss and Condensed Nearest Neighbor Rule methods that select examples to keep from the majority class.<\/li>\n
- How to use Tomek Links and the Edited Nearest Neighbors Rule methods that select examples to delete from the majority class.<\/li>\n
- How to use One-Sided Selection and the Neighborhood Cleaning Rule that combine methods for choosing examples to keep and delete from the majority class.<\/li>\n<\/ul>\n
Discover SMOTE, one-class classification, cost-sensitive learning, threshold moving, and much more in my new book<\/a>, with 30 step-by-step tutorials and full Python source code.<\/p>\n
Let’s get started.<\/p>\n
![\"How](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/01\/How-to-Use-Undersampling-Algorithms-for-Imbalanced-Classification.jpg\")
<\/p>\nHow to Use Undersampling Algorithms for Imbalanced Classification
Photo by nuogein<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
This tutorial is divided into five parts; they are:<\/p>\n
- \n
- Undersampling for Imbalanced Classification<\/li>\n
- Imbalanced-Learn Library<\/li>\n
- Methods that Select Examples to Keep\n
- \n
- Near Miss Undersampling<\/li>\n
- Condensed Nearest Neighbor Rule for Undersampling<\/li>\n<\/ol>\n<\/li>\n
- Methods that Select Examples to Delete\n
- \n
- Tomek Links for Undersampling<\/li>\n
- Edited Nearest Neighbors Rule for Undersampling<\/li>\n<\/ol>\n<\/li>\n
- Combinations of Keep and Delete Methods\n
- \n
- One-Sided Selection for Undersampling<\/li>\n
- Neighborhood Cleaning Rule for Undersampling<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n
Undersampling for Imbalanced Classification<\/h2>\n
Undersampling refers to a group of techniques designed to balance the class distribution for a classification dataset that has a skewed class distribution.<\/p>\n
An imbalanced class distribution will have one or more classes with few examples (the minority classes) and one or more classes with many examples (the majority classes). It is best understood in the context of a binary (two-class) classification problem where class 0 is the majority class and class 1 is the minority class.<\/p>\n
Undersampling techniques remove examples from the training dataset that belong to the majority class in order to better balance the class distribution, such as reducing the skew from a 1:100 to a 1:10, 1:2, or even a 1:1 class distribution. This is different from oversampling that involves adding examples to the minority class in an effort to reduce the skew in the class distribution.<\/p>\n
\n
… undersampling, that consists of reducing the data by eliminating examples belonging to the majority class with the objective of equalizing the number of examples of each class …<\/p>\n<\/blockquote>\n
— Page 82, Learning from Imbalanced Data Sets<\/a>, 2018.<\/p>\n
Undersampling methods can be used directly on a training dataset that can then, in turn, be used to fit a machine learning model. Typically, undersampling methods are used in conjunction with an oversampling technique for the minority class, and this combination often results in better performance than using oversampling or undersampling alone on the training dataset.<\/p>\n
The simplest undersampling technique involves randomly selecting examples from the majority class and deleting them from the training dataset. This is referred to as random undersampling. Although simple and effective, a limitation of this technique is that examples are removed without any concern for how useful or important they might be in determining the decision boundary between the classes. This means it is possible, or even likely, that useful information will be deleted.<\/p>\n
\n
The major drawback of random undersampling is that this method can discard potentially useful data that could be important for the induction process. The removal of data is a critical decision to be made, hence many the proposal of undersampling use heuristics in order to overcome the limitations of the non- heuristics decisions.<\/p>\n<\/blockquote>\n
— Page 83, Learning from Imbalanced Data Sets<\/a>, 2018.<\/p>\n
An extension of this approach is to be more discerning regarding the examples from the majority class that are deleted. This typically involves heuristics or learning models that attempt to identify redundant examples for deletion or useful examples for non-deletion.<\/p>\n
There are many undersampling techniques that use these types of heuristics. In the following sections, we will review some of the more common methods and develop an intuition for their operation on a synthetic imbalanced binary classification dataset.<\/p>\n
We can define a synthetic binary classification dataset using the make_classification() function<\/a> from the scikit-learn library. For example, we can create 10,000 examples with two input variables and a 1:100 distribution as follows:<\/p>\n
...\r\n# define dataset\r\nX, y = make_classification(n_samples=10000, n_features=2, n_redundant=0,\r\nn_clusters_per_class=1, weights=[0.99], flip_y=0, random_state=1)<\/pre>\n
We can then create a scatter plot of the dataset via the scatter() Matplotlib function<\/a> to understand the spatial relationship of the examples in each class and their imbalance.<\/p>\n
...\r\n# scatter plot of examples by class label\r\nfor label, _ in counter.items():\r\n\trow_ix = where(y == label)[0]\r\n\tpyplot.scatter(X[row_ix, 0], X[row_ix, 1], label=str(label))\r\npyplot.legend()\r\npyplot.show()<\/pre>\n
Tying this together, the complete example of creating an imbalanced classification dataset and plotting the examples is listed below.<\/p>\n
# Generate and plot a synthetic imbalanced classification dataset\r\nfrom collections import Counter\r\nfrom sklearn.datasets import make_classification\r\nfrom matplotlib import pyplot\r\nfrom numpy import where\r\n# define dataset\r\nX, y = make_classification(n_samples=10000, n_features=2, n_redundant=0,\r\n\tn_clusters_per_class=1, weights=[0.99], flip_y=0, random_state=1)\r\n# summarize class distribution\r\ncounter = Counter(y)\r\nprint(counter)\r\n# scatter plot of examples by class label\r\nfor label, _ in counter.items():\r\n\trow_ix = where(y == label)[0]\r\n\tpyplot.scatter(X[row_ix, 0], X[row_ix, 1], label=str(label))\r\npyplot.legend()\r\npyplot.show()<\/pre>\n
Running the example first summarizes the class distribution, showing an approximate 1:100 class distribution with about 10,000 examples with class 0 and 100 with class 1.<\/p>\n
Counter({0: 9900, 1: 100})<\/pre>\nNext, a scatter plot is created showing all of the examples in the dataset. We can see a large mass of examples for class 0 (blue) and a small number of examples for class 1 (orange). We can also see that the classes overlap with some examples from class 1 clearly within the part of the feature space that belongs to class 0.<\/p>\n
![\"Scatter](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/10\/Scatter-Plot-of-Imbalanced-Classification-Dataset.png\")
<\/p>\nScatter Plot of Imbalanced Classification Dataset<\/p>\n<\/div>\n
This plot provides the starting point for developing the intuition for the effect that different undersampling techniques have on the majority class.<\/p>\n
Next, we can begin to review popular undersampling methods made available via the imbalanced-learn Python library<\/a>.<\/p>\n
There are many different methods to choose from. We will divide them into methods that select what examples from the majority class to keep, methods that select examples to delete, and combinations of both approaches.<\/p>\n<\/p>\n
<\/div>\n<\/p>\n Want to Get Started With Imbalance Classification?<\/h3>\n
Take my free 7-day email crash course now (with sample code).<\/p>\n
Click to sign-up and also get a free PDF Ebook version of the course.<\/p>\n