Numerical input variables may have a highly skewed or non-standard distribution.<\/p>\n
This could be caused by outliers in the data, multi-modal distributions, highly exponential distributions, and more.<\/p>\n
Many machine learning algorithms prefer or perform better when numerical input variables have a standard probability distribution.<\/p>\n
The discretization transform provides an automatic way to change a numeric input variable to have a different data distribution, which in turn can be used as input to a predictive model.<\/p>\n
In this tutorial, you will discover how to use discretization transforms to map numerical values to discrete categories for machine learning<\/p>\n
After completing this tutorial, you will know:<\/p>\n
- \n
- Many machine learning algorithms prefer or perform better when numerical with non-standard probability distributions are made discrete.<\/li>\n
- Discretization transforms are a technique for transforming numerical input or output variables to have discrete ordinal labels.<\/li>\n
- How to use the KBinsDiscretizer to change the structure and distribution of numeric variables to improve the performance of predictive models.<\/li>\n<\/ul>\n
Let’s get started.<\/p>\n
![\"How](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/05\/How-to-Use-Discretization-Transforms-for-Machine-Learning.jpg\")
<\/p>\nHow to Use Discretization Transforms for Machine Learning
Photo by Kate Russell<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
This tutorial is divided into six parts; they are:<\/p>\n
- \n
- Change Data Distribution<\/li>\n
- Discretization Transforms<\/li>\n
- Sonar Dataset<\/li>\n
- Uniform Discretization Transform<\/li>\n
- K-means Discretization Transform<\/li>\n
- Quantile Discretization Transform<\/li>\n<\/ol>\n
Change Data Distribution<\/h2>\n
Some machine learning algorithms may prefer or require categorical or ordinal input variables, such as some decision tree and rule-based algorithms.<\/p>\n
\n
Some classification and clustering algorithms deal with nominal attributes only and cannot handle ones measured on a numeric scale.<\/p>\n<\/blockquote>\n
— Page 296, Data Mining: Practical Machine Learning Tools and Techniques<\/a>, 4th edition, 2016.<\/p>\n
Further, the performance of many machine learning algorithms degrades for variables that have non-standard probability distributions.<\/p>\n
This applies both to real-valued input variables in the case of classification and regression tasks, and real-valued target variables in the case of regression tasks.<\/p>\n
Some input variables may have a highly skewed distribution, such as an exponential distribution where the most common observations are bunched together. Some input variables may have outliers that cause the distribution to be highly spread.<\/p>\n
These concerns and others, like non-standard distributions and multi-modal distributions, can make a dataset challenging to model with a range of machine learning models.<\/p>\n
As such, it is often desirable to transform each input variable to have a standard probability distribution.<\/p>\n
One approach is to use transform of the numerical variable to have a discrete probability distribution where each numerical value is assigned a label and the labels have an ordered (ordinal) relationship.<\/p>\n
This is called a binning<\/strong> or a discretization transform<\/strong> and can improve the performance of some machine learning models for datasets by making the probability distribution of numerical input variables discrete.<\/p>\n
Discretization Transforms<\/h2>\n
A discretization transform<\/a> will map numerical variables onto discrete values.<\/p>\n
\n
Binning, also known as categorization or discretization, is the process of translating a quantitative variable into a set of two or more qualitative buckets (i.e., categories).<\/p>\n<\/blockquote>\n
— Page 129, Feature Engineering and Selection<\/a>, 2019.<\/p>\n
Values for the variable are grouped together into discrete bins and each bin is assigned a unique integer such that the ordinal relationship between the bins is preserved.<\/p>\n
The use of bins is often referred to as binning or k<\/em>-bins, where k<\/em> refers to the number of groups to which a numeric variable is mapped.<\/p>\n
The mapping provides a high-order ranking of values that can smooth out the relationships between observations. The transformation can be applied to each numeric input variable in the training dataset and then provided as input to a machine learning model to learn a predictive modeling task.<\/p>\n
\n
The determination of the bins must be included inside of the resampling process.<\/p>\n<\/blockquote>\n
— Page 132, Feature Engineering and Selection<\/a>, 2019.<\/p>\n
Different methods for grouping the values into k discrete bins can be used; common techniques include:<\/p>\n
- \n
- Uniform<\/strong>: Each bin has the same width in the span of possible values for the variable.<\/li>\n
- Quantile<\/strong>: Each bin has the same number of values, split based on percentiles.<\/li>\n
- Clustered<\/strong>: Clusters are identified and examples are assigned to each group.<\/li>\n<\/ul>\n
The discretization transform is available in the scikit-learn Python machine learning library via the KBinsDiscretizer class<\/a>.<\/p>\n
The “strategy<\/em>” argument controls the manner in which the input variable is divided, as either “uniform<\/em>,” “quantile<\/em>,” or “kmeans<\/em>.”<\/p>\n
The “n_bins<\/em>” argument controls the number of bins that will be created and must be set based on the choice of strategy, e.g. “uniform<\/em>” is flexible, “quantile<\/em>” must have a “n_bins<\/em>” less than the number of observations or sensible percentiles, and “kmeans<\/em>” must use a value for the number of clusters that can be reasonably found.<\/p>\n
The “encode<\/em>” argument controls whether the transform will map each value to an integer value by setting “ordinal<\/em>” or a one-hot encoding “onehot<\/em>.” An ordinal encoding is almost always preferred, although a one-hot encoding may allow a model to learn non-ordinal relationships between the groups, such as in the case of k<\/em>-means clustering strategy.<\/p>\n
We can demonstrate the KBinsDiscretizer<\/em> with a small worked example. We can generate a sample of random Gaussian numbers<\/a>. The KBinsDiscretizer can then be used to convert the floating values into fixed number of discrete categories with an ranked ordinal relationship.<\/p>\n
The complete example is listed below.<\/p>\n
# demonstration of the discretization transform\r\nfrom numpy.random import randn\r\nfrom sklearn.preprocessing import KBinsDiscretizer\r\nfrom matplotlib import pyplot\r\n# generate gaussian data sample\r\ndata = randn(1000)\r\n# histogram of the raw data\r\npyplot.hist(data, bins=25)\r\npyplot.show()\r\n# reshape data to have rows and columns\r\ndata = data.reshape((len(data),1))\r\n# discretization transform the raw data\r\nkbins = KBinsDiscretizer(n_bins=10, encode='ordinal', strategy='uniform')\r\ndata_trans = kbins.fit_transform(data)\r\n# summarize first few rows\r\nprint(data_trans[:10, :])\r\n# histogram of the transformed data\r\npyplot.hist(data_trans, bins=10)\r\npyplot.show()<\/pre>\n
Running the example first creates a sample of 1,000 random Gaussian floating-point values and plots the data as a histogram.<\/p>\n
![\"Histogram](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/05\/Histogram-of-Data-With-a-Gaussian-Distribution.png\")
<\/p>\nHistogram of Data With a Gaussian Distribution<\/p>\n<\/div>\n
Next the KBinsDiscretizer is used to map the numerical values to categorical values. We configure the transform to create 10 categories (0 to 9), to output the result in ordinal format (integers) and to divide the range of the input data uniformly.<\/p>\n
A sample of the transformed data is printed, clearly showing the integer format of the data as expected.<\/p>\n
[[5.]\r\n [3.]\r\n [2.]\r\n [6.]\r\n [7.]\r\n [5.]\r\n [3.]\r\n [4.]\r\n [4.]\r\n [2.]]<\/pre>\n
Finally, a histogram is created showing the 10 discrete categories and how the observations are distributed across these groups, following the same pattern as the original data with a Gaussian shape.<\/p>\n
![\"Histogram](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/05\/Historam-of-Transformed-Data-With-Discrete-Categories.png\")
<\/p>\nHistogram of Transformed Data With Discrete Categories<\/p>\n<\/div>\n
In the following sections will take a closer look at how to use the discretization transform on a real dataset.<\/p>\n
Next, let’s introduce the dataset.<\/p>\n
Sonar Dataset<\/h2>\n
The sonar dataset is a standard machine learning dataset for binary classification.<\/p>\n
It involves 60 real-valued inputs and a two-class target variable. There are 208 examples in the dataset and the classes are reasonably balanced.<\/p>\n
A baseline classification algorithm can achieve a classification accuracy of about 53.4 percent using repeated stratified 10-fold cross-validation. Top performance<\/a> on this dataset is about 88 percent using repeated stratified 10-fold cross-validation.<\/p>\n
The dataset describes radar returns of rocks or simulated mines.<\/p>\n
You can learn more about the dataset from here:<\/p>\n
- \n
- Sonar Dataset<\/a><\/li>\n
- Sonar Dataset Description<\/a><\/li>\n<\/ul>\n
No need to download the dataset; we will download it automatically from our worked examples.<\/p>\n
First, let’s load and summarize the dataset. The complete example is listed below.<\/p>\n
# load and summarize the sonar dataset\r\nfrom pandas import read_csv\r\nfrom pandas.plotting import scatter_matrix\r\nfrom matplotlib import pyplot\r\n# Load dataset\r\nurl = \"https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/sonar.csv\"\r\ndataset = read_csv(url, header=None)\r\n# summarize the shape of the dataset\r\nprint(dataset.shape)\r\n# summarize each variable\r\nprint(dataset.describe())\r\n# histograms of the variables\r\ndataset.hist()\r\npyplot.show()<\/pre>\n
Running the example first summarizes the shape of the loaded dataset.<\/p>\n
This confirms the 60 input variables, one output variable, and 208 rows of data.<\/p>\n
A statistical summary of the input variables is provided showing that values are numeric and range approximately from 0 to 1.<\/p>\n
(208, 61)\r\n 0 1 2 ... 57 58 59\r\ncount 208.000000 208.000000 208.000000 ... 208.000000 208.000000 208.000000\r\nmean 0.029164 0.038437 0.043832 ... 0.007949 0.007941 0.006507\r\nstd 0.022991 0.032960 0.038428 ... 0.006470 0.006181 0.005031\r\nmin 0.001500 0.000600 0.001500 ... 0.000300 0.000100 0.000600\r\n25% 0.013350 0.016450 0.018950 ... 0.003600 0.003675 0.003100\r\n50% 0.022800 0.030800 0.034300 ... 0.005800 0.006400 0.005300\r\n75% 0.035550 0.047950 0.057950 ... 0.010350 0.010325 0.008525\r\nmax 0.137100 0.233900 0.305900 ... 0.044000 0.036400 0.043900\r\n\r\n[8 rows x 60 columns]<\/pre>\n
Finally, a histogram is created for each input variable.<\/p>\n
If we ignore the clutter of the plots and focus on the histograms themselves, we can see that many variables have a skewed distribution.<\/p>\n
![\"Histogram](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/02\/Histogram-Plots-of-Input-Variables-for-the-Sonar-Binary-Classification-Dataset-1.png\")
<\/p>\nHistogram Plots of Input Variables for the Sonar Binary Classification Dataset<\/p>\n<\/div>\n
Next, let’s fit and evaluate a machine learning model on the raw dataset.<\/p>\n
We will use a k-nearest neighbor algorithm<\/a> with default hyperparameters and evaluate it using repeated stratified k-fold cross-validation<\/a>.<\/p>\n
The complete example is listed below.<\/p>\n
# evaluate knn on the raw sonar dataset\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom pandas import read_csv\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.model_selection import RepeatedStratifiedKFold\r\nfrom sklearn.neighbors import KNeighborsClassifier\r\nfrom sklearn.preprocessing import LabelEncoder\r\nfrom matplotlib import pyplot\r\n# load dataset\r\nurl = \"https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/sonar.csv\"\r\ndataset = read_csv(url, header=None)\r\ndata = dataset.values\r\n# separate into input and output columns\r\nX, y = data[:, :-1], data[:, -1]\r\n# ensure inputs are floats and output is an integer label\r\nX = X.astype('float32')\r\ny = LabelEncoder().fit_transform(y.astype('str'))\r\n# define and configure the model\r\nmodel = KNeighborsClassifier()\r\n# evaluate the model\r\ncv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\r\nn_scores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1, error_score='raise')\r\n# report model performance\r\nprint('Accuracy: %.3f (%.3f)' % (mean(n_scores), std(n_scores)))<\/pre>\nRunning the example evaluates a KNN model on the raw sonar dataset.<\/p>\n
We can see that the model achieved a mean classification accuracy of about 79.7 percent, showing that it has skill (better than 53.4 percent) and is in the ball-park of good performance (88 percent).<\/p>\n
Accuracy: 0.797 (0.073)<\/pre>\n
Next, let’s explore a uniform discretization transform of the dataset.<\/p>\n
Uniform Discretization Transform<\/h2>\n
A uniform discretization transform will preserve the probability distribution of each input variable but will make it discrete with the specified number of ordinal groups or labels.<\/p>\n
We can apply the uniform discretization transform using the KBinsDiscretizer<\/a> class and setting the “strategy<\/em>” argument to “uniform<\/em>.” We must also set the desired number of bins set via the “n_bins<\/em>” argument; in this case, we will use 10.<\/p>\n
Once defined, we can call the fit_transform()<\/em> function and pass it our dataset to create a quantile transformed version of our dataset.<\/p>\n
...\r\n# perform a uniform discretization transform of the dataset\r\ntrans = KBinsDiscretizer(n_bins=10, encode='ordinal', strategy='uniform')\r\ndata = trans.fit_transform(data)<\/pre>\n
Let’s try it on our sonar dataset.<\/p>\n
The complete example of creating a uniform discretization transform of the sonar dataset and plotting histograms of the result is listed below.<\/p>\n
# visualize a uniform ordinal discretization transform of the sonar dataset\r\nfrom pandas import read_csv\r\nfrom pandas import DataFrame\r\nfrom pandas.plotting import scatter_matrix\r\nfrom sklearn.preprocessing import KBinsDiscretizer\r\nfrom matplotlib import pyplot\r\n# load dataset\r\nurl = \"https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/sonar.csv\"\r\ndataset = read_csv(url, header=None)\r\n# retrieve just the numeric input values\r\ndata = dataset.values[:, :-1]\r\n# perform a uniform discretization transform of the dataset\r\ntrans = KBinsDiscretizer(n_bins=10, encode='ordinal', strategy='uniform')\r\ndata = trans.fit_transform(data)\r\n# convert the array back to a dataframe\r\ndataset = DataFrame(data)\r\n# histograms of the variables\r\ndataset.hist()\r\npyplot.show()<\/pre>\n
Running the example transforms the dataset and plots histograms of each input variable.<\/p>\n
We can see that the shape of the histograms generally matches the shape of the raw dataset, although in this case, each variable has a fixed number of 10 values or ordinal groups.<\/p>\n
![\"Histogram](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/02\/Histogram-Plots-of-Uniform-Discretization-Transformed-Input-Variables-for-the-Sonar-Dataset.png\")
<\/p>\nHistogram Plots of Uniform Discretization Transformed Input Variables for the Sonar Dataset<\/p>\n<\/div>\n
Next, let’s evaluate the same KNN model as the previous section, but in this case on a uniform discretization transform of the dataset.<\/p>\n
The complete example is listed below.<\/p>\n
# evaluate knn on the sonar dataset with uniform ordinal discretization transform\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom pandas import read_csv\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.model_selection import RepeatedStratifiedKFold\r\nfrom sklearn.neighbors import KNeighborsClassifier\r\nfrom sklearn.preprocessing import LabelEncoder\r\nfrom sklearn.preprocessing import KBinsDiscretizer\r\nfrom sklearn.pipeline import Pipeline\r\nfrom matplotlib import pyplot\r\n# load dataset\r\nurl = \"https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/sonar.csv\"\r\ndataset = read_csv(url, header=None)\r\ndata = dataset.values\r\n# separate into input and output columns\r\nX, y = data[:, :-1], data[:, -1]\r\n# ensure inputs are floats and output is an integer label\r\nX = X.astype('float32')\r\ny = LabelEncoder().fit_transform(y.astype('str'))\r\n# define the pipeline\r\ntrans = KBinsDiscretizer(n_bins=10, encode='ordinal', strategy='uniform')\r\nmodel = KNeighborsClassifier()\r\npipeline = Pipeline(steps=[('t', trans), ('m', model)])\r\n# evaluate the pipeline\r\ncv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\r\nn_scores = cross_val_score(pipeline, X, y, scoring='accuracy', cv=cv, n_jobs=-1, error_score='raise')\r\n# report pipeline performance\r\nprint('Accuracy: %.3f (%.3f)' % (mean(n_scores), std(n_scores)))<\/pre>\nRunning the example, we can see that the uniform discretization transform results in a lift in performance from 79.7 percent accuracy without the transform to about 82.7 percent with the transform.<\/p>\n
Accuracy: 0.827 (0.082)<\/pre>\n
Next, let’s take a closer look at the k-means discretization transform.<\/p>\n
K-means Discretization Transform<\/h2>\n
A K-means discretization transform will attempt to fit k clusters for each input variable and then assign each observation to a cluster.<\/p>\n
Unless the empirical distribution of the variable is complex, the number of clusters is likely to be small, such as 3-to-5.<\/p>\n
We can apply the K-means discretization transform using the KBinsDiscretizer<\/em> class and setting the “strategy<\/em>” argument to “kmeans<\/em>.” We must also set the desired number of bins set via the “n_bins<\/em>” argument; in this case, we will use three.<\/p>\n
Once defined, we can call the fit_transform()<\/em> function and pass it to our dataset to create a quantile transformed version of our dataset.<\/p>\n
...\r\n# perform a k-means discretization transform of the dataset\r\ntrans = KBinsDiscretizer(n_bins=3, encode='ordinal', strategy='kmeans')\r\ndata = trans.fit_transform(data)<\/pre>\n
Let’s try it on our sonar dataset.<\/p>\n
The complete example of creating a K-means discretization transform of the sonar dataset and plotting histograms of the result is listed below.<\/p>\n
# visualize a k-means ordinal discretization transform of the sonar dataset\r\nfrom pandas import read_csv\r\nfrom pandas import DataFrame\r\nfrom pandas.plotting import scatter_matrix\r\nfrom sklearn.preprocessing import KBinsDiscretizer\r\nfrom matplotlib import pyplot\r\n# load dataset\r\nurl = \"https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/sonar.csv\"\r\ndataset = read_csv(url, header=None)\r\n# retrieve just the numeric input values\r\ndata = dataset.values[:, :-1]\r\n# perform a k-means discretization transform of the dataset\r\ntrans = KBinsDiscretizer(n_bins=3, encode='ordinal', strategy='kmeans')\r\ndata = trans.fit_transform(data)\r\n# convert the array back to a dataframe\r\ndataset = DataFrame(data)\r\n# histograms of the variables\r\ndataset.hist()\r\npyplot.show()<\/pre>\n
Running the example transforms the dataset and plots histograms of each input variable.<\/p>\n
We can see that the observations for each input variable are organized into one of three groups, some of which appear to be quite even in terms of observations, and others much less so.<\/p>\n
![\"Histogram](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/02\/Histogram-Plots-of-K-means-Discretization-Transformed-Input-Variables-for-the-Sonar-Dataset.png\")
<\/p>\nHistogram Plots of K-means Discretization Transformed Input Variables for the Sonar Dataset<\/p>\n<\/div>\n
Next, let’s evaluate the same KNN model as the previous section, but in this case on a K-means discretization transform of the dataset.<\/p>\n
The complete example is listed below.<\/p>\n
# evaluate knn on the sonar dataset with k-means ordinal discretization transform\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom pandas import read_csv\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.model_selection import RepeatedStratifiedKFold\r\nfrom sklearn.neighbors import KNeighborsClassifier\r\nfrom sklearn.preprocessing import LabelEncoder\r\nfrom sklearn.preprocessing import KBinsDiscretizer\r\nfrom sklearn.pipeline import Pipeline\r\nfrom matplotlib import pyplot\r\n# load dataset\r\nurl = \"https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/sonar.csv\"\r\ndataset = read_csv(url, header=None)\r\ndata = dataset.values\r\n# separate into input and output columns\r\nX, y = data[:, :-1], data[:, -1]\r\n# ensure inputs are floats and output is an integer label\r\nX = X.astype('float32')\r\ny = LabelEncoder().fit_transform(y.astype('str'))\r\n# define the pipeline\r\ntrans = KBinsDiscretizer(n_bins=3, encode='ordinal', strategy='kmeans')\r\nmodel = KNeighborsClassifier()\r\npipeline = Pipeline(steps=[('t', trans), ('m', model)])\r\n# evaluate the pipeline\r\ncv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\r\nn_scores = cross_val_score(pipeline, X, y, scoring='accuracy', cv=cv, n_jobs=-1, error_score='raise')\r\n# report pipeline performance\r\nprint('Accuracy: %.3f (%.3f)' % (mean(n_scores), std(n_scores)))<\/pre>\nRunning the example, we can see that the K-means discretization transform results in a lift in performance from 79.7 percent accuracy without the transform to about 81.4 percent with the transform, although slightly less than the uniform distribution in the previous section.<\/p>\n
Accuracy: 0.814 (0.088)<\/pre>\n
Next, let’s take a closer look at the quantile discretization transform.<\/p>\n
Quantile Discretization Transform<\/h2>\n
A quantile discretization transform will attempt to split the observations for each input variable into k groups, where the number of observations assigned to each group is approximately equal.<\/p>\n
Unless there are a large number of observations or a complex empirical distribution, the number of bins must be kept small, such as 5-10.<\/p>\n
We can apply the quantile discretization transform using the KBinsDiscretizer<\/em> class and setting the “strategy<\/em>” argument to “quantile<\/em>.” We must also set the desired number of bins set via the “n_bins<\/em>” argument; in this case, we will use 10.<\/p>\n
...\r\n# perform a quantile discretization transform of the dataset\r\ntrans = KBinsDiscretizer(n_bins=10, encode='ordinal', strategy='quantile')\r\ndata = trans.fit_transform(data)<\/pre>\n
The example below applies the quantile discretization transform and creates histogram plots of each of the transformed variables.<\/p>\n
# visualize a quantile ordinal discretization transform of the sonar dataset\r\nfrom pandas import read_csv\r\nfrom pandas import DataFrame\r\nfrom pandas.plotting import scatter_matrix\r\nfrom sklearn.preprocessing import KBinsDiscretizer\r\nfrom matplotlib import pyplot\r\n# load dataset\r\nurl = \"https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/sonar.csv\"\r\ndataset = read_csv(url, header=None)\r\n# retrieve just the numeric input values\r\ndata = dataset.values[:, :-1]\r\n# perform a quantile discretization transform of the dataset\r\ntrans = KBinsDiscretizer(n_bins=10, encode='ordinal', strategy='quantile')\r\ndata = trans.fit_transform(data)\r\n# convert the array back to a dataframe\r\ndataset = DataFrame(data)\r\n# histograms of the variables\r\ndataset.hist()\r\npyplot.show()<\/pre>\n
Running the example transforms the dataset and plots histograms of each input variable.<\/p>\n
We can see that the histograms all show a uniform probability distribution for each input variable, where each of the 10 groups has the same number of observations.<\/p>\n
![\"Histogram](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/02\/Histogram-Plots-of-Quantile-Discretization-Transformed-Input-Variables-for-the-Sonar-Dataset.png\")
<\/p>\nHistogram Plots of Quantile Discretization Transformed Input Variables for the Sonar Dataset<\/p>\n<\/div>\n
Next, let’s evaluate the same KNN model as the previous section, but in this case, on a quantile discretization transform of the raw dataset.<\/p>\n
The complete example is listed below.<\/p>\n
# evaluate knn on the sonar dataset with quantile ordinal discretization transform\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom pandas import read_csv\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.model_selection import RepeatedStratifiedKFold\r\nfrom sklearn.neighbors import KNeighborsClassifier\r\nfrom sklearn.preprocessing import LabelEncoder\r\nfrom sklearn.preprocessing import KBinsDiscretizer\r\nfrom sklearn.pipeline import Pipeline\r\nfrom matplotlib import pyplot\r\n# load dataset\r\nurl = \"https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/sonar.csv\"\r\ndataset = read_csv(url, header=None)\r\ndata = dataset.values\r\n# separate into input and output columns\r\nX, y = data[:, :-1], data[:, -1]\r\n# ensure inputs are floats and output is an integer label\r\nX = X.astype('float32')\r\ny = LabelEncoder().fit_transform(y.astype('str'))\r\n# define the pipeline\r\ntrans = KBinsDiscretizer(n_bins=10, encode='ordinal', strategy='quantile')\r\nmodel = KNeighborsClassifier()\r\npipeline = Pipeline(steps=[('t', trans), ('m', model)])\r\n# evaluate the pipeline\r\ncv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\r\nn_scores = cross_val_score(pipeline, X, y, scoring='accuracy', cv=cv, n_jobs=-1, error_score='raise')\r\n# report pipeline performance\r\nprint('Accuracy: %.3f (%.3f)' % (mean(n_scores), std(n_scores)))<\/pre>\nRunning the example, we can see that the uniform transform results in a lift in performance from 79.7 percent accuracy without the transform to about 84.0 percent with the transform, better than the uniform and K-means methods of the previous sections.<\/p>\n
Accuracy: 0.840 (0.072)<\/pre>\n
We chose the number of bins as an arbitrary number; in this case, 10.<\/p>\n
This hyperparameter can be tuned to explore the effect of the resolution of the transform on the resulting skill of the model.<\/p>\n
The example below performs this experiment and plots the mean accuracy for different “n_bins<\/em>” values from two to 10.<\/p>\n
# explore number of discrete bins on classification accuracy\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom pandas import read_csv\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.model_selection import RepeatedStratifiedKFold\r\nfrom sklearn.neighbors import KNeighborsClassifier\r\nfrom sklearn.preprocessing import KBinsDiscretizer\r\nfrom sklearn.preprocessing import LabelEncoder\r\nfrom sklearn.pipeline import Pipeline\r\nfrom matplotlib import pyplot\r\n\r\n# get the dataset\r\ndef get_dataset():\r\n\t# load dataset\r\n\turl = \"https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/sonar.csv\"\r\n\tdataset = read_csv(url, header=None)\r\n\tdata = dataset.values\r\n\t# separate into input and output columns\r\n\tX, y = data[:, :-1], data[:, -1]\r\n\t# ensure inputs are floats and output is an integer label\r\n\tX = X.astype('float32')\r\n\ty = LabelEncoder().fit_transform(y.astype('str'))\r\n\treturn X, y\r\n\r\n# get a list of models to evaluate\r\ndef get_models():\r\n\tmodels = dict()\r\n\tfor i in range(2,11):\r\n\t\t# define the pipeline\r\n\t\ttrans = KBinsDiscretizer(n_bins=i, encode='ordinal', strategy='quantile')\r\n\t\tmodel = KNeighborsClassifier()\r\n\t\tmodels[str(i)] = Pipeline(steps=[('t', trans), ('m', model)])\r\n\treturn models\r\n\r\n# evaluate a give model using cross-validation\r\ndef evaluate_model(model):\r\n\tcv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\r\n\tscores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1, error_score='raise')\r\n\treturn scores\r\n\r\n# get the dataset\r\nX, y = get_dataset()\r\n# get the models to evaluate\r\nmodels = get_models()\r\n# evaluate the models and store results\r\nresults, names = list(), list()\r\nfor name, model in models.items():\r\n\tscores = evaluate_model(model)\r\n\tresults.append(scores)\r\n\tnames.append(name)\r\n\tprint('>%s %.3f (%.3f)' % (name, mean(scores), std(scores)))\r\n# plot model performance for comparison\r\npyplot.boxplot(results, labels=names, showmeans=True)\r\npyplot.show()<\/pre>\nRunning the example reports the mean classification accuracy for each value of the “n_bins<\/em>” argument.<\/p>\n
We can see that surprisingly smaller values resulted in better accuracy, with values such as three achieving an accuracy of about 86.7 percent.<\/p>\n
>2 0.806 (0.080)\r\n>3 0.867 (0.070)\r\n>4 0.835 (0.083)\r\n>5 0.838 (0.070)\r\n>6 0.836 (0.071)\r\n>7 0.854 (0.071)\r\n>8 0.837 (0.077)\r\n>9 0.841 (0.069)\r\n>10 0.840 (0.072)<\/pre>\n
Box and whisker plots are created to summarize the classification accuracy scores for each number of discrete bins on the dataset.<\/p>\n
We can see a small bump in accuracy at three bins and the scores drop and remain flat for larger values.<\/p>\n
The results highlight that there is likely some benefit in exploring different numbers of discrete bins for the chosen method to see if better performance can be achieved.<\/p>\n
![\"Box](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/02\/Box-Plots-of-Number-of-Discrete-Bins-vs-Classification-Accuracy-of-KNN-on-the-Sonar-Dataset.png\")
<\/p>\nBox Plots of Number of Discrete Bins vs. Classification Accuracy of KNN on the Sonar Dataset<\/p>\n<\/div>\n
Further Reading<\/h2>\n
This section provides more resources on the topic if you are looking to go deeper.<\/p>\n
Tutorials<\/h3>\n
- \n
- Continuous Probability Distributions for Machine Learning<\/a><\/li>\n
- How to Transform Target Variables for Regression With Scikit-Learn<\/a><\/li>\n<\/ul>\n
Books<\/h3>\n
- \n
- Data Mining: Practical Machine Learning Tools and Techniques<\/a>, 4th edition, 2016.<\/li>\n
- Feature Engineering and Selection<\/a>, 2019.<\/li>\n<\/ul>\n
Dataset<\/h3>\n
- \n
- Sonar Dataset<\/a><\/li>\n
- Sonar Dataset Description<\/a><\/li>\n<\/ul>\n
APIs<\/h3>\n
- \n
- Non-linear transformation, scikit-learn Guide<\/a>.<\/li>\n
- sklearn.preprocessing.KBinsDiscretizer API<\/a>.<\/li>\n<\/ul>\n
Articles<\/h3>\n
- \n
- Discretization of continuous features, Wikipedia<\/a>.<\/li>\n<\/ul>\n
Summary<\/h2>\n
In this tutorial, you discovered how to use discretization transforms to map numerical values to discrete categories for machine learning.<\/p>\n
Specifically, you learned:<\/p>\n
- \n
- Many machine learning algorithms prefer or perform better when numerical with non-standard probability distributions are made discrete.<\/li>\n
- Discretization transforms are a technique for transforming numerical input or output variables to have discrete ordinal labels.<\/li>\n
- How to use the KBinsDiscretizer to change the structure and distribution of numeric variables to improve the performance of predictive models.<\/li>\n<\/ul>\n
Do you have any questions?<\/strong>
\nAsk your questions in the comments below and I will do my best to answer.<\/p>\nThe post How to Use Discretization Transforms for Machine Learning<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n
Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
- sklearn.preprocessing.KBinsDiscretizer API<\/a>.<\/li>\n<\/ul>\n
- Sonar Dataset Description<\/a><\/li>\n<\/ul>\n
- Feature Engineering and Selection<\/a>, 2019.<\/li>\n<\/ul>\n
- How to Transform Target Variables for Regression With Scikit-Learn<\/a><\/li>\n<\/ul>\n
- Sonar Dataset Description<\/a><\/li>\n<\/ul>\n
- Quantile<\/strong>: Each bin has the same number of values, split based on percentiles.<\/li>\n
- Uniform<\/strong>: Each bin has the same width in the span of possible values for the variable.<\/li>\n