Bootstrap aggregation, or bagging, is an ensemble where each model is trained on a different sample of the training dataset.<\/p>\n
The idea of bagging can be generalized to other techniques for changing the training dataset and fitting the same model on each changed version of the data. One approach is to use data transforms that change the scale and probability distribution of input variables as the basis for the training of contributing members to a bagging-like ensemble. We can refer to this as data transform bagging or a data transform ensemble.<\/p>\n
In this tutorial, you will discover how to develop a data transform ensemble.<\/p>\n
After completing this tutorial, you will know:<\/p>\n
- \n
- Data transforms can be used as the basis for a bagging-type ensemble where the same model is trained on different views of a training dataset.<\/li>\n
- How to develop a data transform ensemble for classification and confirm the ensemble performs better than any contributing member.<\/li>\n
- How to develop and evaluate a data transform ensemble for regression predictive modeling.<\/li>\n<\/ul>\n
Let’s get started.<\/p>\n
![\"Develop](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/11\/Develop-a-Bagging-Ensemble-with-Different-Data-Transformations.jpg\")
<\/p>\nDevelop a Bagging Ensemble with Different Data Transformations
Photo by Maciej Kraus<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
This tutorial is divided into three parts; they are:<\/p>\n
- \n
- Data Transform Bagging<\/li>\n
- Data Transform Ensemble for Classification<\/li>\n
- Data Transform Ensemble for Regression<\/li>\n<\/ol>\n
Data Transform Bagging<\/h2>\n
Bootstrap aggregation<\/a>, or bagging for short, is an ensemble learning technique based on the idea of fitting the same model type on multiple different samples of the same training dataset.<\/p>\n
The hope is that small differences in the training dataset used to fit each model will result in small differences in the capabilities of models. For ensemble learning, this is referred to as diversity of ensemble members and is intended to de-correlate the predictions (or prediction errors) made by each contributing member.<\/p>\n
Although it was designed to be used with decision trees and each data sample is made using the bootstrap method (selection with rel-selection), the approach has spawned a whole subfield of study with hundreds of variations on the approach.<\/p>\n
We can construct our own bagging ensembles by changing the dataset used to train each contributing member in new and unique ways.<\/p>\n
One approach would be to apply a different data preparation transform to the dataset for each contributing ensemble member.<\/p>\n
This is based on the premise that we cannot know the representational form for a training dataset that exposes the unknown underlying structure to the dataset to the learning algorithms. This motivates the need to evaluate models with a suite of different data transforms, such as changing the scale and probability distribution, in order to discover what works.<\/p>\n
This approach can be used where a suite of different transforms of the same training dataset is created, a model trained on each, and the predictions combined using simple statistics such as averaging.<\/p>\n
For lack of a better name, we will refer to this as “Data Transform Bagging<\/strong>” or a “Data Transform Ensemble<\/strong>.”<\/p>\n
There are many transforms that we can use, but perhaps a good starting point would be a selection that changes the scale and probability distribution, such as:<\/p>\n
- \n
- Normalization<\/a> (fixed range)<\/li>\n
- Standardization<\/a> (zero mean)<\/li>\n
- Robust Standardization<\/a> (robust to outliers)<\/li>\n
- Power Transform<\/a> (remove skew)<\/li>\n
- Quantile Transform<\/a> (change distribution)<\/li>\n
- Discretization<\/a> (k-bins)<\/li>\n<\/ul>\n
The approach is likely to be more effective when used with a base model that trains different or very different models based on the effects of the data transform.<\/p>\n
Changing the scale of the distribution may only be appropriate with models that are sensitive to changes in the scale of input variables, such as those that calculate a weighted sum, such as logistic regression and neural networks, and those that use distance measures, such as k-nearest neighbors and support vector machines.<\/p>\n
Changes to the probability distribution for input variables would likely impact most machine learning models.<\/p>\n
Now that we are familiar with the approach, let’s explore how we can develop a data transform ensemble for classification problems.<\/p>\n
Data Transform Ensemble for Classification<\/h2>\n
We can develop a data transform approach to bagging for classification using the scikit-learn library.<\/p>\n
The library provides a suite of standard transforms that we can use directly. Each ensemble member can be defined as a Pipeline, with the transform followed by the predictive model, in order to avoid any data leakage and, in turn, produce optimistic results. Finally, a voting ensemble can be used to combine the predictions from each pipeline.<\/p>\n
First, we can define a synthetic binary classification dataset as the basis for exploring this type of ensemble.<\/p>\n
The example below creates a dataset with 1,000 examples each comprising 20 input features, where 15 of them contain information for predicting the target.<\/p>\n
# synthetic classification dataset\r\nfrom sklearn.datasets import make_classification\r\n# define dataset\r\nX, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=1)\r\n# summarize the dataset\r\nprint(X.shape, y.shape)<\/pre>\n
Running the example will create the dataset and summarizes the shape of the data arrays, confirming our expectations.<\/p>\n
(1000, 20) (1000,)<\/pre>\n
Next, we establish a baseline on the problem using the predictive model we intend to use in our ensemble. It is standard practice to use a decision tree in bagging ensembles, so in this case, we will use the DecisionTreeClassifier<\/a> with default hyperparameters.<\/p>\n
We will evaluate the model using standard practices, in this case, repeated stratified k-fold cross-validation with three repeats and 10 folds. The performance will be reported using the mean of the classification accuracy across all folds and repeats.<\/p>\n
The complete example of evaluating a decision tree on the synthetic classification dataset is listed below.<\/p>\n
# evaluate decision tree on synthetic classification dataset\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.model_selection import RepeatedStratifiedKFold\r\nfrom sklearn.tree import DecisionTreeClassifier\r\n# define dataset\r\nX, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=1)\r\n# define the model\r\nmodel = DecisionTreeClassifier()\r\n# define the evaluation procedure\r\ncv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\r\n# evaluate the model\r\nn_scores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\r\n# report performance\r\nprint('Mean Accuracy: %.3f (%.3f)' % (mean(n_scores), std(n_scores)))<\/pre>\nRunning the example reports the mean classification accuracy of the decision tree on the synthetic classification dataset.<\/p>\n
Note<\/strong>: Your results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n
In this case, we can see that the model achieved a classification accuracy of about 82.3 percent.<\/p>\n
This score provides a baseline in performance from which we expect a data transform ensemble to improve upon.<\/p>\n
Mean Accuracy: 0.823 (0.039)<\/pre>\n
Next, we can develop an ensemble of decision trees, each fit on a different transform of the input data.<\/p>\n
First, we can define each ensemble member as a modeling pipeline. The first step will be the data transform and the second will be a decision tree classifier.<\/p>\n
For example, the pipeline for a normalization transform with the MinMaxScaler<\/a> class would look as follows:<\/p>\n
...\r\n# normalization\r\nnorm = Pipeline([('s', MinMaxScaler()), ('m', DecisionTreeClassifier())])<\/pre>\nWe can repeat this for each transform or transform configuration that we want to use and add all of the model pipelines to a list.<\/p>\n
The VotingClassifier<\/a> class can be used to combine the predictions from all of the models. This class takes an “estimators<\/em>” argument that is a list of tuples where each tuple has a name and the model or modeling pipeline. For example:<\/p>\n
...\r\n# normalization\r\nnorm = Pipeline([('s', MinMaxScaler()), ('m', DecisionTreeClassifier())])\r\nmodels.append(('norm', norm))\r\n...\r\n# define the voting ensemble\r\nensemble = VotingClassifier(estimators=models, voting='hard')<\/pre>\nTo make the code easier to read, we can define a function get_ensemble()<\/em> to create the members and data transform ensemble itself.<\/p>\n
# get a voting ensemble of models\r\ndef get_ensemble():\r\n\t# define the base models\r\n\tmodels = list()\r\n\t# normalization\r\n\tnorm = Pipeline([('s', MinMaxScaler()), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('norm', norm))\r\n\t# standardization\r\n\tstd = Pipeline([('s', StandardScaler()), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('std', std))\r\n\t# robust\r\n\trobust = Pipeline([('s', RobustScaler()), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('robust', robust))\r\n\t# power\r\n\tpower = Pipeline([('s', PowerTransformer()), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('power', power))\r\n\t# quantile\r\n\tquant = Pipeline([('s', QuantileTransformer(n_quantiles=100, output_distribution='normal')), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('quant', quant))\r\n\t# kbins\r\n\tkbins = Pipeline([('s', KBinsDiscretizer(n_bins=20, encode='ordinal')), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('kbins', kbins))\r\n\t# define the voting ensemble\r\n\tensemble = VotingClassifier(estimators=models, voting='hard')\r\n\treturn ensemble<\/pre>\nWe can then call this function and evaluate the voting ensemble as per normal, just like we did for the decision tree above.<\/p>\n
Tying this together, the complete example is listed below.<\/p>\n
# evaluate data transform bagging ensemble on a classification dataset\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.model_selection import RepeatedStratifiedKFold\r\nfrom sklearn.preprocessing import MinMaxScaler\r\nfrom sklearn.preprocessing import StandardScaler\r\nfrom sklearn.preprocessing import RobustScaler\r\nfrom sklearn.preprocessing import PowerTransformer\r\nfrom sklearn.preprocessing import QuantileTransformer\r\nfrom sklearn.preprocessing import KBinsDiscretizer\r\nfrom sklearn.tree import DecisionTreeClassifier\r\nfrom sklearn.ensemble import VotingClassifier\r\nfrom sklearn.pipeline import Pipeline\r\n\r\n# get a voting ensemble of models\r\ndef get_ensemble():\r\n\t# define the base models\r\n\tmodels = list()\r\n\t# normalization\r\n\tnorm = Pipeline([('s', MinMaxScaler()), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('norm', norm))\r\n\t# standardization\r\n\tstd = Pipeline([('s', StandardScaler()), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('std', std))\r\n\t# robust\r\n\trobust = Pipeline([('s', RobustScaler()), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('robust', robust))\r\n\t# power\r\n\tpower = Pipeline([('s', PowerTransformer()), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('power', power))\r\n\t# quantile\r\n\tquant = Pipeline([('s', QuantileTransformer(n_quantiles=100, output_distribution='normal')), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('quant', quant))\r\n\t# kbins\r\n\tkbins = Pipeline([('s', KBinsDiscretizer(n_bins=20, encode='ordinal')), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('kbins', kbins))\r\n\t# define the voting ensemble\r\n\tensemble = VotingClassifier(estimators=models, voting='hard')\r\n\treturn ensemble\r\n\r\n# define dataset\r\nX, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=1)\r\n# get models\r\nensemble = get_ensemble()\r\n# define the evaluation procedure\r\ncv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\r\n# evaluate the model\r\nn_scores = cross_val_score(ensemble, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\r\n# report performance\r\nprint('Mean Accuracy: %.3f (%.3f)' % (mean(n_scores), std(n_scores)))<\/pre>\nRunning the example reports the mean classification accuracy of the data transform ensemble on the synthetic classification dataset.<\/p>\n
Note<\/strong>: Your results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n
In this case, we can see that the data transform ensemble achieved a classification accuracy of about 83.8 percent, which is a lift over using a decision tree alone that achieved an accuracy of about 82.3 percent.<\/p>\n
Mean Accuracy: 0.838 (0.042)<\/pre>\n
Although the ensemble performed well compared to a single decision tree, a limitation of this test is that we do not know if the ensemble performed better than any contributing member.<\/p>\n
This is important, as if a contributing member to the ensemble performs better, then it would be simpler and easier to use the member itself as the model instead of the ensemble.<\/p>\n
We can check this by evaluating the performance of each individual model and comparing the results to the ensemble.<\/p>\n
First, we can update the get_ensemble()<\/em> function to return a list of models to evaluate composed of the individual ensemble members as well as the ensemble itself.<\/p>\n
# get a voting ensemble of models\r\ndef get_ensemble():\r\n\t# define the base models\r\n\tmodels = list()\r\n\t# normalization\r\n\tnorm = Pipeline([('s', MinMaxScaler()), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('norm', norm))\r\n\t# standardization\r\n\tstd = Pipeline([('s', StandardScaler()), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('std', std))\r\n\t# robust\r\n\trobust = Pipeline([('s', RobustScaler()), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('robust', robust))\r\n\t# power\r\n\tpower = Pipeline([('s', PowerTransformer()), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('power', power))\r\n\t# quantile\r\n\tquant = Pipeline([('s', QuantileTransformer(n_quantiles=100, output_distribution='normal')), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('quant', quant))\r\n\t# kbins\r\n\tkbins = Pipeline([('s', KBinsDiscretizer(n_bins=20, encode='ordinal')), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('kbins', kbins))\r\n\t# define the voting ensemble\r\n\tensemble = VotingClassifier(estimators=models, voting='hard')\r\n\t# return a list of tuples each with a name and model\r\n\treturn models + [('ensemble', ensemble)]<\/pre>\nWe can call this function and enumerate each model, evaluating it, reporting the performance, and storing the results.<\/p>\n
...\r\n# get models\r\nmodels = get_ensemble()\r\n# evaluate each model\r\nresults = list()\r\nfor name,model in models:\r\n\t# define the evaluation method\r\n\tcv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\r\n\t# evaluate the model on the dataset\r\n\tn_scores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\r\n\t# report performance\r\n\tprint('>%s: %.3f (%.3f)' % (name, mean(n_scores), std(n_scores)))\r\n\tresults.append(n_scores)<\/pre>\nFinally, we can plot the distribution of accuracy scores as box and whisker plots side by side and compare the distribution of scores directly.<\/p>\n
Visually, we would hope that the spread of scores for the ensemble skews higher than any individual member and that the central tendency of the distribution (mean and median) are also higher than any member.<\/p>\n
...\r\n# plot the results for comparison\r\npyplot.boxplot(results, labels=[n for n,_ in models], showmeans=True)\r\npyplot.show()<\/pre>\n
Tying this together, the complete example of comparing the performance of contributing members to the performance of the data transform ensemble is listed below.<\/p>\n
# comparison of data transform ensemble to each contributing member for classification\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.model_selection import RepeatedStratifiedKFold\r\nfrom sklearn.preprocessing import MinMaxScaler\r\nfrom sklearn.preprocessing import StandardScaler\r\nfrom sklearn.preprocessing import RobustScaler\r\nfrom sklearn.preprocessing import PowerTransformer\r\nfrom sklearn.preprocessing import QuantileTransformer\r\nfrom sklearn.preprocessing import KBinsDiscretizer\r\nfrom sklearn.tree import DecisionTreeClassifier\r\nfrom sklearn.ensemble import VotingClassifier\r\nfrom sklearn.pipeline import Pipeline\r\nfrom matplotlib import pyplot\r\n\r\n# get a voting ensemble of models\r\ndef get_ensemble():\r\n\t# define the base models\r\n\tmodels = list()\r\n\t# normalization\r\n\tnorm = Pipeline([('s', MinMaxScaler()), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('norm', norm))\r\n\t# standardization\r\n\tstd = Pipeline([('s', StandardScaler()), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('std', std))\r\n\t# robust\r\n\trobust = Pipeline([('s', RobustScaler()), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('robust', robust))\r\n\t# power\r\n\tpower = Pipeline([('s', PowerTransformer()), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('power', power))\r\n\t# quantile\r\n\tquant = Pipeline([('s', QuantileTransformer(n_quantiles=100, output_distribution='normal')), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('quant', quant))\r\n\t# kbins\r\n\tkbins = Pipeline([('s', KBinsDiscretizer(n_bins=20, encode='ordinal')), ('m', DecisionTreeClassifier())])\r\n\tmodels.append(('kbins', kbins))\r\n\t# define the voting ensemble\r\n\tensemble = VotingClassifier(estimators=models, voting='hard')\r\n\t# return a list of tuples each with a name and model\r\n\treturn models + [('ensemble', ensemble)]\r\n\r\n# define dataset\r\nX, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=1)\r\n# get models\r\nmodels = get_ensemble()\r\n# evaluate each model\r\nresults = list()\r\nfor name,model in models:\r\n\t# define the evaluation method\r\n\tcv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\r\n\t# evaluate the model on the dataset\r\n\tn_scores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\r\n\t# report performance\r\n\tprint('>%s: %.3f (%.3f)' % (name, mean(n_scores), std(n_scores)))\r\n\tresults.append(n_scores)\r\n# plot the results for comparison\r\npyplot.boxplot(results, labels=[n for n,_ in models], showmeans=True)\r\npyplot.show()<\/pre>\nRunning the example first reports the mean and standard classification accuracy of each individual model, ending with the performance of the ensemble that combines the models.<\/p>\n
Note<\/strong>: Your results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n
In this case, we can see that a number of the individual members perform well, such as “kbins<\/em>” that achieves an accuracy of about 83.3 percent, and “std<\/em>” that achieves an accuracy of about 83.1 percent. We can also see that the ensemble achieves better overall performance compared to any contributing member, with an accuracy of about 83.4 percent.<\/p>\n
>norm: 0.821 (0.041)\r\n>std: 0.831 (0.045)\r\n>robust: 0.826 (0.044)\r\n>power: 0.825 (0.045)\r\n>quant: 0.817 (0.042)\r\n>kbins: 0.833 (0.035)\r\n>ensemble: 0.834 (0.040)<\/pre>\n
A figure is also created showing box and whisker plots of classification accuracy for each individual model as well as the data transform ensemble.<\/p>\n
We can see that the distribution for the ensemble is skewed up, which is what we might hope, and that the mean (green triangle) is slightly higher than those of the individual ensemble members.<\/p>\n
![\"Box](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/07\/Box-and-Whisker-Plot-of-Accuracy-Distribution-for-Individual-Models-and-Data-Transform-Ensemble.png\")
<\/p>\nBox and Whisker Plot of Accuracy Distribution for Individual Models and Data Transform Ensemble<\/p>\n<\/div>\n
Now that we are familiar with how to develop a data transform ensemble for classification, let’s look at doing the same for regression.<\/p>\n
Data Transform Ensemble for Regression<\/h2>\n
In this section, we will explore developing a data transform ensemble for a regression predictive modeling problem.<\/p>\n
First, we can define a synthetic binary regression dataset as the basis for exploring this type of ensemble.<\/p>\n
The example below creates a dataset with 1,000 examples each of 100 input features where 10 of them contain information for predicting the target.<\/p>\n
# synthetic regression dataset\r\nfrom sklearn.datasets import make_regression\r\n# define dataset\r\nX, y = make_regression(n_samples=1000, n_features=100, n_informative=10, noise=0.1, random_state=1)\r\n# summarize the dataset\r\nprint(X.shape, y.shape)<\/pre>\n
Running the example creates the dataset and confirms the data has the expected shape.<\/p>\n
(1000, 100) (1000,)<\/pre>\n
Next, we can establish a baseline in performance on the synthetic dataset by fitting and evaluating the base model that we intend to use in the ensemble, in this case, a DecisionTreeRegressor<\/a>.<\/p>\n
The model will be evaluated using repeated k-fold cross-validation with three repeats and 10 folds. Model performance on the dataset will be reported using the mean absolute error, or MAE. The scikit-learn will invert the score (make it negative) so that the framework can maximize the score. As such, we can ignore the sign on the score.<\/p>\n
The example below evaluates the decision tree on the synthetic regression dataset.<\/p>\n
# evaluate decision tree on synthetic regression dataset\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom sklearn.datasets import make_regression\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.model_selection import RepeatedKFold\r\nfrom sklearn.tree import DecisionTreeRegressor\r\n# define dataset\r\nX, y = make_regression(n_samples=1000, n_features=100, n_informative=10, noise=0.1, random_state=1)\r\n# define the model\r\nmodel = DecisionTreeRegressor()\r\n# define the evaluation procedure\r\ncv = RepeatedKFold(n_splits=10, n_repeats=3, random_state=1)\r\n# evaluate the model\r\nn_scores = cross_val_score(model, X, y, scoring='neg_mean_absolute_error', cv=cv, n_jobs=-1)\r\n# report performance\r\nprint('MAE: %.3f (%.3f)' % (mean(n_scores), std(n_scores)))<\/pre>\nRunning the example reports the MAE of the decision tree on the synthetic regression dataset.<\/p>\n
Note<\/strong>: Your results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n
In this case, we can see that the model achieved a MAE of about 139.817. This provides a floor in performance that we expect the ensemble model to improve upon.<\/p>\n
MAE: -139.817 (12.449)<\/pre>\n
Next, we can develop and evaluate the ensemble.<\/p>\n
We will use the same data transforms from the previous section. The VotingRegressor<\/a> will be used to combine the predictions, which is appropriate for regression problems.<\/p>\n
The get_ensemble() function defined below creates the individual models and the ensemble model and combines all of the models as a list of tuples for evaluation.<\/p>\n
# get a voting ensemble of models\r\ndef get_ensemble():\r\n\t# define the base models\r\n\tmodels = list()\r\n\t# normalization\r\n\tnorm = Pipeline([('s', MinMaxScaler()), ('m', DecisionTreeRegressor())])\r\n\tmodels.append(('norm', norm))\r\n\t# standardization\r\n\tstd = Pipeline([('s', StandardScaler()), ('m', DecisionTreeRegressor())])\r\n\tmodels.append(('std', std))\r\n\t# robust\r\n\trobust = Pipeline([('s', RobustScaler()), ('m', DecisionTreeRegressor())])\r\n\tmodels.append(('robust', robust))\r\n\t# power\r\n\tpower = Pipeline([('s', PowerTransformer()), ('m', DecisionTreeRegressor())])\r\n\tmodels.append(('power', power))\r\n\t# quantile\r\n\tquant = Pipeline([('s', QuantileTransformer(n_quantiles=100, output_distribution='normal')), ('m', DecisionTreeRegressor())])\r\n\tmodels.append(('quant', quant))\r\n\t# kbins\r\n\tkbins = Pipeline([('s', KBinsDiscretizer(n_bins=20, encode='ordinal')), ('m', DecisionTreeRegressor())])\r\n\tmodels.append(('kbins', kbins))\r\n\t# define the voting ensemble\r\n\tensemble = VotingRegressor(estimators=models)\r\n\t# return a list of tuples each with a name and model\r\n\treturn models + [('ensemble', ensemble)]<\/pre>\nWe can then call this function and evaluate each contributing modeling pipeline independently and compare the results to the ensemble of the pipelines.<\/p>\n
Our expectation, as before, is that the ensemble results in a lift in performance over any individual model. If it does not, then the top-performing individual model should be chosen instead.<\/p>\n
Tying this together, the complete example for evaluating a data transform ensemble for a regression dataset is listed below.<\/p>\n
# comparison of data transform ensemble to each contributing member for regression\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom sklearn.datasets import make_regression\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.model_selection import RepeatedKFold\r\nfrom sklearn.preprocessing import MinMaxScaler\r\nfrom sklearn.preprocessing import StandardScaler\r\nfrom sklearn.preprocessing import RobustScaler\r\nfrom sklearn.preprocessing import PowerTransformer\r\nfrom sklearn.preprocessing import QuantileTransformer\r\nfrom sklearn.preprocessing import KBinsDiscretizer\r\nfrom sklearn.tree import DecisionTreeRegressor\r\nfrom sklearn.ensemble import VotingRegressor\r\nfrom sklearn.pipeline import Pipeline\r\nfrom matplotlib import pyplot\r\n\r\n# get a voting ensemble of models\r\ndef get_ensemble():\r\n\t# define the base models\r\n\tmodels = list()\r\n\t# normalization\r\n\tnorm = Pipeline([('s', MinMaxScaler()), ('m', DecisionTreeRegressor())])\r\n\tmodels.append(('norm', norm))\r\n\t# standardization\r\n\tstd = Pipeline([('s', StandardScaler()), ('m', DecisionTreeRegressor())])\r\n\tmodels.append(('std', std))\r\n\t# robust\r\n\trobust = Pipeline([('s', RobustScaler()), ('m', DecisionTreeRegressor())])\r\n\tmodels.append(('robust', robust))\r\n\t# power\r\n\tpower = Pipeline([('s', PowerTransformer()), ('m', DecisionTreeRegressor())])\r\n\tmodels.append(('power', power))\r\n\t# quantile\r\n\tquant = Pipeline([('s', QuantileTransformer(n_quantiles=100, output_distribution='normal')), ('m', DecisionTreeRegressor())])\r\n\tmodels.append(('quant', quant))\r\n\t# kbins\r\n\tkbins = Pipeline([('s', KBinsDiscretizer(n_bins=20, encode='ordinal')), ('m', DecisionTreeRegressor())])\r\n\tmodels.append(('kbins', kbins))\r\n\t# define the voting ensemble\r\n\tensemble = VotingRegressor(estimators=models)\r\n\t# return a list of tuples each with a name and model\r\n\treturn models + [('ensemble', ensemble)]\r\n\r\n# generate regression dataset\r\nX, y = make_regression(n_samples=1000, n_features=100, n_informative=10, noise=0.1, random_state=1)\r\n# get models\r\nmodels = get_ensemble()\r\n# evaluate each model\r\nresults = list()\r\nfor name,model in models:\r\n\t# define the evaluation method\r\n\tcv = RepeatedKFold(n_splits=10, n_repeats=3, random_state=1)\r\n\t# evaluate the model on the dataset\r\n\tn_scores = cross_val_score(model, X, y, scoring='neg_mean_absolute_error', cv=cv, n_jobs=-1)\r\n\t# report performance\r\n\tprint('>%s: %.3f (%.3f)' % (name, mean(n_scores), std(n_scores)))\r\n\tresults.append(n_scores)\r\n# plot the results for comparison\r\npyplot.boxplot(results, labels=[n for n,_ in models], showmeans=True)\r\npyplot.show()<\/pre>\nRunning the example first reports the MAE of each individual model, ending with the performance of the ensemble that combines the models.<\/p>\n
Note<\/strong>: Your results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n
We can see that each model performs about the same, with MAE error scores around 140, all higher than the decision tree used in isolation. Interestingly, the ensemble performs the best, out-performing all of the individual members and the tree with no transforms, achieving a MAE of about 126.487.<\/p>\n
This result suggests that although each pipeline performs worse than a single tree without transforms, each pipeline is making different errors and that the average of the models is able to leverage and harness these differences toward lower error.<\/p>\n
>norm: -140.559 (11.783)\r\n>std: -140.582 (11.996)\r\n>robust: -140.813 (11.827)\r\n>power: -141.089 (12.668)\r\n>quant: -141.109 (11.097)\r\n>kbins: -145.134 (11.638)\r\n>ensemble: -126.487 (9.999)<\/pre>\n
A figure is created comparing the distribution of MAE scores for each pipeline and the ensemble.<\/p>\n
As we hoped, the distribution for the ensemble skews higher compared to all of the other models and has a higher (smaller) central tendency (mean and median indicated by the green triangle and orange line respectively).<\/p>\n
![\"Box](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/07\/Box-and-Whisker-Plot-of-MAE-Distributions-for-Individual-Models-and-Data-Transform-Ensemble.png\")
<\/p>\nBox and Whisker Plot of MAE Distributions for Individual Models and Data Transform Ensemble<\/p>\n<\/div>\n
Further Reading<\/h2>\n
This section provides more resources on the topic if you are looking to go deeper.<\/p>\n
Tutorials<\/h3>\n
- \n
- How to Develop Voting Ensembles With Python<\/a><\/li>\n<\/ul>\n
Books<\/h3>\n
- \n
- Pattern Classification Using Ensemble Methods<\/a>, 2010.<\/li>\n
- Ensemble Methods<\/a>, 2012.<\/li>\n
- Ensemble Machine Learning<\/a>, 2012.<\/li>\n<\/ul>\n
APIs<\/h3>\n
- \n
- sklearn.ensemble.VotingClassifier API<\/a>.<\/li>\n
- sklearn.ensemble.VotingRegressor API<\/a>.<\/li>\n<\/ul>\n
Summary<\/h2>\n
In this tutorial, you discovered how to develop a data transform ensemble.<\/p>\n
Specifically, you learned:<\/p>\n
- \n
- Data transforms can be used as the basis for a bagging-type ensemble where the same model is trained on different views of a training dataset.<\/li>\n
- How to develop a data transform ensemble for classification and confirm the ensemble performs better than any contributing member.<\/li>\n
- How to develop and evaluate a data transform ensemble for regression predictive modeling.<\/li>\n<\/ul>\n
Do you have any questions?<\/strong>
\nAsk your questions in the comments below and I will do my best to answer.<\/p>\nThe post Develop a Bagging Ensemble with Different Data Transformations<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n
Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
- sklearn.ensemble.VotingRegressor API<\/a>.<\/li>\n<\/ul>\n
- Ensemble Methods<\/a>, 2012.<\/li>\n
- Pattern Classification Using Ensemble Methods<\/a>, 2010.<\/li>\n
- Standardization<\/a> (zero mean)<\/li>\n
- Normalization<\/a> (fixed range)<\/li>\n