Random Subspace Ensemble<\/strong> is a machine learning algorithm that combines the predictions from multiple decision trees trained on different subsets of columns in the training dataset.<\/p>\n Randomly varying the columns used to train each contributing member of the ensemble has the effect of introducing diversity into the ensemble and, in turn, can lift performance over using a single decision tree.<\/p>\n It is related to other ensembles of decision trees such as bootstrap aggregation (bagging) that creates trees using different samples of rows from the training dataset, and random forest that combines ideas from bagging and the random subspace ensemble.<\/p>\n Although decision trees are often used, the general random subspace method can be used with any machine learning model whose performance varies meaningfully with the choice of input features.<\/p>\n In this tutorial, you will discover how to develop random subspace ensembles for classification and regression.<\/p>\n After completing this tutorial, you will know:<\/p>\n Let’s get started.<\/p>\n How to Develop a Random Subspace Ensemble With Python This tutorial is divided into three parts; they are:<\/p>\n A predictive modeling problem consists of one or more input variables and a target variable.<\/p>\n A variable is a column in the data and is also often referred to as a feature. We can consider all input features together as defining an n-dimensional vector space, where n is the number of input features and each example (input row of data) is a point in the feature space.<\/p>\n This is a common conceptualization in machine learning and as input feature spaces become larger, the distance between points in the space increases, known generally as the curse of dimensionality<\/a>.<\/p>\n A subset of input features can, therefore, be thought of as a subset of the input feature space, or a subspace.<\/p>\n Selecting features is a way of defining a subspace of the input feature space. For example, feature selection<\/a> refers to an attempt to reduce the number of dimensions of the input feature space by selecting a subset of features to keep or a subset of features to delete, often based on their relationship to the target variable.<\/p>\n Alternatively, we can select random subsets of input features to define random subspaces. This can be used as the basis for an ensemble learning algorithm, where a model can be fit on each random subspace of features. This is referred to as a random subspace ensemble or the random subspace method<\/strong>.<\/p>\n The training data is usually described by a set of features. Different subsets of features, or called subspaces, provide different views on the data. Therefore, individual learners trained from different subspaces are usually diverse.<\/p>\n<\/blockquote>\n — Page 116, Ensemble Methods<\/a>, 2012.<\/p>\n It was proposed by Tin Kam Ho in the 1998 paper titled “The Random Subspace Method For Constructing Decision Forests<\/a>” where a decision tree is fit on each random subspace.<\/p>\n More generally, it is a diversity technique for ensemble learning that belongs to a class of methods that change the training dataset for each model in the attempt to reduce the correlation between the predictions of the models in the ensemble.<\/p>\n The procedure is as simple as selecting a random subset of input features (columns) for each model in the ensemble and fitting the model on the model in the entire training dataset. It can be augmented with additional changes, such as using a bootstrap or random sample of the rows in training dataset.<\/p>\n The classifier consists of multiple trees constructed systematically by pseudorandomly selecting subsets of components of the feature vector, that is, trees constructed in randomly chosen subspaces.<\/p>\n<\/blockquote>\n — The Random Subspace Method For Constructing Decision Forests<\/a>, 1998.<\/p>\n As such, the random subspace ensemble is related to bootstrap aggregation (bagging) that introduces diversity by training each model, often a decision tree, on a different random sample of the training dataset, with replacement (e.g. the bootstrap sampling method). The random forest ensemble may also be considered a hybrid of both the bagging and random subset ensemble methods.<\/p>\n Algorithms that use different feature subsets are commonly referred to as random subspace methods …<\/p>\n<\/blockquote>\n — Page 21, Ensemble Machine Learning<\/a>, 2012.<\/p>\n The random subspace method can be used with any machine learning algorithm, although it is well suited to models that are sensitive to large changes to the input features, such as decision trees and k-nearest neighbors.<\/p>\n It is appropriate for datasets that have a large number of input features, as it can result in good performance with good efficiency. If the dataset contains many irrelevant input features, it may be better to use feature selection as a data preparation technique as the prevalence of irrelevant features in subspaces may hurt the performance of the ensemble.<\/p>\n For data with a lot of redundant features, training a learner in a subspace will be not only effective but also efficient.<\/p>\n<\/blockquote>\n — Page 116, Ensemble Methods<\/a>, 2012.<\/p>\n Now that we are familiar with the random subspace ensemble, let’s explore how we can implement the approach.<\/p>\n We can implement the random subspace ensemble using bagging in scikit-learn.<\/p>\n Bagging is provided via the BaggingRegressor<\/a> and BaggingClassifier<\/a> classes.<\/p>\n We can configure bagging to be a random subspace ensemble by setting the “bootstrap<\/em>” argument to “False<\/em>” to turn off sampling of the training dataset rows and setting the maximum number of features to a given value via the “max_features<\/em>” argument.<\/p>\n The default model for bagging is a decision tree, but it can be changed to any model we like.<\/p>\n We can demonstrate using bagging to implement a random subspace ensemble with decision trees for classification and regression.<\/p>\n In this section, we will look at developing a random subspace ensemble using bagging for a classification problem.<\/p>\n First, we can use the make_classification() function<\/a> to create a synthetic binary classification problem with 1,000 examples and 20 input features.<\/p>\n The complete example is listed below.<\/p>\n Running the example creates the dataset and summarizes the shape of the input and output components.<\/p>\n Next, we can configure a bagging model to be a random subspace ensemble for decision trees on this dataset.<\/p>\n Each model will be fit on a random subspace of 10 input features, chosen arbitrarily.<\/p>\n We will evaluate the model using repeated stratified k-fold cross-validation, with three repeats and 10 folds. We will report the mean and standard deviation of the accuracy of the model across all repeats and folds.<\/p>\n Running the example reports the mean and standard deviation accuracy of the model.<\/p>\n Note<\/strong>: Your results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n In this case, we can see the random subspace ensemble with default hyperparameters achieves a classification accuracy of about 85.4 percent on this test dataset.<\/p>\n We can also use the random subspace ensemble model as a final model and make predictions for classification.<\/p>\n First, the ensemble is fit on all available data, then the predict()<\/em> function can be called to make predictions on new data.<\/p>\n The example below demonstrates this on our binary classification dataset.<\/p>\n Running the example fits the random subspace ensemble model on the entire dataset and is then used to make a prediction on a new row of data, as we might when using the model in an application.<\/p>\n Now that we are familiar with using bagging for classification, let’s look at the API for regression.<\/p>\n In this section, we will look at using bagging for a regression problem.<\/p>\n First, we can use the make_regression() function<\/a> to create a synthetic regression problem with 1,000 examples and 20 input features.<\/p>\n The complete example is listed below.<\/p>\n Running the example creates the dataset and summarizes the shape of the input and output components.<\/p>\n Next, we can evaluate a random subspace ensemble via bagging on this dataset.<\/p>\n As before, we must configure bagging to use all rows of the training dataset and specify the number of input features to randomly select.<\/p>\n As we did with the last section, we will evaluate the model using repeated k-fold cross-validation, with three repeats and 10 folds. We will report the mean absolute error (MAE) of the model across all repeats and folds. The scikit-learn library makes the MAE negative so that it is maximized instead of minimized. This means that larger negative MAE are better and a perfect model has a MAE of 0.<\/p>\n The complete example is listed below.<\/p>\n Running the example reports the mean and standard deviation accuracy of the model.<\/p>\n Note<\/strong>: Your results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n In this case, we can see that the bagging ensemble with default hyperparameters achieves a MAE of about 114.<\/p>\n We can also use the random subspace ensemble model as a final model and make predictions for regression.<\/p>\n First, the ensemble is fit on all available data, then the predict() function can be called to make predictions on new data.<\/p>\n The example below demonstrates this on our regression dataset.<\/p>\n Running the example fits the random subspace ensemble model on the entire dataset and is then used to make a prediction on a new row of data, as we might when using the model in an application.<\/p>\n Now that we are familiar with using the scikit-learn API to evaluate and use random subspace ensembles, let’s look at configuring the model.<\/p>\n In this section, we will take a closer look at some of the hyperparameters you should consider tuning for the random subspace ensemble and their effect on model performance.<\/p>\n An important hyperparameter for the random subspace method is the number of decision trees used in the ensemble. More trees will stabilize the variance of the model, countering the effect of the number of features selected by each tree that introduces diversity.<\/p>\n The number of trees can be set via the “n_estimators<\/em>” argument and defaults to 10.<\/p>\n The example below explores the effect of the number of trees with values between 10 to 5,000.<\/p>\n Running the example first reports the mean accuracy for each configured number of decision trees.<\/p>\n Note<\/strong>: Your results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n In this case, we can see that that performance appears to continue to improve as the number of ensemble members is increased to 5,000.<\/p>\n A box and whisker plot is created for the distribution of accuracy scores for each configured number of trees.<\/p>\n We can see the general trend of further improvement with the number of decision trees used in the ensemble.<\/p>\n Box Plot of Random Subspace Ensemble Size vs. Classification Accuracy<\/p>\n<\/div>\n The number of features selected for each random subspace controls the diversity of the ensemble.<\/p>\n Fewer features mean more diversity, whereas more features mean less diversity. More diversity may require more trees to reduce the variance of predictions made by the model.<\/p>\n We can vary the diversity of the ensemble by varying the number of random features selected by setting the “max_features<\/em>” argument.<\/p>\n The example below varies the value from 1 to 20 with a fixed number of trees in the ensemble.<\/p>\n Running the example first reports the mean accuracy for each number of features.<\/p>\n Note<\/strong>: Your results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n In this case, we can see that perhaps using 8 to 11 features in the random subspaces might be appropriate on this dataset when using 100 decision trees. This might suggest increasing the number of trees to a large value first, then tuning the number of features selected in each subset.<\/p>\n A box and whisker plot is created for the distribution of accuracy scores for each number of random subset features.<\/p>\n We can see a general trend of increasing accuracy to a point and a steady decrease in performance after 11 features.<\/p>\n Box Plot of Random Subspace Ensemble Features vs. Classification Accuracy<\/p>\n<\/div>\n Decision trees are the most common algorithm used in a random subspace ensemble.<\/p>\n The reason for this is that they are easy to configure and work well on most problems.<\/p>\n Other algorithms can be used to construct random subspaces and must be configured to have a modestly high variance. One example is the k-nearest neighbors algorithm where the k<\/em> value can be set to a low value.<\/p>\n The algorithm used in the ensemble is specified via the “base_estimator<\/em>” argument and must be set to an instance of the algorithm and algorithm configuration to use.<\/p>\n The example below demonstrates using a KNeighborsClassifier<\/a> as the base algorithm used in the random subspace ensemble via the bagging class. Here, the algorithm is used with default hyperparameters where k is set to 5.<\/p>\n The complete example is listed below.<\/p>\n Running the example reports the mean and standard deviation accuracy of the model.<\/p>\n Note<\/strong>: Your results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n In this case, we can see the random subspace ensemble with KNN and default hyperparameters achieves a classification accuracy of about 90 percent on this test dataset.<\/p>\n This section provides more resources on the topic if you are looking to go deeper.<\/p>\n In this tutorial, you discovered how to develop random subspace ensembles for classification and regression.<\/p>\n Specifically, you learned:<\/p>\n Do you have any questions?<\/strong> The post How to Develop a Random Subspace Ensemble With Python<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"\n
![\"How](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/11\/How-to-Develop-a-Random-Subspace-Ensemble-With-Python.jpg\")
<\/p>\n
Photo by Marsel Minga<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
\n
\n
\n
Random Subspace Ensemble<\/h2>\n
\n
\n
\n
\n
Random Subspace Ensemble via Bagging<\/h2>\n
Random Subspace Ensemble for Classification<\/h3>\n
# test classification dataset\r\nfrom sklearn.datasets import make_classification\r\n# define dataset\r\nX, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=5)\r\n# summarize the dataset\r\nprint(X.shape, y.shape)<\/pre>\n
(1000, 20) (1000,)<\/pre>\n
...\r\n# define the random subspace ensemble model\r\nmodel = BaggingClassifier(bootstrap=False, max_features=10)<\/pre>\n
# evaluate random subspace ensemble via bagging for classification\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.model_selection import RepeatedStratifiedKFold\r\nfrom sklearn.ensemble import BaggingClassifier\r\n# define dataset\r\nX, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=5)\r\n# define the random subspace ensemble model\r\nmodel = BaggingClassifier(bootstrap=False, max_features=10)\r\n# define the evaluation method\r\ncv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\r\n# evaluate the model on the dataset\r\nn_scores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\r\n# report performance\r\nprint('Mean Accuracy: %.3f (%.3f)' % (mean(n_scores), std(n_scores)))<\/pre>\nMean Accuracy: 0.854 (0.039)<\/pre>\n
# make predictions using random subspace ensemble via bagging for classification\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.ensemble import BaggingClassifier\r\n# define dataset\r\nX, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=5)\r\n# define the model\r\nmodel = BaggingClassifier(bootstrap=False, max_features=10)\r\n# fit the model on the whole dataset\r\nmodel.fit(X, y)\r\n# make a single prediction\r\nrow = [[-4.7705504,-1.88685058,-0.96057964,2.53850317,-6.5843005,3.45711663,-7.46225013,2.01338213,-0.45086384,-1.89314931,-2.90675203,-0.21214568,-0.9623956,3.93862591,0.06276375,0.33964269,4.0835676,1.31423977,-2.17983117,3.1047287]]\r\nyhat = model.predict(row)\r\nprint('Predicted Class: %d' % yhat[0])<\/pre>\nPredicted Class: 1<\/pre>\n
Random Subspace Ensemble for Regression<\/h3>\n
# test regression dataset\r\nfrom sklearn.datasets import make_regression\r\n# define dataset\r\nX, y = make_regression(n_samples=1000, n_features=20, n_informative=15, noise=0.1, random_state=5)\r\n# summarize the dataset\r\nprint(X.shape, y.shape)<\/pre>\n
(1000, 20) (1000,)<\/pre>\n
...\r\n# define the model\r\nmodel = BaggingRegressor(bootstrap=False, max_features=10)<\/pre>\n
# evaluate random subspace ensemble via bagging for regression\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom sklearn.datasets import make_regression\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.model_selection import RepeatedKFold\r\nfrom sklearn.ensemble import BaggingRegressor\r\n# define dataset\r\nX, y = make_regression(n_samples=1000, n_features=20, n_informative=15, noise=0.1, random_state=5)\r\n# define the model\r\nmodel = BaggingRegressor(bootstrap=False, max_features=10)\r\n# define the evaluation procedure\r\ncv = RepeatedKFold(n_splits=10, n_repeats=3, random_state=1)\r\n# evaluate the model\r\nn_scores = cross_val_score(model, X, y, scoring='neg_mean_absolute_error', cv=cv, n_jobs=-1, error_score='raise')\r\n# report performance\r\nprint('MAE: %.3f (%.3f)' % (mean(n_scores), std(n_scores)))<\/pre>\nMAE: -114.630 (10.920)<\/pre>\n
# random subspace ensemble via bagging for making predictions for regression\r\nfrom sklearn.datasets import make_regression\r\nfrom sklearn.ensemble import BaggingRegressor\r\n# define dataset\r\nX, y = make_regression(n_samples=1000, n_features=20, n_informative=15, noise=0.1, random_state=5)\r\n# define the model\r\nmodel = BaggingRegressor(bootstrap=False, max_features=10)\r\n# fit the model on the whole dataset\r\nmodel.fit(X, y)\r\n# make a single prediction\r\nrow = [[0.88950817,-0.93540416,0.08392824,0.26438806,-0.52828711,-1.21102238,-0.4499934,1.47392391,-0.19737726,-0.22252503,0.02307668,0.26953276,0.03572757,-0.51606983,-0.39937452,1.8121736,-0.00775917,-0.02514283,-0.76089365,1.58692212]]\r\nyhat = model.predict(row)\r\nprint('Prediction: %d' % yhat[0])<\/pre>\nPrediction: -157<\/pre>\n
Random Subspace Ensemble Hyperparameters<\/h2>\n
Explore Number of Trees<\/h3>\n
# explore random subspace ensemble number of trees effect on performance\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.model_selection import RepeatedStratifiedKFold\r\nfrom sklearn.ensemble import BaggingClassifier\r\nfrom matplotlib import pyplot\r\n\r\n# get the dataset\r\ndef get_dataset():\r\n\tX, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=5)\r\n\treturn X, y\r\n\r\n# get a list of models to evaluate\r\ndef get_models():\r\n\tmodels = dict()\r\n\tn_trees = [10, 50, 100, 500, 1000, 5000]\r\n\tfor n in n_trees:\r\n\t\tmodels[str(n)] = BaggingClassifier(n_estimators=n, bootstrap=False, max_features=10)\r\n\treturn models\r\n\r\n# evaluate a given model using cross-validation\r\ndef evaluate_model(model):\r\n\tcv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\r\n\tscores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\r\n\treturn scores\r\n\r\n# define dataset\r\nX, y = get_dataset()\r\n# get the models to evaluate\r\nmodels = get_models()\r\n# evaluate the models and store results\r\nresults, names = list(), list()\r\nfor name, model in models.items():\r\n\tscores = evaluate_model(model)\r\n\tresults.append(scores)\r\n\tnames.append(name)\r\n\tprint('>%s %.3f (%.3f)' % (name, mean(scores), std(scores)))\r\n# plot model performance for comparison\r\npyplot.boxplot(results, labels=names, showmeans=True)\r\npyplot.show()<\/pre>\n>10 0.853 (0.030)\r\n>50 0.885 (0.038)\r\n>100 0.891 (0.034)\r\n>500 0.894 (0.036)\r\n>1000 0.894 (0.034)\r\n>5000 0.896 (0.033)<\/pre>\n
![\"Box](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/11\/Box-Plot-of-Random-Subspace-Ensemble-Size-vs.-Classification-Accuracy.png\")
<\/p>\nExplore Number of Features<\/h3>\n
# explore random subspace ensemble number of features effect on performance\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.model_selection import RepeatedStratifiedKFold\r\nfrom sklearn.ensemble import BaggingClassifier\r\nfrom matplotlib import pyplot\r\n\r\n# get the dataset\r\ndef get_dataset():\r\n\tX, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=5)\r\n\treturn X, y\r\n\r\n# get a list of models to evaluate\r\ndef get_models():\r\n\tmodels = dict()\r\n\tfor n in range(1,21):\r\n\t\tmodels[str(n)] = BaggingClassifier(n_estimators=100, bootstrap=False, max_features=n)\r\n\treturn models\r\n\r\n# evaluate a given model using cross-validation\r\ndef evaluate_model(model):\r\n\tcv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\r\n\tscores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\r\n\treturn scores\r\n\r\n# define dataset\r\nX, y = get_dataset()\r\n# get the models to evaluate\r\nmodels = get_models()\r\n# evaluate the models and store results\r\nresults, names = list(), list()\r\nfor name, model in models.items():\r\n\tscores = evaluate_model(model)\r\n\tresults.append(scores)\r\n\tnames.append(name)\r\n\tprint('>%s %.3f (%.3f)' % (name, mean(scores), std(scores)))\r\n# plot model performance for comparison\r\npyplot.boxplot(results, labels=names, showmeans=True)\r\npyplot.show()<\/pre>\n>1 0.607 (0.036)\r\n>2 0.771 (0.042)\r\n>3 0.837 (0.036)\r\n>4 0.858 (0.037)\r\n>5 0.869 (0.034)\r\n>6 0.883 (0.033)\r\n>7 0.887 (0.038)\r\n>8 0.894 (0.035)\r\n>9 0.893 (0.035)\r\n>10 0.885 (0.038)\r\n>11 0.892 (0.034)\r\n>12 0.883 (0.036)\r\n>13 0.881 (0.044)\r\n>14 0.875 (0.038)\r\n>15 0.869 (0.041)\r\n>16 0.861 (0.044)\r\n>17 0.851 (0.041)\r\n>18 0.831 (0.046)\r\n>19 0.815 (0.046)\r\n>20 0.801 (0.049)<\/pre>\n
![\"Box](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/11\/Box-Plot-of-Random-Subspace-Ensemble-Features-vs.-Classification-Accuracy.png\")
<\/p>\nExplore Alternate Algorithm<\/h3>\n
...\r\n# define the model\r\nmodel = BaggingClassifier(base_estimator=KNeighborsClassifier(), bootstrap=False, max_features=10)<\/pre>\n
# evaluate random subspace ensemble with knn algorithm for classification\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.model_selection import RepeatedStratifiedKFold\r\nfrom sklearn.neighbors import KNeighborsClassifier\r\nfrom sklearn.ensemble import BaggingClassifier\r\n# define dataset\r\nX, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=5)\r\n# define the model\r\nmodel = BaggingClassifier(base_estimator=KNeighborsClassifier(), bootstrap=False, max_features=10)\r\n# define the evaluation procedure\r\ncv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\r\n# evaluate the model\r\nn_scores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1, error_score='raise')\r\n# report performance\r\nprint('Accuracy: %.3f (%.3f)' % (mean(n_scores), std(n_scores)))<\/pre>\nAccuracy: 0.901 (0.032)<\/pre>\n<\/p>\n
Further Reading<\/h2>\n
Papers<\/h3>\n
\n
Books<\/h3>\n
\n
APIs<\/h3>\n
\n
Articles<\/h3>\n
\n
Summary<\/h2>\n
\n
\nAsk your questions in the comments below and I will do my best to answer.<\/p>\n