Automated Machine Learning (AutoML) refers to techniques for automatically discovering well-performing models for predictive modeling tasks with very little user involvement.<\/p>\n
Auto-Sklearn<\/strong> is an open-source library for performing AutoML in Python. It makes use of the popular Scikit-Learn machine learning library for data transforms and machine learning algorithms and uses a Bayesian Optimization search procedure to efficiently discover a top-performing model pipeline for a given dataset.<\/p>\n In this tutorial, you will discover how to use Auto-Sklearn for AutoML with Scikit-Learn machine learning algorithms in Python.<\/p>\n After completing this tutorial, you will know:<\/p>\n Let’s get started.<\/p>\n Auto-Sklearn for Automated Machine Learning in Python This tutorial is divided into four parts; they are:<\/p>\n Automated Machine Learning<\/a>, or AutoML for short, is a process of discovering the best-performing pipeline of data transforms, model, and model configuration for a dataset.<\/p>\n AutoML often involves the use of sophisticated optimization algorithms, such as Bayesian Optimization<\/a>, to efficiently navigate the space of possible models and model configurations and quickly discover what works well for a given predictive modeling task. It allows non-expert machine learning practitioners to quickly and easily discover what works well or even best for a given dataset with very little technical background or direct input.<\/p>\n Auto-Sklearn<\/a> is an open-source Python library for AutoML using machine learning models from the scikit-learn machine learning library.<\/p>\n It was developed by Matthias Feurer<\/a>, et al. and described in their 2015 paper titled “Efficient and Robust Automated Machine Learning<\/a>.”<\/p>\n … we introduce a robust new AutoML system based on scikit-learn (using 15 classifiers, 14 feature preprocessing methods, and 4 data preprocessing methods, giving rise to a structured hypothesis space with 110 hyperparameters).<\/p>\n<\/blockquote>\n — Efficient and Robust Automated Machine Learning<\/a>, 2015.<\/p>\n The benefit of Auto-Sklearn is that, in addition to discovering the data preparation and model that performs for a dataset, it also is able to learn from models that performed well on similar datasets and is able to automatically create an ensemble of top-performing models discovered as part of the optimization process.<\/p>\n This system, which we dub AUTO-SKLEARN, improves on existing AutoML methods by automatically taking into account past performance on similar datasets, and by constructing ensembles from the models evaluated during the optimization.<\/p>\n<\/blockquote>\n — Efficient and Robust Automated Machine Learning<\/a>, 2015.<\/p>\n The authors provide a useful depiction of their system in the paper, provided below.<\/p>\n Overview of the Auto-Sklearn System. The first step is to install the Auto-Sklearn library, which can be achieved using pip, as follows:<\/p>\n Once installed, we can import the library and print the version number to confirm it was installed successfully:<\/p>\n Running the example prints the version number.<\/p>\n Your version number should be the same or higher.<\/p>\n Using Auto-Sklearn is straightforward.<\/p>\n Depending on whether your prediction task is classification or regression, you create and configure an instance of the AutoSklearnClassifier<\/a> or AutoSklearnRegressor<\/a> class, fit it on your dataset, and that’s it. The resulting model can then be used to make predictions directly or saved to file (using pickle) for later use.<\/p>\n There are a ton of configuration options provided as arguments to the AutoSklearn class.<\/p>\n By default, the search will use a train-test split of your dataset during the search, and this default is recommended both for speed and simplicity.<\/p>\n Importantly, you should set the “n_jobs<\/em>” argument to the number of cores in your system, e.g. 8 if you have 8 cores.<\/p>\n The optimization process will run for as long as you allow, measure in minutes. By default, it will run for one hour.<\/p>\n I recommend setting the “time_left_for_this_task<\/em>” argument for the number of seconds you want the process to run. E.g. less than 5-10 minutes is probably plenty for many small predictive modeling tasks (sub 1,000 rows).<\/p>\n We will use 5 minutes (300 seconds) for the examples in this tutorial. We will also limit the time allocated to each model evaluation to 30 seconds via the “per_run_time_limit<\/em>” argument. For example:<\/p>\n You can limit the algorithms considered in the search, as well as the data transforms.<\/p>\n By default, the search will create an ensemble of top-performing models discovered as part of the search. Sometimes, this can lead to overfitting and can be disabled by setting the “ensemble_size<\/em>” argument to 1 and “initial_configurations_via_metalearning<\/em>” to 0.<\/p>\n At the end of a run, the list of models can be accessed, as well as other details.<\/p>\n Perhaps the most useful feature is the sprint_statistics()<\/em> function that summarizes the search and the performance of the final model.<\/p>\n Now that we are familiar with the Auto-Sklearn library, let’s look at some worked examples.<\/p>\n In this section, we will use Auto-Sklearn to discover a model for the sonar dataset.<\/p>\n The sonar dataset is a standard machine learning dataset comprised of 208 rows of data with 60 numerical input variables and a target variable with two class values, e.g. binary classification.<\/p>\n Using a test harness of repeated stratified 10-fold cross-validation with three repeats, a naive model can achieve an accuracy of about 53 percent. A top-performing model can achieve accuracy on this same test harness of about 88 percent. This provides the bounds of expected performance on this dataset.<\/p>\n The dataset involves predicting whether sonar returns indicate a rock or simulated mine.<\/p>\n No need to download the dataset; we will download it automatically as part of our worked examples.<\/p>\n The example below downloads the dataset and summarizes its shape.<\/p>\n Running the example downloads the dataset and splits it into input and output elements. As expected, we can see that there are 208 rows of data with 60 input variables.<\/p>\n We will use Auto-Sklearn to find a good model for the sonar dataset.<\/p>\n First, we will split the dataset into train and test sets and allow the process to find a good model on the training set, then later evaluate the performance of what was found on the holdout test set.<\/p>\n The AutoSklearnClassifier<\/em> is configured to run for 5 minutes with 8 cores and limit each model evaluation to 30 seconds.<\/p>\n The search is then performed on the training dataset.<\/p>\n Afterward, a summary of the search and best-performing model is reported.<\/p>\n Finally, we evaluate the performance of the model that was prepared on the holdout test dataset.<\/p>\n Tying this together, the complete example is listed below.<\/p>\n Running the example will take about five minutes, given the hard limit we imposed on the run.<\/p>\n Note<\/strong>: Your results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n At the end of the run, a summary is printed showing that 1,054 models were evaluated and the estimated performance of the final model was 91 percent.<\/p>\n We then evaluate the model on the holdout dataset and see that classification accuracy of 81.2 percent was achieved, which is reasonably skillful.<\/p>\n In this section, we will use Auto-Sklearn to discover a model for the auto insurance dataset.<\/p>\n The auto insurance dataset is a standard machine learning dataset comprised of 63 rows of data with one numerical input variable and a numerical target variable.<\/p>\n Using a test harness of repeated stratified 10-fold cross-validation with three repeats, a naive model can achieve a mean absolute error (MAE) of about 66. A top-performing model can achieve a MAE on this same test harness of about 28. This provides the bounds of expected performance on this dataset.<\/p>\n The dataset involves predicting the total amount in claims (thousands of Swedish Kronor) given the number of claims for different geographical regions.<\/p>\n No need to download the dataset; we will download it automatically as part of our worked examples.<\/p>\n The example below downloads the dataset and summarizes its shape.<\/p>\n Running the example downloads the dataset and splits it into input and output elements. As expected, we can see that there are 63 rows of data with one input variable.<\/p>\n We will use Auto-Sklearn to find a good model for the auto insurance dataset.<\/p>\n We can use the same process as was used in the previous section, although we will use the AutoSklearnRegressor<\/em> class instead of the AutoSklearnClassifier<\/em>.<\/p>\n By default, the regressor will optimize the R^2 metric.<\/p>\n In this case, we are interested in the mean absolute error, or MAE, which we can specify via the “metric<\/em>” argument when calling the fit()<\/em> function.<\/p>\n The complete example is listed below.<\/p>\n Running the example will take about five minutes, given the hard limit we imposed on the run.<\/p>\n You might see some warning messages during the run and you can safely ignore them, such as:<\/p>\n Note<\/strong>: Your results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n At the end of the run, a summary is printed showing that 1,759 models were evaluated and the estimated performance of the final model was a MAE of 29.<\/p>\n We then evaluate the model on the holdout dataset and see that a MAE of 26 was achieved, which is a great result.<\/p>\n This section provides more resources on the topic if you are looking to go deeper.<\/p>\n In this tutorial, you discovered how to use Auto-Sklearn for AutoML with Scikit-Learn machine learning algorithms in Python.<\/p>\n Specifically, you learned:<\/p>\n Do you have any questions?<\/strong> The post Auto-Sklearn for Automated Machine Learning in Python<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":" Author: Jason Brownlee Automated Machine Learning (AutoML) refers to techniques for automatically discovering well-performing models for predictive modeling tasks with very little user involvement. Auto-Sklearn […] Read More<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":3845,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[24],"tags":[],"_links":{"self":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/3844"}],"collection":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/comments?post=3844"}],"version-history":[{"count":0,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/3844\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media\/3845"}],"wp:attachment":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media?parent=3844"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/categories?post=3844"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/tags?post=3844"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
![\"Auto-Sklearn](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/06\/Auto-Sklearn-for-Automated-Machine-Learning-in-Python.jpg\")
<\/p>\n
Photo by Richard<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
\n
AutoML With Auto-Sklearn<\/h2>\n
\n
\n
![\"Overview](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/03\/Overview-of-the-Auto-Sklearn-System.png\")
<\/p>\n
Taken from: Efficient and Robust Automated Machine Learning, 2015.<\/p>\n<\/div>\nInstall and Using Auto-Sklearn<\/h2>\n
sudo pip install autosklearn<\/pre>\n
# print autosklearn version\r\nimport autosklearn\r\nprint('autosklearn: %s' % autosklearn.__version__)<\/pre>\nautosklearn: 0.6.0<\/pre>\n
...\r\n# define search\r\nmodel = AutoSklearnClassifier()\r\n# perform the search\r\nmodel.fit(X_train, y_train)<\/pre>\n
...\r\n# define search\r\nmodel = AutoSklearnClassifier(time_left_for_this_task=120, per_run_time_limit=30, n_jobs=8)<\/pre>\n
...\r\n# define search\r\nmodel = AutoSklearnClassifier(ensemble_size=1, initial_configurations_via_metalearning=0)<\/pre>\n
...\r\n# summarize performance\r\nprint(model.sprint_statistics())<\/pre>\n
Auto-Sklearn for Classification<\/h2>\n
\n
# summarize the sonar dataset\r\nfrom pandas import read_csv\r\n# load dataset\r\nurl = 'https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/sonar.csv'\r\ndataframe = read_csv(url, header=None)\r\n# split into input and output elements\r\ndata = dataframe.values\r\nX, y = data[:, :-1], data[:, -1]\r\nprint(X.shape, y.shape)<\/pre>\n
(208, 60) (208,)<\/pre>\n
...\r\n# split into train and test sets\r\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=1)<\/pre>\n
...\r\n# define search\r\nmodel = AutoSklearnClassifier(time_left_for_this_task=5*60, per_run_time_limit=30, n_jobs=8)<\/pre>\n
...\r\n# perform the search\r\nmodel.fit(X_train, y_train)<\/pre>\n
...\r\n# summarize\r\nprint(model.sprint_statistics())<\/pre>\n
...\r\n# evaluate best model\r\ny_hat = model.predict(X_test)\r\nacc = accuracy_score(y_test, y_hat)\r\nprint(\"Accuracy: %.3f\" % acc)<\/pre>\n
# example of auto-sklearn for the sonar classification dataset\r\nfrom pandas import read_csv\r\nfrom sklearn.model_selection import train_test_split\r\nfrom sklearn.preprocessing import LabelEncoder\r\nfrom sklearn.metrics import accuracy_score\r\nfrom autosklearn.classification import AutoSklearnClassifier\r\n# load dataset\r\nurl = 'https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/sonar.csv'\r\ndataframe = read_csv(url, header=None)\r\n# print(dataframe.head())\r\n# split into input and output elements\r\ndata = dataframe.values\r\nX, y = data[:, :-1], data[:, -1]\r\n# minimally prepare dataset\r\nX = X.astype('float32')\r\ny = LabelEncoder().fit_transform(y.astype('str'))\r\n# split into train and test sets\r\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=1)\r\n# define search\r\nmodel = AutoSklearnClassifier(time_left_for_this_task=5*60, per_run_time_limit=30, n_jobs=8)\r\n# perform the search\r\nmodel.fit(X_train, y_train)\r\n# summarize\r\nprint(model.sprint_statistics())\r\n# evaluate best model\r\ny_hat = model.predict(X_test)\r\nacc = accuracy_score(y_test, y_hat)\r\nprint(\"Accuracy: %.3f\" % acc)<\/pre>\nauto-sklearn results:\r\nDataset name: f4c282bd4b56d4db7e5f7fe1a6a8edeb\r\nMetric: accuracy\r\nBest validation score: 0.913043\r\nNumber of target algorithm runs: 1054\r\nNumber of successful target algorithm runs: 952\r\nNumber of crashed target algorithm runs: 94\r\nNumber of target algorithms that exceeded the time limit: 8\r\nNumber of target algorithms that exceeded the memory limit: 0<\/pre>\n
Accuracy: 0.812<\/pre>\n<\/p>\n
Auto-Sklearn for Regression<\/h2>\n
\n
# summarize the auto insurance dataset\r\nfrom pandas import read_csv\r\n# load dataset\r\nurl = 'https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/auto-insurance.csv'\r\ndataframe = read_csv(url, header=None)\r\n# split into input and output elements\r\ndata = dataframe.values\r\nX, y = data[:, :-1], data[:, -1]\r\nprint(X.shape, y.shape)<\/pre>\n
(63, 1) (63,)<\/pre>\n
...\r\n# define search\r\nmodel = AutoSklearnRegressor(time_left_for_this_task=5*60, per_run_time_limit=30, n_jobs=8)<\/pre>\n
...\r\n# perform the search\r\nmodel.fit(X_train, y_train, metric=auto_mean_absolute_error)<\/pre>\n
# example of auto-sklearn for the insurance regression dataset\r\nfrom pandas import read_csv\r\nfrom sklearn.model_selection import train_test_split\r\nfrom sklearn.metrics import mean_absolute_error\r\nfrom autosklearn.regression import AutoSklearnRegressor\r\nfrom autosklearn.metrics import mean_absolute_error as auto_mean_absolute_error\r\n# load dataset\r\nurl = 'https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/auto-insurance.csv'\r\ndataframe = read_csv(url, header=None)\r\n# split into input and output elements\r\ndata = dataframe.values\r\ndata = data.astype('float32')\r\nX, y = data[:, :-1], data[:, -1]\r\n# split into train and test sets\r\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=1)\r\n# define search\r\nmodel = AutoSklearnRegressor(time_left_for_this_task=5*60, per_run_time_limit=30, n_jobs=8)\r\n# perform the search\r\nmodel.fit(X_train, y_train, metric=auto_mean_absolute_error)\r\n# summarize\r\nprint(model.sprint_statistics())\r\n# evaluate best model\r\ny_hat = model.predict(X_test)\r\nmae = mean_absolute_error(y_test, y_hat)\r\nprint(\"MAE: %.3f\" % mae)<\/pre>\nTarget Algorithm returned NaN or inf as quality. Algorithm run is treated as CRASHED, cost is set to 1.0 for quality scenarios. (Change value through \"cost_for_crash\"-option.)<\/pre>\n<\/p>\n
auto-sklearn results:\r\nDataset name: ff51291d93f33237099d48c48ee0f9ad\r\nMetric: mean_absolute_error\r\nBest validation score: 29.911203\r\nNumber of target algorithm runs: 1759\r\nNumber of successful target algorithm runs: 1362\r\nNumber of crashed target algorithm runs: 394\r\nNumber of target algorithms that exceeded the time limit: 3\r\nNumber of target algorithms that exceeded the memory limit: 0<\/pre>\n
MAE: 26.498<\/pre>\n<\/p>\n
Further Reading<\/h2>\n
\n
Summary<\/h2>\n
\n
\nAsk your questions in the comments below and I will do my best to answer.<\/p>\n