The Leave-One-Out Cross-Validation<\/strong>, or LOOCV<\/strong>, procedure is used to estimate the performance of machine learning algorithms when they are used to make predictions on data not used to train the model.<\/p>\n It is a computationally expensive procedure to perform, although it results in a reliable and unbiased estimate of model performance. Although simple to use and no configuration to specify, there are times when the procedure should not be used, such as when you have a very large dataset or a computationally expensive model to evaluate.<\/p>\n In this tutorial, you will discover how to evaluate machine learning models using leave-one-out cross-validation.<\/p>\n After completing this tutorial, you will know:<\/p>\n Let’s get started.<\/p>\n LOOCV for Evaluating Machine Learning Algorithms This tutorial is divided into three parts; they are:<\/p>\n Cross-validation, or k-fold cross-validation, is a procedure used to estimate the performance of a machine learning algorithm when making predictions on data not used during the training of the model.<\/p>\n The cross-validation has a single hyperparameter “k<\/em>” that controls the number of subsets that a dataset is split into. Once split, each subset is given the opportunity to be used as a test set while all other subsets together are used as a training dataset.<\/p>\n This means that k-fold cross-validation involves fitting and evaluating k<\/em> models. This, in turn, provides k estimates of a model’s performance on the dataset, which can be reported using summary statistics such as the mean and standard deviation. This score can then be used to compare and ultimately select a model and configuration to use as the “final model<\/em>” for a dataset.<\/p>\n Typical values for k are k=3, k=5, and k=10, with 10 representing the most common value. This is because, given extensive testing, 10-fold cross-validation provides a good balance of low computational cost and low bias in the estimate of model performance as compared to other k values and a single train-test split.<\/p>\n For more on k-fold cross-validation, see the tutorial:<\/p>\n Leave-one-out cross-validation, or LOOCV, is a configuration of k-fold cross-validation where k<\/em> is set to the number of examples in the dataset.<\/p>\n LOOCV is an extreme version of k-fold cross-validation that has the maximum computational cost. It requires one model to be created and evaluated for each example in the training dataset.<\/p>\n The benefit of so many fit and evaluated models is a more robust estimate of model performance as each row of data is given an opportunity to represent the entirety of the test dataset.<\/p>\n Given the computational cost, LOOCV is not appropriate for very large datasets such as more than tens or hundreds of thousands of examples, or for models that are costly to fit, such as neural networks.<\/p>\n Given the improved estimate of model performance, LOOCV is appropriate when an accurate estimate of model performance is critical. This particularly case when the dataset is small, such as less than thousands of examples, can lead to model overfitting during training and biased estimates of model performance.<\/p>\n Further, given that no sampling of the training dataset is used, this estimation procedure is deterministic, unlike train-test splits and other k-fold cross-validation confirmations that provide a stochastic estimate of model performance.<\/p>\n Once models have been evaluated using LOOCV and a final model and configuration chosen, a final model is then fit on all available data and used to make predictions on new data.<\/p>\n Now that we are familiar with the LOOCV procedure, let’s look at how we can use the method in Python.<\/p>\n The scikit-learn Python machine learning library provides an implementation of the LOOCV via the LeaveOneOut class<\/a>.<\/p>\n The method has no configuration, therefore, no arguments are provided to create an instance of the class.<\/p>\n Once created, the split()<\/em> function can be called and provided the dataset to enumerate.<\/p>\n Each iteration will return the row indices that can be used for the train and test sets from the provided dataset.<\/p>\n These indices can be used on the input (X<\/em>) and output (y<\/em>) columns of the dataset array to split the dataset.<\/p>\n The training set can be used to fit a model and the test set can be used to evaluate it by first making a prediction and calculating a performance metric on the predicted values versus the expected values.<\/p>\n Scores can be saved from each evaluation and a final mean estimate of model performance can be presented.<\/p>\n We can tie this together and demonstrate how to use LOOCV to evaluate a RandomForestClassifier<\/a> model for a synthetic binary classification dataset created with the make_blobs() function<\/a>.<\/p>\n The complete example is listed below.<\/p>\n Running the example manually estimates the performance of the random forest classifier on the synthetic dataset.<\/p>\n Given that the dataset has 100 examples, it means that 100 train\/test splits of the dataset were created, with each single row of the dataset given an opportunity to be used as the test set. Similarly, 100 models are created and evaluated.<\/p>\n The classification accuracy across all predictions is then reported, in this case as 99 percent.<\/p>\n A downside of enumerating the folds manually is that it is slow and involves a lot of code that could introduce bugs.<\/p>\n An alternative to evaluating a model using LOOCV is to use the cross_val_score() function<\/a>.<\/p>\n This function takes the model, the dataset, and the instantiated LOOCV object set via the “cv<\/em>” argument. A sample of accuracy scores is then returned that can be summarized by calculating the mean and standard deviation.<\/p>\n We can also set the “n_jobs<\/em>” argument to -1 to use all CPU cores, greatly decreasing the computational cost in fitting and evaluating so many models.<\/p>\n The example below demonstrates evaluating the RandomForestClassifier<\/em> using LOOCV on the same synthetic dataset using the cross_val_score()<\/em> function.<\/p>\n Running the example automatically estimates the performance of the random forest classifier on the synthetic dataset.<\/p>\n The mean classification accuracy across all folds matches our manual estimate previously.<\/p>\n Now that we are familiar with how to use the LeaveOneOut class, let’s look at how we can use it to evaluate a machine learning model on real datasets.<\/p>\n In this section, we will explore using the LOOCV procedure to evaluate machine learning models on standard classification and regression predictive modeling datasets.<\/p>\n We will demonstrate how to use LOOCV to evaluate a random forest algorithm on the sonar dataset.<\/p>\n The sonar dataset is a standard machine learning dataset comprising 208 rows of data with 60 numerical input variables and a target variable with two class values, e.g. binary classification.<\/p>\n The dataset involves predicting whether sonar returns indicate a rock or simulated mine.<\/p>\n No need to download the dataset; we will download it automatically as part of our worked examples.<\/p>\n The example below downloads the dataset and summarizes its shape.<\/p>\n Running the example downloads the dataset and splits it into input and output elements. As expected, we can see that there are 208 rows of data with 60 input variables.<\/p>\n We can now evaluate a model using LOOCV.<\/p>\n First, the loaded dataset must be split into input and output components.<\/p>\n Next, we define the LOOCV procedure.<\/p>\n We can then define the model to evaluate.<\/p>\n Then use the cross_val_score()<\/em> function to enumerate the folds, fit models, then make and evaluate predictions. We can then report the mean and standard deviation of model performance.<\/p>\n Tying this together, the complete example is listed below.<\/p>\n Running the example first loads the dataset and confirms the number of rows in the input and output elements.<\/p>\n The model is then evaluated using LOOCV and the estimated performance when making predictions on new data has an accuracy of about 82.2 percent.<\/p>\n We will demonstrate how to use LOOCV to evaluate a random forest algorithm on the housing dataset.<\/p>\n The housing dataset is a standard machine learning dataset comprising 506 rows of data with 13 numerical input variables and a numerical target variable.<\/p>\n The dataset involves predicting the house price given details of the house’s suburb in the American city of Boston.<\/p>\n No need to download the dataset; we will download it automatically as part of our worked examples.<\/p>\n The example below downloads and loads the dataset as a Pandas DataFrame and summarizes the shape of the dataset.<\/p>\n Running the example confirms the 506 rows of data and 13 input variables and single numeric target variables (14 in total).<\/p>\n We can now evaluate a model using LOOCV.<\/p>\n First, the loaded dataset must be split into input and output components.<\/p>\n Next, we define the LOOCV procedure.<\/p>\n We can then define the model to evaluate.<\/p>\n Then use the cross_val_score()<\/em> function to enumerate the folds, fit models, then make and evaluate predictions. We can then report the mean and standard deviation of model performance.<\/p>\n In this case, we use the mean absolute error (MAE) performance metric appropriate for regression.<\/p>\n Tying this together, the complete example is listed below.<\/p>\n Running the example first loads the dataset and confirms the number of rows in the input and output elements.<\/p>\n The model is evaluated using LOOCV and the performance of the model when making predictions on new data is a mean absolute error of about 2.180 (thousands of dollars).<\/p>\n This section provides more resources on the topic if you are looking to go deeper.<\/p>\n In this tutorial, you discovered how to evaluate machine learning models using leave-one-out cross-validation.<\/p>\n Specifically, you learned:<\/p>\n Do you have any questions?<\/strong> The post LOOCV for Evaluating Machine Learning Algorithms<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"\n
![\"LOOCV](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/07\/LOOCV-for-Evaluating-Machine-Learning-Algorithms.jpg\")
<\/p>\n
Photo by Heather Harvey<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
\n
\n
LOOCV Model Evaluation<\/h2>\n
\n
\n
\n
LOOCV Procedure in Scikit-Learn<\/h2>\n
...\r\n# create loocv procedure\r\ncv = LeaveOneOut()<\/pre>\n
...\r\nfor train_ix, test_ix in cv.split(X):\r\n\t...<\/pre>\n
...\r\n# split data\r\nX_train, X_test = X[train_ix, :], X[test_ix, :]\r\ny_train, y_test = y[train_ix], y[test_ix]<\/pre>\n
...\r\n# fit model\r\nmodel = RandomForestClassifier(random_state=1)\r\nmodel.fit(X_train, y_train)\r\n# evaluate model\r\nyhat = model.predict(X_test)<\/pre>\n
# loocv to manually evaluate the performance of a random forest classifier\r\nfrom sklearn.datasets import make_blobs\r\nfrom sklearn.model_selection import LeaveOneOut\r\nfrom sklearn.ensemble import RandomForestClassifier\r\nfrom sklearn.metrics import accuracy_score\r\n# create dataset\r\nX, y = make_blobs(n_samples=100, random_state=1)\r\n# create loocv procedure\r\ncv = LeaveOneOut()\r\n# enumerate splits\r\ny_true, y_pred = list(), list()\r\nfor train_ix, test_ix in cv.split(X):\r\n\t# split data\r\n\tX_train, X_test = X[train_ix, :], X[test_ix, :]\r\n\ty_train, y_test = y[train_ix], y[test_ix]\r\n\t# fit model\r\n\tmodel = RandomForestClassifier(random_state=1)\r\n\tmodel.fit(X_train, y_train)\r\n\t# evaluate model\r\n\tyhat = model.predict(X_test)\r\n\t# store\r\n\ty_true.append(y_test[0])\r\n\ty_pred.append(yhat[0])\r\n# calculate accuracy\r\nacc = accuracy_score(y_true, y_pred)\r\nprint('Accuracy: %.3f' % acc)<\/pre>\nAccuracy: 0.990<\/pre>\n
# loocv to automatically evaluate the performance of a random forest classifier\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom sklearn.datasets import make_blobs\r\nfrom sklearn.model_selection import LeaveOneOut\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.ensemble import RandomForestClassifier\r\n# create dataset\r\nX, y = make_blobs(n_samples=100, random_state=1)\r\n# create loocv procedure\r\ncv = LeaveOneOut()\r\n# create model\r\nmodel = RandomForestClassifier(random_state=1)\r\n# evaluate model\r\nscores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\r\n# report performance\r\nprint('Accuracy: %.3f (%.3f)' % (mean(scores), std(scores)))<\/pre>\nAccuracy: 0.990 (0.099)<\/pre>\n
LOOCV to Evaluate Machine Learning Models<\/h2>\n
LOOCV for Classification<\/h3>\n
\n
# summarize the sonar dataset\r\nfrom pandas import read_csv\r\n# load dataset\r\nurl = 'https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/sonar.csv'\r\ndataframe = read_csv(url, header=None)\r\n# split into input and output elements\r\ndata = dataframe.values\r\nX, y = data[:, :-1], data[:, -1]\r\nprint(X.shape, y.shape)<\/pre>\n
(208, 60) (208,)<\/pre>\n
...\r\n# split into inputs and outputs\r\nX, y = data[:, :-1], data[:, -1]\r\nprint(X.shape, y.shape)<\/pre>\n
...\r\n# create loocv procedure\r\ncv = LeaveOneOut()<\/pre>\n
...\r\n# create model\r\nmodel = RandomForestClassifier(random_state=1)<\/pre>\n
...\r\n# evaluate model\r\nscores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\r\n# report performance\r\nprint('Accuracy: %.3f (%.3f)' % (mean(scores), std(scores)))<\/pre>\n# loocv evaluate random forest on the sonar dataset\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom pandas import read_csv\r\nfrom sklearn.model_selection import LeaveOneOut\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.ensemble import RandomForestClassifier\r\n# load dataset\r\nurl = 'https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/sonar.csv'\r\ndataframe = read_csv(url, header=None)\r\ndata = dataframe.values\r\n# split into inputs and outputs\r\nX, y = data[:, :-1], data[:, -1]\r\nprint(X.shape, y.shape)\r\n# create loocv procedure\r\ncv = LeaveOneOut()\r\n# create model\r\nmodel = RandomForestClassifier(random_state=1)\r\n# evaluate model\r\nscores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\r\n# report performance\r\nprint('Accuracy: %.3f (%.3f)' % (mean(scores), std(scores)))<\/pre>\n(208, 60) (208,)\r\nAccuracy: 0.822 (0.382)<\/pre>\n<\/p>\n
LOOCV for Regression<\/h3>\n
\n
# load and summarize the housing dataset\r\nfrom pandas import read_csv\r\n# load dataset\r\nurl = 'https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/housing.csv'\r\ndataframe = read_csv(url, header=None)\r\n# summarize shape\r\nprint(dataframe.shape)<\/pre>\n
(506, 14)<\/pre>\n
...\r\n# split into inputs and outputs\r\nX, y = data[:, :-1], data[:, -1]\r\nprint(X.shape, y.shape)<\/pre>\n
...\r\n# create loocv procedure\r\ncv = LeaveOneOut()<\/pre>\n
...\r\n# create model\r\nmodel = RandomForestRegressor(random_state=1)<\/pre>\n
...\r\n# evaluate model\r\nscores = cross_val_score(model, X, y, scoring='neg_mean_absolute_error', cv=cv, n_jobs=-1)\r\n# force positive\r\nscores = absolute(scores)\r\n# report performance\r\nprint('MAE: %.3f (%.3f)' % (mean(scores), std(scores)))<\/pre>\n# loocv evaluate random forest on the housing dataset\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom numpy import absolute\r\nfrom pandas import read_csv\r\nfrom sklearn.model_selection import LeaveOneOut\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.ensemble import RandomForestRegressor\r\n# load dataset\r\nurl = 'https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/housing.csv'\r\ndataframe = read_csv(url, header=None)\r\ndata = dataframe.values\r\n# split into inputs and outputs\r\nX, y = data[:, :-1], data[:, -1]\r\nprint(X.shape, y.shape)\r\n# create loocv procedure\r\ncv = LeaveOneOut()\r\n# create model\r\nmodel = RandomForestRegressor(random_state=1)\r\n# evaluate model\r\nscores = cross_val_score(model, X, y, scoring='neg_mean_absolute_error', cv=cv, n_jobs=-1)\r\n# force positive\r\nscores = absolute(scores)\r\n# report performance\r\nprint('MAE: %.3f (%.3f)' % (mean(scores), std(scores)))<\/pre>\n(506, 13) (506,)\r\nMAE: 2.180 (2.346)<\/pre>\n<\/p>\n
Further Reading<\/h2>\n
Tutorials<\/h3>\n
\n
APIs<\/h3>\n
\n
Summary<\/h2>\n
\n
\nAsk your questions in the comments below and I will do my best to answer.<\/p>\n