Machine learning model performance often improves with dataset size for predictive modeling.<\/p>\n
This depends on the specific datasets and on the choice of model, although it often means that using more data can result in better performance and that discoveries made using smaller datasets to estimate model performance often scale to using larger datasets.<\/p>\n
The problem is the relationship is unknown for a given dataset and model, and may not exist for some datasets and models. Additionally, if such a relationship does exist, there may be a point or points of diminishing returns where adding more data may not improve model performance or where datasets are too small to effectively capture the capability of a model at a larger scale.<\/p>\n
These issues can be addressed by performing a sensitivity analysis<\/strong> to quantify the relationship between dataset size and model performance. Once calculated, we can interpret the results of the analysis and make decisions about how much data is enough, and how small a dataset may be to effectively estimate performance on larger datasets.<\/p>\n In this tutorial, you will discover how to perform a sensitivity analysis of dataset size vs. model performance.<\/p>\n After completing this tutorial, you will know:<\/p>\n Let\u2019s get started.<\/p>\n Sensitivity Analysis of Dataset Size vs. Model Performance This tutorial is divided into three parts; they are:<\/p>\n The amount of training data required for a machine learning predictive model is an open question.<\/p>\n It depends on your choice of model, on the way you prepare the data, and on the specifics of the data itself.<\/p>\n For more on the challenge of selecting a training dataset size, see the tutorial:<\/p>\n One way to approach this problem is to perform a sensitivity analysis<\/a> and discover how the performance of your model on your dataset varies with more or less data.<\/p>\n This might involve evaluating the same model with different sized datasets and looking for a relationship between dataset size and performance or a point of diminishing returns.<\/p>\n Typically, there is a strong relationship between training dataset size and model performance, especially for nonlinear models. The relationship often involves an improvement in performance to a point and a general reduction in the expected variance of the model as the dataset size is increased.<\/p>\n Knowing this relationship for your model and dataset can be helpful for a number of reasons, such as:<\/p>\n You can evaluate a large number of models and model configurations quickly on a smaller sample of the dataset with confidence that the performance will likely generalize in a specific way to a larger training dataset.<\/p>\n This may allow evaluating many more models and configurations than you may otherwise be able to given the time available, and in turn, perhaps discover a better overall performing model.<\/p>\n You may also be able to generalize and estimate the expected performance of model performance to much larger datasets and estimate whether it is worth the effort or expense of gathering more training data.<\/p>\n Now that we are familiar with the idea of performing a sensitivity analysis of model performance to dataset size, let\u2019s look at a worked example.<\/p>\n Before we dive into a sensitivity analysis, let\u2019s select a dataset and baseline model for the investigation.<\/p>\n We will use a synthetic binary (two-class) classification dataset in this tutorial. This is ideal as it allows us to scale the number of generated samples for the same problem as needed.<\/p>\n The make_classification() scikit-learn function<\/a> can be used to create a synthetic classification dataset. In this case, we will use 20 input features (columns) and generate 1,000 samples (rows). The seed for the pseudo-random number generator is fixed to ensure the same base \u201cproblem\u201d is used each time samples are generated.<\/p>\n The example below generates the synthetic classification dataset and summarizes the shape of the generated data.<\/p>\n Running the example generates the data and reports the size of the input and output components, confirming the expected shape.<\/p>\n Next, we can evaluate a predictive model on this dataset.<\/p>\n We will use a decision tree (DecisionTreeClassifier<\/a>) as the predictive model. It was chosen because it is a nonlinear algorithm and has a high variance, which means that we would expect performance to improve with increases in the size of the training dataset.<\/p>\n We will use a best practice of repeated stratified k-fold cross-validation<\/a> to evaluate the model on the dataset, with 3 repeats and 10 folds.<\/p>\n The complete example of evaluating the decision tree model on the synthetic classification dataset is listed below.<\/p>\n Running the example creates the dataset then estimates the performance of the model on the problem using the chosen test harness.<\/p>\n Note<\/strong>: Your results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n In this case, we can see that the mean classification accuracy is about 82.7%.<\/p>\n Next, let\u2019s look at how we might perform a sensitivity analysis of dataset size on model performance.<\/p>\n The previous section showed how to evaluate a chosen model on the available dataset.<\/p>\n It raises questions, such as:<\/p>\n Will the model perform better on more data?<\/p>\n<\/blockquote>\n More generally, we may have sophisticated questions such as:<\/p>\n Does the estimated performance hold on smaller or larger samples from the problem domain?<\/p>\n<\/blockquote>\n These are hard questions to answer, but we can approach them by using a sensitivity analysis. Specifically, we can use a sensitivity analysis to learn:<\/p>\n How sensitive is model performance to dataset size?<\/p>\n<\/blockquote>\n Or more generally:<\/p>\n What is the relationship of dataset size to model performance?<\/p>\n<\/blockquote>\n There are many ways to perform a sensitivity analysis, but perhaps the simplest approach is to define a test harness to evaluate model performance and then evaluate the same model on the same problem with differently sized datasets.<\/p>\n This will allow the train and test portions of the dataset to increase with the size of the overall dataset.<\/p>\n To make the code easier to read, we will split it up into functions.<\/p>\n First, we can define a function that will prepare (or load) the dataset of a given size. The number of rows in the dataset is specified by an argument to the function.<\/p>\n If you are using this code as a template, this function can be changed to load your dataset from file and select a random sample of a given size.<\/p>\n Next, we need a function to evaluate a model on a loaded dataset.<\/p>\n We will define a function that takes a dataset and returns a summary of the performance of the model evaluated using the test harness on the dataset.<\/p>\n This function is listed below, taking the input and output elements of a dataset and returning the mean and standard deviation of the decision tree model on the dataset.<\/p>\n Next, we can define a range of different dataset sizes to evaluate.<\/p>\n The sizes should be chosen proportional to the amount of data you have available and the amount of running time you are willing to expend.<\/p>\n In this case, we will keep the sizes modest to limit running time, from 50 to one million rows on a rough log10 scale.<\/p>\n Next, we can enumerate each dataset size, create the dataset, evaluate a model on the dataset, and store the results for later analysis.<\/p>\n Next, we can summarize the relationship between the dataset size and model performance.<\/p>\n In this case, we will simply plot the result with error bars so we can spot any trends visually.<\/p>\n We will use the standard deviation as a measure of uncertainty on the estimated model performance. This can be achieved by multiplying the value by 2 to cover approximately 95% of the expected performance if the performance follows a normal distribution.<\/p>\n This can be shown on the plot as an error bar around the mean expected performance for a dataset size.<\/p>\n To make the plot more readable, we can change the scale of the x-axis to log, given that our dataset sizes are on a rough log10 scale.<\/p>\n And that\u2019s it.<\/p>\n We would generally expect mean model performance to increase with dataset size. We would also expect the uncertainty in model performance to decrease with dataset size.<\/p>\n Tying this all together, the complete example of performing a sensitivity analysis of dataset size on model performance is listed below.<\/p>\n Running the example reports the status along the way of dataset size vs. estimated model performance.<\/p>\n Note<\/strong>: Your results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n In this case, we can see the expected trend of increasing mean model performance with dataset size and decreasing model variance measured using the standard deviation of classification accuracy.<\/p>\n We can see that there is perhaps a point of diminishing returns in estimating model performance at perhaps 10,000 or 50,000 rows.<\/p>\n Specifically, we do see an improvement in performance with more rows, but we can probably capture this relationship with little variance with 10K or 50K rows of data.<\/p>\n We can also see a drop-off in estimated performance with 1,000,000 rows of data, suggesting that we are probably maxing out the capability of the model above 100,000 rows and are instead measuring statistical noise in the estimate.<\/p>\n This might mean an upper bound on expected performance and likely that more data beyond this point will not improve the specific model and configuration on the chosen test harness.<\/p>\n The plot makes the relationship between dataset size and estimated model performance much clearer.<\/p>\n The relationship is nearly linear with a log dataset size. The change in the uncertainty shown as the error bar also dramatically decreases on the plot from very large values with 50 or 100 samples, to modest values with 5,000 and 10,000 samples and practically gone beyond these sizes.<\/p>\n Given the modest spread with 5,000 and 10,000 samples and the practically log-linear relationship, we could probably get away with using 5K or 10K rows to approximate model performance.<\/p>\n Line Plot With Error Bars of Dataset Size vs. Model Performance<\/p>\n<\/div>\n We could use these findings as the basis for testing additional model configurations and even different model types.<\/p>\n The danger is that different models may perform very differently with more or less data and it may be wise to repeat the sensitivity analysis with a different chosen model to confirm the relationship holds. Alternately, it may be interesting to repeat the analysis with a suite of different model types.<\/p>\n This section provides more resources on the topic if you are looking to go deeper.<\/p>\n In this tutorial, you discovered how to perform a sensitivity analysis of dataset size vs. model performance.<\/p>\n Specifically, you learned:<\/p>\n Do you have any questions?<\/strong> The post Sensitivity Analysis of Dataset Size vs. Model Performance<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"\n
![\"Sensitivity](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/05\/Sensitivity-Analysis-of-Dataset-Size-vs.-Model-Performance.jpg\")
<\/p>\n
Photo by Graeme Churchard<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
\n
Dataset Size Sensitivity Analysis<\/h2>\n
\n
\n
Synthetic Prediction Task and Baseline Model<\/h2>\n
# test classification dataset\r\nfrom sklearn.datasets import make_classification\r\n# define dataset\r\nX, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=1)\r\n# summarize the dataset\r\nprint(X.shape, y.shape)<\/pre>\n
(1000, 20) (1000,)<\/pre>\n
# evaluate a decision tree model on the synthetic classification dataset\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.model_selection import RepeatedStratifiedKFold\r\nfrom sklearn.tree import DecisionTreeClassifier\r\n# load dataset\r\nX, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=1)\r\n# define model evaluation procedure\r\ncv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\r\n# define model\r\nmodel = DecisionTreeClassifier()\r\n# evaluate model\r\nscores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\r\n# report performance\r\nprint('Mean Accuracy: %.3f (%.3f)' % (scores.mean(), scores.std()))<\/pre>\nMean Accuracy: 0.827 (0.042)<\/pre>\n
Sensitivity Analysis of Dataset Size<\/h2>\n
\n
\n
\n
\n
# load dataset\r\ndef load_dataset(n_samples):\r\n\t# define the dataset\r\n\tX, y = make_classification(n_samples=int(n_samples), n_features=20, n_informative=15, n_redundant=5, random_state=1)\r\n\treturn X, y<\/pre>\n
# evaluate a model\r\ndef evaluate_model(X, y):\r\n\t# define model evaluation procedure\r\n\tcv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\r\n\t# define model\r\n\tmodel = DecisionTreeClassifier()\r\n\t# evaluate model\r\n\tscores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\r\n\t# return summary stats\r\n\treturn [scores.mean(), scores.std()]<\/pre>\n
...\r\n# define number of samples to consider\r\nsizes = [50, 100, 500, 1000, 5000, 10000, 50000, 100000, 500000, 1000000]<\/pre>\n
...\r\n# evaluate each number of samples\r\nmeans, stds = list(), list()\r\nfor n_samples in sizes:\r\n\t# get a dataset\r\n\tX, y = load_dataset(n_samples)\r\n\t# evaluate a model on this dataset size\r\n\tmean, std = evaluate_model(X, y)\r\n\t# store\r\n\tmeans.append(mean)\r\n\tstds.append(std)<\/pre>\n
...\r\n# define error bar as 2 standard deviations from the mean or 95%\r\nerr = [min(1, s * 2) for s in stds]\r\n# plot dataset size vs mean performance with error bars\r\npyplot.errorbar(sizes, means, yerr=err, fmt='-o')<\/pre>\n
...\r\n# change the scale of the x-axis to log\r\nax = pyplot.gca()\r\nax.set_xscale(\"log\", nonpositive='clip')\r\n# show the plot\r\npyplot.show()<\/pre>\n
# sensitivity analysis of model performance to dataset size\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.model_selection import RepeatedStratifiedKFold\r\nfrom sklearn.tree import DecisionTreeClassifier\r\nfrom matplotlib import pyplot\r\n\r\n# load dataset\r\ndef load_dataset(n_samples):\r\n\t# define the dataset\r\n\tX, y = make_classification(n_samples=int(n_samples), n_features=20, n_informative=15, n_redundant=5, random_state=1)\r\n\treturn X, y\r\n\r\n# evaluate a model\r\ndef evaluate_model(X, y):\r\n\t# define model evaluation procedure\r\n\tcv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\r\n\t# define model\r\n\tmodel = DecisionTreeClassifier()\r\n\t# evaluate model\r\n\tscores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\r\n\t# return summary stats\r\n\treturn [scores.mean(), scores.std()]\r\n\r\n# define number of samples to consider\r\nsizes = [50, 100, 500, 1000, 5000, 10000, 50000, 100000, 500000, 1000000]\r\n# evaluate each number of samples\r\nmeans, stds = list(), list()\r\nfor n_samples in sizes:\r\n\t# get a dataset\r\n\tX, y = load_dataset(n_samples)\r\n\t# evaluate a model on this dataset size\r\n\tmean, std = evaluate_model(X, y)\r\n\t# store\r\n\tmeans.append(mean)\r\n\tstds.append(std)\r\n\t# summarize performance\r\n\tprint('>%d: %.3f (%.3f)' % (n_samples, mean, std))\r\n# define error bar as 2 standard deviations from the mean or 95%\r\nerr = [min(1, s * 2) for s in stds]\r\n# plot dataset size vs mean performance with error bars\r\npyplot.errorbar(sizes, means, yerr=err, fmt='-o')\r\n# change the scale of the x-axis to log\r\nax = pyplot.gca()\r\nax.set_xscale(\"log\", nonpositive='clip')\r\n# show the plot\r\npyplot.show()<\/pre>\n>50: 0.673 (0.141)\r\n>100: 0.703 (0.135)\r\n>500: 0.809 (0.055)\r\n>1000: 0.826 (0.044)\r\n>5000: 0.835 (0.016)\r\n>10000: 0.866 (0.011)\r\n>50000: 0.900 (0.005)\r\n>100000: 0.912 (0.003)\r\n>500000: 0.938 (0.001)\r\n>1000000: 0.936 (0.001)<\/pre>\n
![\"Line](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/02\/Line-Plot-with-Error-Bars-of-Dataset-Size-vs-Model-Performance.png\")
<\/p>\nFurther Reading<\/h2>\n
Tutorials<\/h3>\n
\n
APIs<\/h3>\n
\n
Articles<\/h3>\n
\n
Summary<\/h2>\n
\n
\nAsk your questions in the comments below and I will do my best to answer.<\/p>\n