{"id":559,"date":"2018-05-29T19:00:39","date_gmt":"2018-05-29T19:00:39","guid":{"rendered":"https:\/\/www.aiproblog.com\/index.php\/2018\/05\/29\/prediction-intervals-for-machine-learning\/"},"modified":"2018-05-29T19:00:39","modified_gmt":"2018-05-29T19:00:39","slug":"prediction-intervals-for-machine-learning","status":"publish","type":"post","link":"https:\/\/www.aiproblog.com\/index.php\/2018\/05\/29\/prediction-intervals-for-machine-learning\/","title":{"rendered":"Prediction Intervals for Machine Learning"},"content":{"rendered":"<p>Author: Jason Brownlee<\/p>\n<div>\n<p>A prediction from a machine learning perspective is a single point that hides the uncertainty of that prediction.<\/p>\n<p>Prediction intervals provide a way to quantify and communicate the uncertainty in a prediction. They are different from confidence intervals that instead seek to quantify the uncertainty in a population parameter such as a mean or standard deviation. Prediction intervals describe the uncertainty for a single specific outcome.<\/p>\n<p>In this tutorial, you will discover the prediction interval and how to calculate it for a simple linear regression model.<\/p>\n<p>After completing this tutorial, you will know:<\/p>\n<ul>\n<li>That a prediction interval quantifies the uncertainty of a single point prediction.<\/li>\n<li>That prediction intervals can be estimated analytically for simple models, but are more challenging for nonlinear machine learning models.<\/li>\n<li>How to calculate the prediction interval for a simple linear regression model.<\/li>\n<\/ul>\n<p>Let\u2019s get started.<\/p>\n<div id=\"attachment_5331\" style=\"width: 650px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-5331\" src=\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2018\/05\/Prediction-Intervals-for-Machine-Learning.jpg\" alt=\"Prediction Intervals for Machine Learning\" width=\"640\" height=\"427\" srcset=\"http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2018\/05\/Prediction-Intervals-for-Machine-Learning.jpg 640w, http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2018\/05\/Prediction-Intervals-for-Machine-Learning-300x200.jpg 300w\" sizes=\"(max-width: 640px) 100vw, 640px\"><\/p>\n<p class=\"wp-caption-text\">Prediction Intervals for Machine Learning<br \/>Photo by <a href=\"https:\/\/www.flickr.com\/photos\/jim_bendon_1957\/7899315252\/\">Jim Bendon<\/a>, some rights reserved.<\/p>\n<\/div>\n<h2>Tutorial Overview<\/h2>\n<p>This tutorial is divided into 5 parts; they are:<\/p>\n<ol>\n<li>What Is Wrong With a Point Estimate?<\/li>\n<li>What Is a Prediction Interval?<\/li>\n<li>How to Calculate a Prediction Interval<\/li>\n<li>Prediction Interval for Linear Regression<\/li>\n<li>Worked Example<\/li>\n<\/ol>\n<p><!-- Start shortcoder --><\/p>\n<div class=\"woo-sc-hr\"><\/div>\n<p><center><\/p>\n<h3>Need help with Statistics for Machine Learning?<\/h3>\n<p>Take my free 7-day email crash course now (with sample code).<\/p>\n<p>Click to sign-up and also get a free PDF Ebook version of the course.<\/p>\n<p><a href=\"https:\/\/machinelearningmastery.lpages.co\/leadbox\/142f75173f72a2%3A164f8be4f346dc\/5750943224168448\/\" target=\"_blank\" style=\"background: rgb(255, 206, 10); color: rgb(255, 255, 255); text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-weight: bold; font-size: 16px; line-height: 20px; padding: 10px; display: inline-block; max-width: 300px; border-radius: 5px; text-shadow: rgba(0, 0, 0, 0.25) 0px -1px 1px; box-shadow: rgba(255, 255, 255, 0.5) 0px 1px 3px inset, rgba(0, 0, 0, 0.5) 0px 1px 3px;\">Download Your FREE Mini-Course<\/a><script data-leadbox=\"142f75173f72a2:164f8be4f346dc\" data-url=\"https:\/\/machinelearningmastery.lpages.co\/leadbox\/142f75173f72a2%3A164f8be4f346dc\/5750943224168448\/\" data-config=\"%7B%7D\" type=\"text\/javascript\" src=\"https:\/\/machinelearningmastery.lpages.co\/leadbox-1526328103.js\"><\/script><\/p>\n<p><\/center><\/p>\n<div class=\"woo-sc-hr\"><\/div>\n<p><!-- End shortcoder v4.1.7--><\/p>\n<h2>Why Calculate a Prediction Interval?<\/h2>\n<p>In predictive modeling, a prediction or a forecast is a single outcome value given some input variables.<\/p>\n<p>For example:<\/p>\n<pre class=\"crayon-plain-tag\">yhat = model.predict(X)<\/pre>\n<p>Where <em>yhat<\/em> is the estimated outcome or prediction made by the trained model for the given input data <em>X<\/em>.<\/p>\n<p>This is a point prediction.<\/p>\n<p>By definition, it is an estimate or an approximation and contains some uncertainty.<\/p>\n<p>The uncertainty comes from the errors in the model itself and noise in the input data. The model is an approximation of the relationship between the input variables and the output variables.<\/p>\n<p>Given the process used to choose and tune the model, it will be the best approximation made given available information, but it will still make errors. Data from the domain will naturally obscure the underlying and unknown relationship between the input and output variables. This will make it a challenge to fit the model, and will also make it a challenge for a fit model to make predictions.<\/p>\n<p>Given these two main sources of error, their point prediction from a predictive model is insufficient for describing the true uncertainty of the prediction.<\/p>\n<h2>What Is a Prediction Interval?<\/h2>\n<p>A prediction interval is a quantification of the uncertainty on a prediction.<\/p>\n<p>It provides a probabilistic upper and lower bounds on the estimate of an outcome variable.<\/p>\n<blockquote>\n<p>A prediction interval for a single future observation is an interval that will, with a specified degree of confidence, contain a future randomly selected observation from a distribution.<\/p>\n<\/blockquote>\n<p>\u2014 Page 27, <a href=\"http:\/\/amzn.to\/2G8w3IL\">Statistical Intervals: A Guide for Practitioners and Researchers<\/a>, 2017.<\/p>\n<p>Prediction intervals are most commonly used when making predictions or forecasts with a regression model, where a quantity is being predicted.<\/p>\n<p>An example of the presentation of a prediction interval is as follows:<\/p>\n<p style=\"text-align: center;\"><em>Given a prediction of \u2018y\u2019 given \u2018x\u2019, there is a 95% likelihood that the range \u2018a\u2019 to \u2018b\u2019 covers the true outcome.<\/em><\/p>\n<p>The prediction interval surrounds the prediction made by the model and hopefully covers the range of the true outcome.<\/p>\n<p>The diagram below helps to visually understand the relationship between the prediction, prediction interval, and the actual outcome.<\/p>\n<div id=\"attachment_5327\" style=\"width: 620px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-5327\" src=\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2018\/04\/Relationship-between-prediction-actual-value-and-prediction-interval.png\" alt=\"Relationship between prediction, actual value and prediction interval\" width=\"610\" height=\"566\" srcset=\"http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2018\/04\/Relationship-between-prediction-actual-value-and-prediction-interval.png 610w, http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2018\/04\/Relationship-between-prediction-actual-value-and-prediction-interval-300x278.png 300w\" sizes=\"(max-width: 610px) 100vw, 610px\"><\/p>\n<p class=\"wp-caption-text\">Relationship between prediction, actual value and prediction interval.<br \/>Taken from \u201cMachine learning approaches for estimation of prediction interval for the model output\u201d, 2006.<\/p>\n<\/div>\n<p>A prediction interval is different from a confidence interval.<\/p>\n<p>A confidence interval quantifies the uncertainty on an estimated population variable, such as the mean or standard deviation. Whereas a prediction interval quantifies the uncertainty on a single observation estimated from the population.<\/p>\n<p>In predictive modeling, a confidence interval can be used to quantify the uncertainty of the estimated skill of a model, whereas a prediction interval can be used to quantify the uncertainty of a single forecast.<\/p>\n<p>A prediction interval is often larger than the confidence interval as it must take the confidence interval and the variance in the output variable being predicted into account.<\/p>\n<blockquote>\n<p>Prediction intervals will always be wider than confidence intervals because they account for the uncertainty associated with e [error], the irreducible error.<\/p>\n<\/blockquote>\n<p>\u2014 Page 103, <a href=\"http:\/\/amzn.to\/2p1CKoB\">An Introduction to Statistical Learning: with Applications in R<\/a>, 2013.<\/p>\n<h2>How to Calculate a Prediction Interval<\/h2>\n<p>A prediction interval is calculated as some combination of the estimated variance of the model and the variance of the outcome variable.<\/p>\n<p>Prediction intervals are easy to describe, but difficult to calculate in practice.<\/p>\n<p>In simple cases like linear regression, we can estimate the confidence interval directly.<\/p>\n<p>In the cases of nonlinear regression algorithms, such as artificial neural networks, it is a lot more challenging and requires the choice and implementation of specialized techniques. General techniques such as the bootstrap resampling method can be used, but are computationally expensive to calculate.<\/p>\n<p>The paper \u201c<a href=\"http:\/\/alumnus.caltech.edu\/~amir\/pred-intv-2.pdf\">A Comprehensive Review of Neural Network-based Prediction Intervals and New Advances<\/a>\u201d provides a reasonably recent study of prediction intervals for nonlinear models in the context of neural networks. The following list summarizes some methods that can be used for prediction uncertainty for nonlinear machine learning models:<\/p>\n<ul>\n<li>The Delta Method, from the field of nonlinear regression.<\/li>\n<li>The Bayesian Method, from Bayesian modeling and statistics.<\/li>\n<li>The Mean-Variance Estimation Method, using estimated statistics.<\/li>\n<li>The Bootstrap Method, using data resampling and developing an ensemble of models.<\/li>\n<\/ul>\n<p>We can make the calculation of a prediction interval concrete with a worked example in the next section.<\/p>\n<h2>Prediction Interval for Linear Regression<\/h2>\n<p>A linear regression is a model that describes the linear combination of inputs to calculate the output variables.<\/p>\n<p>For example, an estimated linear regression model may be written as:<\/p>\n<pre class=\"crayon-plain-tag\">yhat = b0 + b1 . x<\/pre>\n<p>Where <em>yhat<\/em>\u00a0is the prediction, <em>b0<\/em> and <em>b1<\/em> are coefficients of the model estimated from training data and <em>x<\/em>\u00a0is the input variable.<\/p>\n<p>We do not know the true values of the coefficients <em>b0<\/em> and <em>b1<\/em>. We also do not know the true population parameters such as mean and standard deviation for <em>x<\/em> or <em>y<\/em>. All of these elements must be estimated, which introduces uncertainty into the use of the model in order to make predictions.<\/p>\n<p>We can make some assumptions, such as the distributions of <em>x<\/em> and <em>y<\/em> and the prediction errors made by the model, called residuals, are Gaussian.<\/p>\n<p>The prediction interval around <em>yhat<\/em> can be calculated as follows:<\/p>\n<pre class=\"crayon-plain-tag\">yhat +\/- z * sigma<\/pre>\n<p>Where <em>yhat<\/em> is the predicted value, <em>z<\/em> is the critical value from the Gaussian distribution (e.g. 1.96 for a 95% interval) and <em>sigma<\/em>\u00a0is the standard deviation of the predicted distribution.<\/p>\n<p>We do not known in practice. We can calculate an unbiased estimate of the of the predicted standard deviation as follows (taken from <em>Machine learning approaches for estimation of prediction interval for the model output<\/em>):<\/p>\n<pre class=\"crayon-plain-tag\">stdev = sqrt(1 \/ (N - 2) * e(i)^2 for i to N)<\/pre>\n<p>Where <em>stdev<\/em> is an unbiased estimate of the standard deviation for the predicted distribution, <em>n<\/em> are the total predictions made, and <em>e(i)<\/em> is the difference between the ith prediction and actual value.<\/p>\n<h2>Worked Example<\/h2>\n<p>Let\u2019s make the case of linear regression prediction intervals concrete with a worked example.<\/p>\n<p>First, let\u2019s define a simple two-variable dataset where the output variable (<em>y<\/em>) depends on the input variable (<em>x<\/em>) with some Gaussian noise.<\/p>\n<p>The example below defines the dataset we will use for this example.<\/p>\n<pre class=\"crayon-plain-tag\"># generate related variables\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom numpy.random import randn\r\nfrom numpy.random import seed\r\nfrom matplotlib import pyplot\r\n# seed random number generator\r\nseed(1)\r\n# prepare data\r\nx = 20 * randn(1000) + 100\r\ny = x + (10 * randn(1000) + 50)\r\n# summarize\r\nprint('x: mean=%.3f stdv=%.3f' % (mean(x), std(x)))\r\nprint('y: mean=%.3f stdv=%.3f' % (mean(y), std(y)))\r\n# plot\r\npyplot.scatter(x, y)\r\npyplot.show()<\/pre>\n<p>Running the example first prints the mean and standard deviations of the two variables.<\/p>\n<pre class=\"crayon-plain-tag\">x: mean=100.776 stdv=19.620\r\ny: mean=151.050 stdv=22.358<\/pre>\n<p>A plot of the dataset is then created.<\/p>\n<p>We can see the clear linear relationship between the variables with the spread of the points highlighting the noise or random error in the relationship.<\/p>\n<div id=\"attachment_5328\" style=\"width: 1034px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"size-large wp-image-5328\" src=\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2018\/04\/Scatter-Plot-of-Related-Variables-1024x768.png\" alt=\"Scatter Plot of Related Variables\" width=\"1024\" height=\"768\" srcset=\"http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2018\/04\/Scatter-Plot-of-Related-Variables-1024x768.png 1024w, http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2018\/04\/Scatter-Plot-of-Related-Variables-300x225.png 300w, http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2018\/04\/Scatter-Plot-of-Related-Variables-768x576.png 768w, http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2018\/04\/Scatter-Plot-of-Related-Variables.png 1280w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/p>\n<p class=\"wp-caption-text\">Scatter Plot of Related Variables<\/p>\n<\/div>\n<p>Next, we can develop a simple linear regression that given the input variable <em>x<\/em>, will predict the <em>y<\/em> variable. We can use the <a href=\"https:\/\/docs.scipy.org\/doc\/scipy\/reference\/generated\/scipy.stats.linregress.html\">linregress() SciPy function<\/a> to fit the model and return the <em>b0<\/em> and <em>b1<\/em> coefficients for the model.<\/p>\n<pre class=\"crayon-plain-tag\"># fit linear regression model\r\nb1, b0, r_value, p_value, std_err = linregress(x, y)<\/pre>\n<p>We can use the coefficients to calculate the predicted <em>y<\/em> values, called <em>yhat<\/em>, for each of the input variables. The resulting points will form a line that represents the learned relationship.<\/p>\n<pre class=\"crayon-plain-tag\"># make prediction\r\nyhat = b0 + b1 * x<\/pre>\n<p>The complete example is listed below.<\/p>\n<pre class=\"crayon-plain-tag\"># simple linear regression model\r\nfrom numpy.random import randn\r\nfrom numpy.random import seed\r\nfrom scipy.stats import linregress\r\nfrom matplotlib import pyplot\r\n# seed random number generator\r\nseed(1)\r\n# prepare data\r\nx = 20 * randn(1000) + 100\r\ny = x + (10 * randn(1000) + 50)\r\n# fit linear regression model\r\nb1, b0, r_value, p_value, std_err = linregress(x, y)\r\nprint('b0=%.3f, b1=%.3f' % (b1, b0))\r\n# make prediction\r\nyhat = b0 + b1 * x\r\n# plot data and predictions\r\npyplot.scatter(x, y)\r\npyplot.plot(x, yhat, color='r')\r\npyplot.show()<\/pre>\n<p>Running the example fits the model and prints the coefficients.<\/p>\n<pre class=\"crayon-plain-tag\">b0=1.011, b1=49.117<\/pre>\n<p>The coefficients are then used with the inputs from the dataset to make a prediction. The resulting inputs and predicted <em>y<\/em>-values are plotted as a line on top of the scatter plot for the dataset.<\/p>\n<p>We can clearly see that the model has learned the underlying relationship in the dataset.<\/p>\n<div id=\"attachment_5329\" style=\"width: 1034px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"size-large wp-image-5329\" src=\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2018\/04\/Scatter-Plot-of-Dataset-with-Line-for-Simple-Linear-Regression-Model-1024x768.png\" alt=\"Scatter Plot of Dataset with Line for Simple Linear Regression Model\" width=\"1024\" height=\"768\" srcset=\"http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2018\/04\/Scatter-Plot-of-Dataset-with-Line-for-Simple-Linear-Regression-Model-1024x768.png 1024w, http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2018\/04\/Scatter-Plot-of-Dataset-with-Line-for-Simple-Linear-Regression-Model-300x225.png 300w, http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2018\/04\/Scatter-Plot-of-Dataset-with-Line-for-Simple-Linear-Regression-Model-768x576.png 768w, http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2018\/04\/Scatter-Plot-of-Dataset-with-Line-for-Simple-Linear-Regression-Model.png 1280w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/p>\n<p class=\"wp-caption-text\">Scatter Plot of Dataset with Line for Simple Linear Regression Model<\/p>\n<\/div>\n<p>We are now ready to make a prediction with our simple linear regression model and add a prediction interval.<\/p>\n<p>We will fit the model as before. This time we will take one sample from the dataset to demonstrate the prediction interval. We will use the input to make a prediction, calculate the prediction interval for the prediction, and compare the prediction and interval to the known expected value.<\/p>\n<p>First, let\u2019s define the input, prediction, and expected values.<\/p>\n<pre class=\"crayon-plain-tag\">x_in = x[0]\r\ny_out = y[0]\r\nyhat_out = yhat[0]<\/pre>\n<p>Next, we can estimate the standard deviation in the prediction direction.<\/p>\n<pre class=\"crayon-plain-tag\">SE = sqrt(1 \/ (N - 2) * e(i)^2 for i to N)<\/pre>\n<p>We can calculate this directly using the NumPy arrays as follows:<\/p>\n<pre class=\"crayon-plain-tag\"># estimate stdev of yhat\r\nsum_errs = arraysum((y - yhat)**2)\r\nstdev = sqrt(1\/(len(y)-2) * sum_errs)<\/pre>\n<p>Next, we can calculate the prediction interval for our chosen input:<\/p>\n<pre class=\"crayon-plain-tag\">interval = z . stdev<\/pre>\n<p>We will use the significance level of 95%, which is the Gaussian critical value of 1.69.<\/p>\n<p>Once the interval is calculated, we can summarize the bounds on the prediction to the user.<\/p>\n<pre class=\"crayon-plain-tag\"># calculate prediction interval\r\ninterval = 1.96 * stdev\r\nlower, upper = y_out - interval, y_out + interval<\/pre>\n<p>We can tie all of this together. The complete example is listed below.<\/p>\n<pre class=\"crayon-plain-tag\"># linear regression prediction with prediction interval\r\nfrom numpy.random import randn\r\nfrom numpy.random import seed\r\nfrom numpy import power\r\nfrom numpy import sqrt\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom numpy import sum as arraysum\r\nfrom scipy.stats import linregress\r\nfrom matplotlib import pyplot\r\n# seed random number generator\r\nseed(1)\r\n# prepare data\r\nx = 20 * randn(1000) + 100\r\ny = x + (10 * randn(1000) + 50)\r\n# fit linear regression model\r\nb1, b0, r_value, p_value, std_err = linregress(x, y)\r\n# make predictions\r\nyhat = b0 + b1 * x\r\n# define new input, expected value and prediction\r\nx_in = x[0]\r\ny_out = y[0]\r\nyhat_out = yhat[0]\r\n# estimate stdev of yhat\r\nsum_errs = arraysum((y - yhat)**2)\r\nstdev = sqrt(1\/(len(y)-2) * sum_errs)\r\n# calculate prediction interval\r\ninterval = 1.96 * stdev\r\nprint('Prediction Interval: %.3f' % interval)\r\nlower, upper = y_out - interval, y_out + interval\r\nprint('95%% likelihood that the true value is between %.3f and %.3f' % (lower, upper))\r\nprint('True value: %.3f' % yhat_out)\r\n# plot dataset and prediction with interval\r\npyplot.scatter(x, y)\r\npyplot.plot(x, yhat, color='red')\r\npyplot.errorbar(x_in, yhat_out, yerr=interval, color='black', fmt='o')\r\npyplot.show()<\/pre>\n<p>Running the example estimates the <em>yhat<\/em> standard deviation and then calculates the confidence interval.<\/p>\n<p>Once calculated, the prediction interval is presented to the user for the given input variable. Because we contrived this example, we know the true outcome, which we also display. We can see that in this case, the 95% prediction interval does cover the true expected value.<\/p>\n<pre class=\"crayon-plain-tag\">linear regression model.},style=None]\r\nPrediction Interval: 20.204\r\n95% likelihood that the true value is between 160.750 and 201.159\r\nTrue value: 183.124<\/pre>\n<p>A plot is also created showing the raw dataset as a scatter plot, the predictions for the dataset as a red line, and the prediction and prediction interval as a black dot and line respectively.<\/p>\n<div id=\"attachment_5330\" style=\"width: 1034px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"size-large wp-image-5330\" src=\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2018\/04\/Scatter-Plot-of-Dataset-With-Linear-Model-and-Prediction-Interval-1024x768.png\" alt=\"Scatter Plot of Dataset With Linear Model and Prediction Interval\" width=\"1024\" height=\"768\" srcset=\"http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2018\/04\/Scatter-Plot-of-Dataset-With-Linear-Model-and-Prediction-Interval-1024x768.png 1024w, http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2018\/04\/Scatter-Plot-of-Dataset-With-Linear-Model-and-Prediction-Interval-300x225.png 300w, http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2018\/04\/Scatter-Plot-of-Dataset-With-Linear-Model-and-Prediction-Interval-768x576.png 768w, http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2018\/04\/Scatter-Plot-of-Dataset-With-Linear-Model-and-Prediction-Interval.png 1280w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/p>\n<p class=\"wp-caption-text\">Scatter Plot of Dataset With Linear Model and Prediction Interval<\/p>\n<\/div>\n<h2>Extensions<\/h2>\n<p>This section lists some ideas for extending the tutorial that you may wish to explore.<\/p>\n<ul>\n<li>Summarize the difference between tolerance, confidence, and prediction intervals.<\/li>\n<li>Develop a linear regression model for a standard machine learning dataset and calculate prediction intervals for a small test set.<\/li>\n<li>Describe in detail how one nonlinear prediction interval method works.<\/li>\n<\/ul>\n<p>If you explore any of these extensions, I\u2019d love to know.<\/p>\n<h2>Further Reading<\/h2>\n<p>This section provides more resources on the topic if you are looking to go deeper.<\/p>\n<h3>Posts<\/h3>\n<ul>\n<li><a href=\"https:\/\/machinelearningmastery.com\/report-classifier-performance-confidence-intervals\/\">How to Report Classifier Performance with Confidence Intervals<\/a><\/li>\n<li><a href=\"https:\/\/machinelearningmastery.com\/calculate-bootstrap-confidence-intervals-machine-learning-results-python\/\">How to Calculate Bootstrap Confidence Intervals For Machine Learning Results in Python<\/a><\/li>\n<li><a href=\"https:\/\/machinelearningmastery.com\/time-series-forecast-uncertainty-using-confidence-intervals-python\/\">Understand Time Series Forecast Uncertainty Using Confidence Intervals with Python<\/a><\/li>\n<li><a href=\"https:\/\/machinelearningmastery.com\/estimate-number-experiment-repeats-stochastic-machine-learning-algorithms\/\">Estimate the Number of Experiment Repeats for Stochastic Machine Learning Algorithms<\/a><\/li>\n<\/ul>\n<h3>Books<\/h3>\n<ul>\n<li><a href=\"http:\/\/amzn.to\/2oQW6No\">Understanding The New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis<\/a>, 2017.<\/li>\n<li><a href=\"http:\/\/amzn.to\/2G8w3IL\">Statistical Intervals: A Guide for Practitioners and Researchers<\/a>, 2017.<\/li>\n<li><a href=\"http:\/\/amzn.to\/2p1CKoB\">An Introduction to Statistical Learning: with Applications in R<\/a>, 2013.<\/li>\n<li><a href=\"http:\/\/amzn.to\/2FJlj5H\">Introduction to the New Statistics: Estimation, Open Science, and Beyond<\/a>, 2016.<\/li>\n<li><a href=\"http:\/\/amzn.to\/2tFQ1Io\">Forecasting: principles and practice<\/a>, 2013.<\/li>\n<\/ul>\n<h3>Papers<\/h3>\n<ul>\n<li><a href=\"https:\/\/pdfs.semanticscholar.org\/31c8\/17950a35c282c9da7ba7c7f927a6ff28a5af.pdf\">A comparison of some error estimates for neural network models<\/a>, 1995.<\/li>\n<li><a href=\"https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0893608006000153\">Machine learning approaches for estimation of prediction interval for the model output<\/a>, 2006.<\/li>\n<li><a href=\"http:\/\/alumnus.caltech.edu\/~amir\/pred-intv-2.pdf\">A Comprehensive Review of Neural Network-based Prediction Intervals and New Advances<\/a>, 2010.<\/li>\n<\/ul>\n<h3>API<\/h3>\n<ul>\n<li><a href=\"https:\/\/docs.scipy.org\/doc\/scipy\/reference\/generated\/scipy.stats.linregress.html\">scipy.stats.linregress() API<\/a><\/li>\n<li><a href=\"https:\/\/matplotlib.org\/api\/_as_gen\/matplotlib.pyplot.scatter.html\">matplotlib.pyplot.scatter() API<\/a><\/li>\n<li><a href=\"https:\/\/matplotlib.org\/api\/_as_gen\/matplotlib.pyplot.errorbar.html\">matplotlib.pyplot.errorbar() API<\/a><\/li>\n<\/ul>\n<h3>Articles<\/h3>\n<ul>\n<li><a href=\"https:\/\/en.wikipedia.org\/wiki\/Prediction_interval\">Prediction interval on Wikipedia<\/a><\/li>\n<li><a href=\"https:\/\/stats.stackexchange.com\/questions\/226565\/bootstrap-prediction-interval\">Bootstrap prediction interval on Cross Validated<\/a><\/li>\n<\/ul>\n<h2>Summary<\/h2>\n<p>In this tutorial, you discovered the prediction interval and how to calculate it for a simple linear regression model.<\/p>\n<p>Specifically, you learned:<\/p>\n<ul>\n<li>That a prediction interval quantifies the uncertainty of a single point prediction.<\/li>\n<li>That prediction intervals can be estimated analytically for simple models but are more challenging for nonlinear machine learning models.<\/li>\n<li>How to calculate the prediction interval for a simple linear regression model.<\/li>\n<\/ul>\n<p>Do you have any questions?<br \/>\nAsk your questions in the comments below and I will do my best to answer.<\/p>\n<p>The post <a rel=\"nofollow\" href=\"https:\/\/machinelearningmastery.com\/prediction-intervals-for-machine-learning\/\">Prediction Intervals for Machine Learning<\/a> appeared first on <a rel=\"nofollow\" href=\"https:\/\/machinelearningmastery.com\/\">Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n<p><a href=\"https:\/\/machinelearningmastery.com\/prediction-intervals-for-machine-learning\/\">Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Author: Jason Brownlee A prediction from a machine learning perspective is a single point that hides the uncertainty of that prediction. Prediction intervals provide a [&hellip;] <span class=\"read-more-link\"><a class=\"read-more\" href=\"https:\/\/www.aiproblog.com\/index.php\/2018\/05\/29\/prediction-intervals-for-machine-learning\/\">Read More<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":560,"comment_status":"registered_only","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[24],"tags":[],"_links":{"self":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/559"}],"collection":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/comments?post=559"}],"version-history":[{"count":0,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/559\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media\/560"}],"wp:attachment":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media?parent=559"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/categories?post=559"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/tags?post=559"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}