The use of training, validation and test datasets is common but not easily understood.\u00a0<\/p>\n
\u00a0<\/p>\n
In this post, I attempt to clarify this concept. The post is part of my forthcoming book on learning Artificial Intelligence, Machine Learning and Deep Learning based on high school maths<\/strong>.\u00a0 If you want to know more about the book, please follow me on Linkedin Ajit Jaokar<\/a><\/p>\n \u00a0<\/p>\n Jason Brownlee provides a good explanation on the three-way data splits (training, test and validation)<\/a><\/p>\n \u2013 Training set: A set of examples used for learning, that is to fit the parameters of the classifier.<\/em><\/p>\n \u2013 Validation set: A set of examples used to tune the parameters of a classifier, for example to choose the number of hidden units in a neural network.<\/em><\/p>\n \u2013 Test set: A set of examples used only to assess the performance of a fully-specified classifier.<\/em><\/p>\n And then comes up with an important statement: Reference to a \u201cvalidation dataset\u201d disappears if the practitioner is choosing to tune model hyperparameters using k-fold cross-validation with the training dataset.<\/em><\/strong><\/p>\n So, here, I try and explain these ideas in more detail from the source by Ricardo Gutierrez-Osuna Wright State University<\/a><\/p>\n \u00a0<\/p>\n Validation techniques are motivated by two fundamental problems in pattern recognition: model selection and performance estimation<\/strong><\/p>\n Model selection<\/strong>: involves selecting optimal parameters or a model. Pattern recognition techniques have one or more free parameters \u2013 for example – the number of neighbours in a kNN classification and the network size, learning parameters and weights in MLPs. The selection of these hyperparameters determines the efficiency of the solution. Hyperparameters are set by the user. In contrast, the parameters of a model are learned from the data. \u00a0<\/p>\n \u00a0<\/p>\n Performance estimation:<\/strong> \u00a0\u00a0Once we have chosen a model, we need to estimate its performance. \u00a0If we had access to an unlimited set of samples (or the whole population) \u2013 it is easy to estimate the performance. However, in practise, we have access to a smaller sample of the population. If we use the entire dataset to train the model, the model is likely to overfit. Overfitting is essentially \u2018learning the noise\u2019 from the training data. Since our goal is to find the best model that can give optimal results on unseen data, overfitting is not a good option. We can address this problem by evaluating the error function using data which is independent of that used for training. \u00a0<\/p>\n \u00a0<\/p>\n The first approach is to split the model into training and test dataset. This is the holdout method<\/strong> where you use the training dataset to train the classifier and the test dataset to estimate the error of the trained classifier.\u00a0 The holdout method has limitations: for example, it is not suitable for sparse datasets The limitations of the holdout can be overcome with a family of resampling methods such as Cross Validation<\/strong>.<\/p>\n \u00a0<\/p>\n Finally, the\u00a0test dataset<\/strong>\u00a0is a dataset used to provide an unbiased evaluation of a\u00a0final<\/em>\u00a0model fit on the training dataset. The test dataset is used to obtain the performance characteristics such as accuracy, sensitivity, specificity, F-measure, and so on.<\/p>\n \u00a0<\/p>\n The overall steps are:<\/p>\n \u00a0<\/p>\n source for image and steps<\/strong> –\u00a0source by Ricardo Gutierrez-Osuna Wright State University<\/a><\/p>\n I hope you found this useful. The post is part of my forthcoming book on learning Artificial Intelligence, Machine Learning and Deep Learning based on high school maths<\/strong>.\u00a0 If you want to know more about the book, please follow me on Linkedin Ajit Jaokar<\/a><\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"Background<\/h2>\n
Understanding model validation<\/h2>\n
Putting it all together<\/h2>\n
\n
\n
<\/a>\u00a0<\/p>\n