Author: Stephanie Glen<\/p>\n
Evaluating a model is just as important as creating the model in the first place. Even if you use the most statistically sound tools to create your model, the end result may not be what you expected. Which metric you use to test your model depends on the type of data you\u2019re working with and your comfort level with statistics.<\/p>\n
Model evaluation techniques answer three main questions:<\/strong><\/p>\n The following summary of model evaluation techniques is by no means exhaustive; it\u2019s intended to be a starting point if you\u2019re unfamiliar with the available techniques. In part 1, I discuss some of the common Statistical Tools and Tests. Part 2 will cover tools for Clustering and Classification.<\/p>\n 1. Confidence Interval<\/strong><\/p>\n A confidence interval<\/a> is a measure of how reliable a statistical estimate<\/strong> is. \u00a0For example, you might have calculated a mean\u00a0<\/a>or standard deviation<\/a> for your set. But how reliable is that estimate? A confidence interval will give you an easy-to-understand boundary. For example, if your calculated mean is 10 feet, a confidence interval might tell you that 99% of results will fall between 9 feet and 11 feet (i.e. 1 foot either side of the mean).<\/p>\n 2. Root Mean Square Error (RMSE)<\/strong><\/p>\n RMSE is a measure of how spread out residuals<\/a>\u00a0are. In other words, it tells you how concentrated the data is around the\u00a0line of best fit<\/a>.\u00a0. It\u2019s one of the most popular metrics for evaluating continuous data, and is widely used in Excel. However, it\u2019s a lot trickier to understand than simple statistics like confidence intervals because of the complex calculations involved. It also penalizes higher differences, meaning that it\u2019s sensitive to outliers<\/a>. Even if you don\u2019t understand the calculations behind RMSE, Excel will still spit out an answer, leaving you to puzzle over the significance of the result. To further the complications, RMSE (and other similar metrics like bias<\/a>\u00a0and correlation coefficients)<\/a> can really only be fully understood if you are very familiar with the underlying data\/model.<\/p>\n \u00a0L^1 version of RSME<\/strong>.<\/p>\n RSME is sensitive to outliers, one reason why it\u2019s fallen out of favor with many data scientists. An alternative is a more modern version, like the L^1 version\u00a0described here<\/a>. \u00a0<\/p>\n \u00a0<\/p>\n While basic statistical tools (like those listed above) are fairly easy to understand for the non-statistician, delving into the arena of statistical tests requires you to have a lot more in-depth knowledge not only of statistics, but of your data and model. The major advantages to running a statistical test is that it gives you great confidence in your results. For many professional arenas and publications, statistical tests are an absolute must. The downside is that you really have to know your data inside and out in order to interpret the results from these tests; \u00a0otherwise, it\u2019s easy to misinterpret them.<\/p>\n 1. Kolmogorov-Smirnov Test<\/u><\/strong>.<\/u><\/p>\n The Kolmogorov-Smirnov Goodness of Fit Test (K-S test)<\/a> is a distribution free test<\/a>\u00a0that compares your data with a known distribution (usually a normal distribution<\/a>) and lets you know if they have the same distribution. The fact that you don\u2019t have to know the underlying distribution is a great advantage, but the test has several drawbacks, including the fact that you have to specify the\u00a0location<\/a>,\u00a0scale<\/a>, and\u00a0shape<\/a>\u00a0parameters; These cannot be estimated from the data, as it will invalidate the test. Another big disadvantage is that it can\u2019t be usually be used for discrete data<\/a>\u00a0without some particularly cumbersome calculations or an add-on for software.<\/p>\n \u00a02. Lilliefors Test<\/strong><\/p>\n Lilliefors test<\/a> is\u00a0 a corrected version of the\u00a0<\/strong>K-S test for normality, generally gives a more accurate approximation of the test statistic\u2019s distribution. This is especially true if you don\u2019t know the population mean and standard deviation (which is usually the case). Many statistical packages (like SPSS) combine the two tests as a \u201cLilliefors corrected\u201d K-S test.<\/p>\n 3.\u00a0<\/i><\/strong>Chi Square<\/u>.<\/u><\/p>\n The Chi-Square test is similar to Kolmogorov-Smirnov, but it is a parametric test<\/a>. This means that you have to know the underlying distribution in order to work with it. While the K-S test performs poorly when you estimate population parameters, chi-square can be successfully run with estimations. That said, a downside is that it doesn\u2019t work well with small sample sizes.<\/p>\n In general, K-S is usually preferred for it\u2019s higher power<\/a>. However, when you don\u2019t know a certain population parameter (i.e. the one you’re trying to estimate, like the mean), chi-square may be the better choice.<\/p>\n<\/p>\n Comparison of the Goodness-of-Fit Tests: the Pearson Chi-square and Kolmogorov-Smirnov Tests<\/a><\/p>\n 11 Important Model Evaluation Techniques Everyone Should Know<\/a><\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"\n
BASIC STATISTICAL TOOLS<\/h2>\n
STATISTICAL TESTS<\/h2>\n
References<\/h2>\n