Author: Stephanie Glen<\/p>\n
In my previous posts, I compared model evaluation techniques using Statistical Tools & Tests<\/a> and commonly used Classification and Clustering evaluation techniques<\/a><\/p>\n In this post, I’ll take a look at how you can compare regression models. Comparing regression models is perhaps one of the trickiest tasks to complete in the “comparing models” arena; The reason is that there are literally dozens<\/em> of statistics you can calculate to compare regression models, including:<\/p>\n 1. Error measures in the estimation period<\/strong> (in-sample testing) or validation period<\/strong> (out-of-sample testing):<\/p>\n 2. Tests on Residuals and Goodness-of-Fit:<\/strong><\/p>\n This list isn’t exhaustive–there are many other tools, tests and plots at your disposal. Rather than discuss the statistics in detail, I chose to focus this post on\u00a0comparing a few of the most popular regression model evaluation techniques<\/strong> and discuss when you might want to use them (or when you might not<\/em> want to). The techniques listed below tend to be on the “easier to use and understand” end of the spectrum, so if you’re new to model comparison it’s a good place to start.<\/p>\n The first question you should be asking is: How well do I know my data?<\/em> In order to evaluate regression models, you need to know what results would be reasonable for your particular situation. For example, if you compare changes in mean or variance, one model might give you impossible results, another might be overly complicated for the task at hand. The ideal model isn’t one that’s just “correct”, it also needs to be relatively simple and useful for the decision making process–something that won’t be immediately obvious unless you know your data really well.<\/p>\n Which technique you choose is largely dependent on the software you have at hand (i.e. R, SPSS, or Excel).\u00a0If you’re using Excel, a word of advice: stop<\/strong>. It was never designed for serious statistical work and has significant statistical problems<\/a>. Duke University’s Robert Nau<\/a> puts it best: “It’s a toy (a clumsy one at that), not a tool for serious work.”\u00a0<\/span><\/i> The number of models you’re testing also comes into play. Arguably, which statistic you use (r-squared<\/a>, p-values<\/a> etc.) are mostly personal preference (although the test you use might force that choice upon you). Each of the statistics has it’s pluses and minuses, its advantages and disadvantages. I won’t be bloating this article out with all the comparisons between the statistics, but if you’re interested I’ve linked where possible to articles that explain those in detail.<\/p>\n Nested models<\/a> are models that are subsets of one another; If you can get one model by constraining the parameters of another, then that model is nested. Nested models require different techniques to evaluate models and there isn’t a single, agreed-upon way to test for the “best” model.<\/p>\n Possibly the easiest<\/strong> (it can be used with a very basic understanding of statistics) way to compare nested models is to simply measure how well each model performs reclassification<\/a>. The “better” model will have higher rates of correct reclassification. A chi-square<\/a>\u00a0analysis can be used, although if you run a test for sphericity<\/a>\u00a0you must use a different chi-square value.\u00a0<\/p>\n If you’re comparing nested models (perhaps you want to know if the simplest model is adequate), you can compare them with a t-statistic<\/a><\/strong>. You can only run a test for significance<\/a>\u00a0against a single extra coefficient<\/a>. In other words, you can’t run it if you have more than one additional coefficient from one model to the next. This article<\/a> has instructions in R, as well as a fairly detailed overview on running the\u00a0general regression test<\/em>\u00a0or the extra\u00a0sum of squares test.<\/em> \u00a0<\/p>\n According to Calvin Garbin of the University of Nebraska Lincoln,<\/a> with SPSS<\/strong> you can compare nested models in two different ways using r-squared<\/a>:<\/strong><\/p>\n Gabin’s article has a couple of excellent examples of how to perform the above tasks as well as SPSS procedures for comparing non-nested models using correlations<\/a>.<\/p>\n An ANOVA\u00a0<\/a>F-test can compare two nested models, where one is a subset of the other. It tests a single predictor variable, but can be used to\u00a0test multiple predictors at a time.\u00a0<\/p>\n Multiple models<\/strong> can be compared using forward selection<\/a>, backward elimination, or stepwise selection. Basically, these are all variants of each other\u00a0and involve\u00a0removing predictors with the smallest f-value<\/a> \/ t-value or\u00a0largest associated p-value<\/a>. These techniques can only be used on nested models, but they can all miss optimal models and–if you run all three on the same models–they may not agree with each other.<\/p>\n<\/p>\n Non nested models have fewer options for comparison between models. As the models aren’t nested, neither will your results (e.g. a chi-square statistic<\/a>). In layman’s terms, if your models are nested then you’re comparing apples to apples, which is much easier than comparing apples to oranges.<\/p>\n One of the simplest comparison methods is the Bayesian information criterion.<\/a><\/strong>\u00a0Despite the daunting math behind the calculations, most statistical software will calculate<\/span>\u00a0<\/strong>the BIC for each model.\u00a0<\/span><\/strong>This leaves you to simply interpret the results:\u00a0<\/span><\/strong>The model with the lowest BIC is considered the best. It’s often preferred over other Bayesian methods like Bayes Factors<\/a>, because BIC doesn’t require you to have knowledge about priors<\/a>.\u00a0<\/span><\/p>\n Akaike\u2019s Information Criterion<\/strong><\/a>\u00a0is similar to BIC, except that the BIC tends to favor models with fewer parameters. AIC ranks each model\u00a0from best to worst. A major downside is that it doesn’t say anything about quality<\/em>; It will choose the “best” even if you input a series of poor quality models.\u00a0<\/span><\/p>\n The benefit of the Cox test<\/a><\/strong>\u00a0is it’s relatively simple (in comparison to the BIC or AIC) to understand what the test is doing behind the scenes.\u00a0 Let’s say you were comparing models A and B. If model A contains the the correct regressors, then those regressors fit from model B to model A should yield zero further explanatory value. If there is further explanatory value, then model 1 doesn’t contain the correct regressor set. You run the test twice–the second time from B to A–and compare your findings.\u00a0<\/a>See: Performing the Cox Test in R.<\/a><\/p>\n<\/p>\n Comparing Models<\/a><\/p>\n Comparing Nested Models<\/a><\/p>\n Outline: Significance Testing<\/a><\/p>\n Linear Regression Models<\/a><\/p>\n Non nested model selection criteria<\/a><\/p>\n Cox test for comparing non nested models<\/a><\/p>\n Quiz #3 Research Hypotheses that Involve Comparing Non-Nested Models<\/a><\/p>\n R. Davidson & J. MacKinnon (1981). Several Tests for Model Specification in the Presence of Alternative Hypotheses.\u00a0<\/span>Econometrica<\/em>,\u00a0<\/span>49<\/b>, 781-793.<\/span><\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":" Author: Stephanie Glen In my previous posts, I compared model evaluation techniques using Statistical Tools & Tests and commonly used Classification and Clustering evaluation techniques […] Read More<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":457,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[26],"tags":[],"_links":{"self":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/2396"}],"collection":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/comments?post=2396"}],"version-history":[{"count":0,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/2396\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media\/457"}],"wp:attachment":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media?parent=2396"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/categories?post=2396"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/tags?post=2396"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
\n
Where to Start<\/h2>\n
Nested Models<\/h2>\n
\n
change F-test<\/em>.<\/li>\nNon Nested Models<\/h2>\n
References<\/h2>\n