Summary:<\/em><\/strong>\u00a0 Exceptions sometimes make the best rules.\u00a0 Here\u2019s an example of well accepted variable reduction techniques resulting in an inferior model and a case for dramatically expanding the number of variables we start with.<\/em><\/p>\n \u00a0<\/p>\n Certainly this is a rule I embrace when building models so I was surprised and then delighted to find a well conducted study by Lexis\/Nexis that lays out a case where this clearly isn\u2019t true.<\/p>\n \u00a0<\/p>\n A Little Background<\/strong><\/span><\/p>\n In highly regulated industries like insurance and lending the variables that are allowed for use are highly regulated as are the modeling techniques.\u00a0 Techniques are generally limited to those that are highly explainable, mostly GLM and simple decision trees.\u00a0 Data can\u2019t include anything that is overtly discriminatory under the law so, for example, race, sex, and age can\u2019t be used, or at least not directly.\u00a0 All of this works against model accuracy.<\/p>\n Traditionally what agencies could use to build risk models has been defined as \u2018traditional data\u2019, that which the consumer has submitted with their application and the data that can be added from the major credit rating agencies.\u00a0 In this last case Experian and the others offer some 250 different variables and except for those that are specifically excluded by law, this seems like a pretty good sized inventory of predictive features.<\/p>\n But in the US and especially abroad the market contains many \u2018thin-file\u2019 or \u2018no-file\u2019 consumers who would like to borrow but for which traditional data sources simply don\u2019t exist.\u00a0 Millennials feature in this group because their cohort is young and doesn\u2019t yet have much borrowing or credit history.\u00a0 But also in this group are the folks judged to be marginal credit risks, some of whom could be good customers if only we knew how to judge the risk.<\/p>\n \u00a0<\/p>\n Enter the World of Alternative Data<\/strong><\/span><\/p>\n \u2018Alternative data\u2019 is considered to be any data not directly related to the consumer\u2019s credit behavior, basically anything other than the application data and consumer credit bureau data.\u00a0 A variety of agencies are prepared to provide it and it can include:<\/p>\n As it turns out lenders have been embracing alternative data for the last several years and see real improvements in their credit models, particularly at the low end of the scores.\u00a0 Even the CFPB has provisionally endorsed this to bring credit to the underserved.<\/p>\n \u00a0<\/p>\n From a data science perspective, in this example we started out with on the order of 250 candidate features from \u2018traditional data\u2019, and now, using \u2018alternative data\u2019 we can add an additional 1,050 features.\u00a0 What\u2019s the first thing you do when you have 1,300 candidate variables?\u00a0 You go through the steps necessary to identify only the most predictive variables and discard the rest.<\/p>\n \u00a0<\/p>\n Here\u2019s Where It Gets Interesting<\/strong><\/span><\/p>\n Lexis\/Nexis, the provider of the alternative data, set out to demonstrate that a credit model built on all 1,300 features was superior to one built on only 250 traditional features.\u00a0 The data was drawn from a full-file auto lending portfolio of just under 11 million instances.\u00a0 You and I might have concluded that even 250 was too many but in order to keep the test rigorous they introduced these constraints.\u00a0<\/p>\n \u00a0<\/p>\n What Happened<\/strong><\/span><\/p>\n As you might expect, the first and most important variable was the same for both models but began to diverge at the second variable.\u00a0 The second variable in the 1,300 model was actually 296th<\/sup> based on the earlier predictive power analysis.\u00a0<\/p>\n When the model was completed the alternative data made up 25% of the model\u2019s accuracy although none would have been included based on the top 250 predictive variables.<\/p>\n The KS (Kolmogorov-Smirnov) statistic was 4.3% better for the 1,300 model compared to the 250 model.<\/p>\n \u00a0<\/p>\n The Business Importance<\/strong><\/span><\/p>\n The distribution of scores and charge offs for each models was very similar but in the bottom 5% of scores things changed.\u00a0 There was a 6.4% increase in the number of predicted charge offs in this bottom group.\u00a0<\/p>\n Since the distributions are the essentially the same this can be seen as higher scores that might have been rated credit worthy migrating into the lowest categories of credit worthiness allowing better decisions about denial or pricing based on risk.\u00a0 Conversely it appears that some lowest rated borrowers were given a boost with the additional data.<\/p>\n That also translates to a competitive advantage for those using the alternative data compared to those who don\u2019t.\u00a0 You can see the original study here<\/u><\/em><\/a>.<\/p>\n \u00a0<\/p>\n There are Four Lesson for Data Scientists Here<\/span><\/strong><\/p>\n \u00a0 On the other hand just this week a major newspaper in England published as expose of comparative auto insurance rates where it discovered that individuals applying with a Hotmail account were paying as much as 7% more than those with Gmail accounts.\u00a0 Apparently British insurers had found a legitimate correlation between risk and this alternative data.\u00a0 It did not sit well with the public and the companies are now on the defensive.<\/p>\n \u00a0<\/p>\n \u00a0<\/p>\n About the author:\u00a0 Bill Vorhies is Editorial Director for Data Science Central and has practiced as a data scientist since 2001.\u00a0 He can be reached at:<\/p>\n Bill@DataScienceCentral.com<\/a><\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":" Author: William Vorhies Summary:\u00a0 Exceptions sometimes make the best rules.\u00a0 Here\u2019s an example of well accepted variable reduction techniques resulting in an inferior model and […] Read More<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":473,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[26],"tags":[],"_links":{"self":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/827"}],"collection":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/comments?post=827"}],"version-history":[{"count":0,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/827\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media\/462"}],"wp:attachment":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media?parent=827"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/categories?post=827"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/tags?post=827"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![](\"http:\/\/api.ning.com\/files\/GOtZVJLj-wqtnowdqSWYTxbiW5p7RH0eJtYjWiuVHQSNXOG042ek9lS8az8bL747rQegHxP8CGuHgeVC1nE1AnVbe0mC8Esy\/noexception.png\")
<\/a>One of the things that keeps us data scientists on our toes is that the well-established rules-of-thumb don\u2019t always work.\u00a0 Certainly one of the most well-worn of these rules is the parsimonious model; always seek to create the best model with the fewest variables.\u00a0 And woe to you who violate this rule.\u00a0 Your model will over fit, include false random correlations, or at very least will just be judged to be slow and clunky.<\/p>\n\n
<\/a>From a Data Science Perspective<\/strong><\/span><\/p>\n\n
\n
<\/a><\/p>\n\n