In\u00a0part 1<\/a>, I compared a few model evaluation techniques that fall under the umbrella of ‘general statistical tools and tests’. Here in Part 2 I compare three of the more popular model evaluation techniques for classification and clustering: confusion matrix<\/a>, gain and lift chart, and ROC curve<\/a>. The main difference between the three techniques is that each focuses on a different type of result:<\/p>\n That said, you’ll want to choose a method that gives you the answers you need for the particular field you’re in. For example, while a confusion matrix can be a great tool for comparing models, it isn’t much good for marketing decisions (where the gain and lift chart would be a better choice).<\/p>\n Other less popular (but still valid) tools include the K-S chart<\/a>\u00a0and Gini Coefficient<\/a>.<\/p>\n A confusion matrix, in predictive analytics, shows the rate of false positives, false negatives, true positives and true negatives for a test or predictor. In machine learning, a confusion matrix can be used to show how well a classification model performs on a set of test data. Correctly assigned values appear in their relative diagonal box:<\/p>\n Wrongly assigned observations are labeled as either false positives (box b) or false negatives (box c).<\/p>\n Confusion matrices can give you a good idea about how effective your model is. It can also help you choose between multiple competing models.But sometimes you want to know how a particular model does with more data<\/strong>; For example, does a model perform better with 60% of data, compared to 50%? This is where gain and lift charts come in.<\/p>\n The following gains chart, run on a validation set, shows that with 50% of the data, the model contains 90% of targets, Adding more data adds a negligible increase in the percentage of targets included in the model.<\/p>\n A lift chart shows you how much better your model performs, compared to random selection. The “lift” is the ratio of results with and without the model; Better models have higher lifts.<\/p>\n While the confusion matrix gives proportions between all negatives and positives, Gain and lift charts focus on the true positives.<\/strong> One of their most common uses is in marketing, to decide if a prospective client is worth calling.<\/p>\n Gain and lift charts work with a sample<\/strong> (a fraction of the population). In comparison, a\u00a0confusion matrix uses the whole population to evaluate a model.<\/p>\n A\u00a0Receiver Operating Characteristic (ROC) Curve<\/a> is a way to compare models. It is a plot of the\u00a0<\/span>true positive rate<\/a>\u00a0against the\u00a0<\/span>false positive rate<\/a>.\u00a0 It’s similar to the gain and lift chart, but instead\u00a0of just true positives, this time the focus is on a graphical representation of true positives vs. false positives.<\/span><\/p>\n In layman’s terms, the closer the graph is to the top and left borders, the more accurate the model. If you’e familiar with calculus (specifically, areas under the curve), the ideal model has an area of 1; a random model (with a 50%) chance is shown with the black diagonal on the graph. Also shown on the above example; two models in blue and red. The blue line represents a more accurate model as it is closer to the top and left borders.\u00a0<\/span><\/p>\n 11 Important Model Evaluation Techniques Everyone Should Know<\/a><\/p>\n How to evaluate classification models for business analytics – Part 1<\/span><\/a><\/p>\n How to determine the best model?<\/a><\/span><\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"\n
Confusion Matrix<\/h2>\n
<\/a><\/p>\n
\n
\n
Gain and Lift Charts<\/h2>\n
<\/a><\/p>\n
ROC Curve<\/h2>\n
<\/a><\/span><\/p>\n
References<\/h2>\n