Multinomial logistic regression<\/strong> is an extension of logistic regression that adds native support for multi-class classification problems.<\/p>\n Logistic regression, by default, is limited to two-class classification problems. Some extensions like one-vs-rest can allow logistic regression to be used for multi-class classification problems, although they require that the classification problem first be transformed into multiple binary classification problems.<\/p>\n Instead, the multinomial logistic regression algorithm is an extension to the logistic regression model that involves changing the loss function to cross-entropy loss and predict probability distribution to a multinomial probability distribution to natively support multi-class classification problems.<\/p>\n In this tutorial, you will discover how to develop multinomial logistic regression models in Python.<\/p>\n After completing this tutorial, you will know:<\/p>\n Let’s get started.<\/p>\n Multinomial Logistic Regression With Python This tutorial is divided into three parts; they are:<\/p>\n Logistic regression is a classification algorithm.<\/p>\n It is intended for datasets that have numerical input variables and a categorical target variable that has two values or classes. Problems of this type are referred to as binary classification problems.<\/p>\n Logistic regression is designed for two-class problems, modeling the target using a binomial probability distribution function<\/a>. The class labels are mapped to 1 for the positive class or outcome and 0 for the negative class or outcome. The fit model predicts the probability that an example belongs to class 1.<\/p>\n By default, logistic regression cannot be used for classification tasks that have more than two class labels, so-called multi-class classification.<\/p>\n Instead, it requires modification to support multi-class classification problems.<\/p>\n One popular approach for adapting logistic regression to multi-class classification problems is to split the multi-class classification problem into multiple binary classification problems and fit a standard logistic regression model on each subproblem. Techniques of this type include one-vs-rest and one-vs-one wrapper models<\/a>.<\/p>\n An alternate approach involves changing the logistic regression model to support the prediction of multiple class labels directly. Specifically, to predict the probability that an input example belongs to each known class label.<\/p>\n The probability distribution that defines multi-class probabilities is called a multinomial probability distribution. A logistic regression model that is adapted to learn and predict a multinomial probability distribution is referred to as Multinomial Logistic Regression. Similarly, we might refer to default or standard logistic regression as Binomial Logistic Regression.<\/p>\n If you are new to binomial and multinomial probability distributions, you may want to read the tutorial:<\/p>\n Changing logistic regression from binomial to multinomial probability requires a change to the loss function used to train the model (e.g. log loss to cross-entropy loss), and a change to the output from a single probability value to one probability for each class label.<\/p>\n Now that we are familiar with multinomial logistic regression, let’s look at how we might develop and evaluate multinomial logistic regression models in Python.<\/p>\n In this section, we will develop and evaluate a multinomial logistic regression model using the scikit-learn Python machine learning library.<\/p>\n First, we will define a synthetic multi-class classification dataset to use as the basis of the investigation. This is a generic dataset that you can easily replace with your own loaded dataset later.<\/p>\n The make_classification() function<\/a> can be used to generate a dataset with a given number of rows, columns, and classes. In this case, we will generate a dataset with 1,000 rows, 10 input variables or columns, and 3 classes.<\/p>\n The example below generates the dataset and summarizes the shape of the arrays and the distribution of examples across the three classes.<\/p>\n Running the example confirms that the dataset has 1,000 rows and 10 columns, as we expected, and that the rows are distributed approximately evenly across the three classes, with about 334 examples in each class.<\/p>\n Logistic regression is supported in the scikit-learn library via the LogisticRegression class<\/a>.<\/p>\n The LogisticRegression<\/em> class can be configured for multinomial logistic regression by setting the “multi_class<\/em>” argument to “multinomial<\/em>” and the “solver<\/em>” argument to a solver that supports multinomial logistic regression, such as “lbfgs<\/em>“.<\/p>\n The multinomial logistic regression model will be fit using cross-entropy loss<\/a> and will predict the integer value for each integer encoded class label.<\/p>\n Now that we are familiar with the multinomial logistic regression API, we can look at how we might evaluate a multinomial logistic regression model on our synthetic multi-class classification dataset.<\/p>\n It is a good practice to evaluate classification models using repeated stratified k-fold cross-validation<\/a>. The stratification ensures that each cross-validation fold has approximately the same distribution of examples in each class as the whole training dataset.<\/p>\n We will use three repeats with 10 folds, which is a good default, and evaluate model performance using classification accuracy given that the classes are balanced.<\/p>\n The complete example of evaluating multinomial logistic regression for multi-class classification is listed below.<\/p>\n Running the example reports the mean classification accuracy across all folds and repeats of the evaluation procedure.<\/p>\n Note<\/strong>: Your results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n In this case, we can see that the multinomial logistic regression model with default penalty achieved a mean classification accuracy of about 68.1 percent on our synthetic classification dataset.<\/p>\n We may decide to use the multinomial logistic regression model as our final model and make predictions on new data.<\/p>\n This can be achieved by first fitting the model on all available data, then calling the predict()<\/em> function to make a prediction for new data.<\/p>\n The example below demonstrates how to make a prediction for new data using the multinomial logistic regression model.<\/p>\n Running the example first fits the model on all available data, then defines a row of data, which is provided to the model in order to make a prediction.<\/p>\n In this case, we can see that the model predicted the class “1” for the single row of data.<\/p>\n A benefit of multinomial logistic regression is that it can predict calibrated probabilities across all known class labels in the dataset.<\/p>\n This can be achieved by calling the predict_proba()<\/em> function on the model.<\/p>\n The example below demonstrates how to predict a multinomial probability distribution for a new example using the multinomial logistic regression model.<\/p>\n Running the example first fits the model on all available data, then defines a row of data, which is provided to the model in order to predict class probabilities.<\/p>\n Note<\/strong>: Your results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n In this case, we can see that class 1 (e.g. the array index is mapped to the class integer value) has the largest predicted probability with about 0.50.<\/p>\n Now that we are familiar with evaluating and using multinomial logistic regression models, let’s explore how we might tune the model hyperparameters.<\/p>\n An important hyperparameter to tune for multinomial logistic regression is the penalty term.<\/p>\n This term imposes pressure on the model to seek smaller model weights. This is achieved by adding a weighted sum of the model coefficients to the loss function, encouraging the model to reduce the size of the weights along with the error while fitting the model.<\/p>\n A popular type of penalty is the L2 penalty that adds the (weighted) sum of the squared coefficients to the loss function. A weighting of the coefficients can be used that reduces the strength of the penalty from full penalty to a very slight penalty.<\/p>\n By default, the LogisticRegression<\/em> class uses the L2 penalty with a weighting of coefficients set to 1.0. The type of penalty can be set via the “penalty<\/em>” argument with values of “l1<\/em>“, “l2<\/em>“, “elasticnet<\/em>” (e.g. both), although not all solvers support all penalty types. The weighting of the coefficients in the penalty can be set via the “C<\/em>” argument.<\/p>\n The weighting for the penalty is actually the inverse weighting, perhaps penalty = 1 – C.<\/p>\n From the documentation:<\/p>\n C : float, default=1.0 This means that values close to 1.0 indicate very little penalty and values close to zero indicate a strong penalty. A C value of 1.0 may indicate no penalty at all.<\/p>\n The penalty can be disabled by setting the “penalty<\/em>” argument to the string “none<\/em>“.<\/p>\n Now that we are familiar with the penalty, let’s look at how we might explore the effect of different penalty values on the performance of the multinomial logistic regression model.<\/p>\n It is common to test penalty values on a log scale in order to quickly discover the scale of penalty that works well for a model. Once found, further tuning at that scale may be beneficial.<\/p>\n We will explore the L2 penalty with weighting values in the range from 0.0001 to 1.0 on a log scale, in addition to no penalty or 0.0.<\/p>\n The complete example of evaluating L2 penalty values for multinomial logistic regression is listed below.<\/p>\n Running the example reports the mean classification accuracy for each configuration along the way.<\/p>\n Note<\/strong>: Your results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n In this case, we can see that a C value of 1.0 has the best score of about 77.7 percent, which is the same as using no penalty that achieves the same score.<\/p>\n A box and whisker plot is created for the accuracy scores for each configuration and all plots are shown side by side on a figure on the same scale for direct comparison.<\/p>\n In this case, we can see that the larger penalty we use on this dataset (i.e. the smaller the C value), the worse the performance of the model.<\/p>\n Box and Whisker Plots of L2 Penalty Configuration vs. Accuracy for Multinomial Logistic Regression<\/p>\n<\/div>\n This section provides more resources on the topic if you are looking to go deeper.<\/p>\n In this tutorial, you discovered how to develop multinomial logistic regression models in Python.<\/p>\n Specifically, you learned:<\/p>\n Do you have any questions?<\/strong> The post Multinomial Logistic Regression With Python<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":" Author: Jason Brownlee Multinomial logistic regression is an extension of logistic regression that adds native support for multi-class classification problems. Logistic regression, by default, is […] Read More<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":4251,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[24],"tags":[],"_links":{"self":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/4250"}],"collection":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/comments?post=4250"}],"version-history":[{"count":0,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/4250\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media\/4251"}],"wp:attachment":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media?parent=4250"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/categories?post=4250"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/tags?post=4250"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
![\"Multinomial](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/01\/Multinomial-Logistic-Regression-With-Python.jpg\")
<\/p>\n
Photo by Nicolas Rénac<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
\n
Multinomial Logistic Regression<\/h2>\n
\n
\n
Evaluate Multinomial Logistic Regression Model<\/h2>\n
# test classification dataset\r\nfrom collections import Counter\r\nfrom sklearn.datasets import make_classification\r\n# define dataset\r\nX, y = make_classification(n_samples=1000, n_features=10, n_informative=5, n_redundant=5, n_classes=3, random_state=1)\r\n# summarize the dataset\r\nprint(X.shape, y.shape)\r\nprint(Counter(y))<\/pre>\n
(1000, 10) (1000,)\r\nCounter({1: 334, 2: 334, 0: 332})<\/pre>\n...\r\n# define the multinomial logistic regression model\r\nmodel = LogisticRegression(multi_class='multinomial', solver='lbfgs')<\/pre>\n
# evaluate multinomial logistic regression model\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.model_selection import RepeatedStratifiedKFold\r\nfrom sklearn.linear_model import LogisticRegression\r\n# define dataset\r\nX, y = make_classification(n_samples=1000, n_features=10, n_informative=5, n_redundant=5, n_classes=3, random_state=1)\r\n# define the multinomial logistic regression model\r\nmodel = LogisticRegression(multi_class='multinomial', solver='lbfgs')\r\n# define the model evaluation procedure\r\ncv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\r\n# evaluate the model and collect the scores\r\nn_scores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\r\n# report the model performance\r\nprint('Mean Accuracy: %.3f (%.3f)' % (mean(n_scores), std(n_scores)))<\/pre>\nMean Accuracy: 0.681 (0.042)<\/pre>\n
# make a prediction with a multinomial logistic regression model\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.linear_model import LogisticRegression\r\n# define dataset\r\nX, y = make_classification(n_samples=1000, n_features=10, n_informative=5, n_redundant=5, n_classes=3, random_state=1)\r\n# define the multinomial logistic regression model\r\nmodel = LogisticRegression(multi_class='multinomial', solver='lbfgs')\r\n# fit the model on the whole dataset\r\nmodel.fit(X, y)\r\n# define a single row of input data\r\nrow = [1.89149379, -0.39847585, 1.63856893, 0.01647165, 1.51892395, -3.52651223, 1.80998823, 0.58810926, -0.02542177, -0.52835426]\r\n# predict the class label\r\nyhat = model.predict([row])\r\n# summarize the predicted class\r\nprint('Predicted Class: %d' % yhat[0])<\/pre>\nPredicted Class: 1<\/pre>\n
# predict probabilities with a multinomial logistic regression model\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.linear_model import LogisticRegression\r\n# define dataset\r\nX, y = make_classification(n_samples=1000, n_features=10, n_informative=5, n_redundant=5, n_classes=3, random_state=1)\r\n# define the multinomial logistic regression model\r\nmodel = LogisticRegression(multi_class='multinomial', solver='lbfgs')\r\n# fit the model on the whole dataset\r\nmodel.fit(X, y)\r\n# define a single row of input data\r\nrow = [1.89149379, -0.39847585, 1.63856893, 0.01647165, 1.51892395, -3.52651223, 1.80998823, 0.58810926, -0.02542177, -0.52835426]\r\n# predict a multinomial probability distribution\r\nyhat = model.predict_proba([row])\r\n# summarize the predicted probabilities\r\nprint('Predicted Probabilities: %s' % yhat[0])<\/pre>\nPredicted Probabilities: [0.16470456 0.50297138 0.33232406]<\/pre>\n
Tune Penalty for Multinomial Logistic Regression<\/h2>\n
...\r\n# define the multinomial logistic regression model with a default penalty\r\nLogisticRegression(multi_class='multinomial', solver='lbfgs', penalty='l2', C=1.0)<\/pre>\n
\n
\nInverse of regularization strength; must be a positive float. Like in support vector machines, smaller values specify stronger regularization.<\/p>\n<\/blockquote>\n\n
...\r\n# define the multinomial logistic regression model without a penalty\r\nLogisticRegression(multi_class='multinomial', solver='lbfgs', penalty='none')<\/pre>\n
# tune regularization for multinomial logistic regression\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.model_selection import RepeatedStratifiedKFold\r\nfrom sklearn.linear_model import LogisticRegression\r\nfrom matplotlib import pyplot\r\n\r\n# get the dataset\r\ndef get_dataset():\r\n\tX, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=1, n_classes=3)\r\n\treturn X, y\r\n\r\n# get a list of models to evaluate\r\ndef get_models():\r\n\tmodels = dict()\r\n\tfor p in [0.0, 0.0001, 0.001, 0.01, 0.1, 1.0]:\r\n\t\t# create name for model\r\n\t\tkey = '%.4f' % p\r\n\t\t# turn off penalty in some cases\r\n\t\tif p == 0.0:\r\n\t\t\t# no penalty in this case\r\n\t\t\tmodels[key] = LogisticRegression(multi_class='multinomial', solver='lbfgs', penalty='none')\r\n\t\telse:\r\n\t\t\tmodels[key] = LogisticRegression(multi_class='multinomial', solver='lbfgs', penalty='l2', C=p)\r\n\treturn models\r\n\r\n# evaluate a give model using cross-validation\r\ndef evaluate_model(model, X, y):\r\n\t# define the evaluation procedure\r\n\tcv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\r\n\t# evaluate the model\r\n\tscores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\r\n\treturn scores\r\n\r\n# define dataset\r\nX, y = get_dataset()\r\n# get the models to evaluate\r\nmodels = get_models()\r\n# evaluate the models and store results\r\nresults, names = list(), list()\r\nfor name, model in models.items():\r\n\t# evaluate the model and collect the scores\r\n\tscores = evaluate_model(model, X, y)\r\n\t# store the results\r\n\tresults.append(scores)\r\n\tnames.append(name)\r\n\t# summarize progress along the way\r\n\tprint('>%s %.3f (%.3f)' % (name, mean(scores), std(scores)))\r\n# plot model performance for comparison\r\npyplot.boxplot(results, labels=names, showmeans=True)\r\npyplot.show()<\/pre>\n>0.0000 0.777 (0.037)\r\n>0.0001 0.683 (0.049)\r\n>0.0010 0.762 (0.044)\r\n>0.0100 0.775 (0.040)\r\n>0.1000 0.774 (0.038)\r\n>1.0000 0.777 (0.037)<\/pre>\n
![\"Box](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/09\/Box-and-Whisker-Plots-of-L2-Penalty-Configuration-vs-Accuracy-for-Multinomial-Logistic-Regression.png\")
<\/p>\nFurther Reading<\/h2>\n
Related Tutorials<\/h3>\n
\n
APIs<\/h3>\n
\n
Articles<\/h3>\n
\n
Summary<\/h2>\n
\n
\nAsk your questions in the comments below and I will do my best to answer.<\/p>\n