The Gaussian Processes Classifier<\/strong> is a classification machine learning algorithm.<\/p>\n Gaussian Processes are a generalization of the Gaussian probability distribution and can be used as the basis for sophisticated non-parametric machine learning algorithms for classification and regression.<\/p>\n They are a type of kernel model, like SVMs, and unlike SVMs, they are capable of predicting highly calibrated class membership probabilities, although the choice and configuration of the kernel used at the heart of the method can be challenging.<\/p>\n In this tutorial, you will discover the Gaussian Processes Classifier classification machine learning algorithm.<\/p>\n After completing this tutorial, you will know:<\/p>\n Let’s get started.<\/p>\n Gaussian Processes for Classification With Python This tutorial is divided into three parts; they are:<\/p>\n Gaussian Processes, or GP for short, are a generalization of the Gaussian probability distribution<\/a> (e.g. the bell-shaped function).<\/p>\n Gaussian probability distribution functions summarize the distribution of random variables, whereas Gaussian processes summarize the properties of the functions, e.g. the parameters of the functions. As such, you can think of Gaussian processes as one level of abstraction or indirection above Gaussian functions.<\/p>\n A Gaussian process is a generalization of the Gaussian probability distribution. Whereas a probability distribution describes random variables which are scalars or vectors (for multivariate distributions), a stochastic process governs the properties of functions.<\/p>\n<\/blockquote>\n — Page 2, Gaussian Processes for Machine Learning<\/a>, 2006.<\/p>\n Gaussian processes can be used as a machine learning algorithm for classification predictive modeling.<\/p>\n Gaussian processes are a type of kernel method, like SVMs, although they are able to predict highly calibrated probabilities, unlike SVMs.<\/p>\n Gaussian processes require specifying a kernel that controls how examples relate to each other; specifically, it defines the covariance function of the data. This is called the latent function or the “nuisance<\/em>” function.<\/p>\n The latent function f plays the role of a nuisance function: we do not observe values of f itself (we observe only the inputs X and the class labels y) and we are not particularly interested in the values of f …<\/p>\n<\/blockquote>\n — Page 40, Gaussian Processes for Machine Learning<\/a>, 2006.<\/p>\n The way that examples are grouped using the kernel controls how the model “perceives<\/em>” the examples, given that it assumes that examples that are “close<\/em>” to each other have the same class label.<\/p>\n Therefore, it is important to both test different kernel functions for the model and different configurations for sophisticated kernel functions.<\/p>\n … a covariance function is the crucial ingredient in a Gaussian process predictor, as it encodes our assumptions about the function which we wish to learn.<\/p>\n<\/blockquote>\n — Page 79, Gaussian Processes for Machine Learning<\/a>, 2006.<\/p>\n It also requires a link function that interprets the internal representation and predicts the probability of class membership. The logistic function can be used, allowing the modeling of a Binomial probability distribution<\/a> for binary classification.<\/p>\n For the binary discriminative case one simple idea is to turn the output of a regression model into a class probability using a response function (the inverse of a link function), which “squashes” its argument, which can lie in the domain (−inf, inf), into the range [0, 1], guaranteeing a valid probabilistic interpretation.<\/p>\n<\/blockquote>\n — Page 35, Gaussian Processes for Machine Learning<\/a>, 2006.<\/p>\n Gaussian processes and Gaussian processes for classification is a complex topic.<\/p>\n To learn more see the text:<\/p>\n The Gaussian Processes Classifier is available in the scikit-learn Python machine learning library via the GaussianProcessClassifier class<\/a>.<\/p>\n The class allows you to specify the kernel to use via the “kernel<\/em>” argument and defaults to 1 * RBF(1.0), e.g. a RBF kernel.<\/p>\n Given that a kernel is specified, the model will attempt to best configure the kernel for the training dataset.<\/p>\n This is controlled via setting an “optimizer<\/em>“, the number of iterations for the optimizer via the “max_iter_predict<\/em>“, and the number of repeats of this optimization process performed in an attempt to overcome local optima “n_restarts_optimizer<\/em>“.<\/p>\n By default, a single optimization run is performed, and this can be turned off by setting “optimize<\/em>” to None<\/em>.<\/p>\n We can demonstrate the Gaussian Processes Classifier with a worked example.<\/p>\n First, let’s define a synthetic classification dataset.<\/p>\n We will use the make_classification() function<\/a> to create a dataset with 100 examples, each with 20 input variables.<\/p>\n The example below creates and summarizes the dataset.<\/p>\n Running the example creates the dataset and confirms the number of rows and columns of the dataset.<\/p>\n We can fit and evaluate a Gaussian Processes Classifier model using repeated stratified k-fold cross-validation via the RepeatedStratifiedKFold class<\/a>. We will use 10 folds and three repeats in the test harness.<\/p>\n We will use the default configuration.<\/p>\n The complete example of evaluating the Gaussian Processes Classifier model for the synthetic binary classification task is listed below.<\/p>\n Running the example evaluates the Gaussian Processes Classifier algorithm on the synthetic dataset and reports the average accuracy across the three repeats of 10-fold cross-validation.<\/p>\n Your specific results may vary given the stochastic nature of the learning algorithm. Consider running the example a few times.<\/p>\n In this case, we can see that the model achieved a mean accuracy of about 79.0 percent.<\/p>\n We may decide to use the Gaussian Processes Classifier as our final model and make predictions on new data.<\/p>\n This can be achieved by fitting the model pipeline on all available data and calling the predict()<\/em> function passing in a new row of data.<\/p>\n We can demonstrate this with a complete example listed below.<\/p>\n Running the example fits the model and makes a class label prediction for a new row of data.<\/p>\n Next, we can look at configuring the model hyperparameters.<\/p>\n The hyperparameters for the Gaussian Processes Classifier method must be configured for your specific dataset.<\/p>\n Perhaps the most important hyperparameter is the kernel controlled via the “kernel<\/em>” argument. The scikit-learn library provides many built-in kernels that can be used.<\/p>\n Perhaps some of the more common examples include:<\/p>\n You can learn more about the kernels offered by the library here:<\/p>\n We will evaluate the performance of the Gaussian Processes Classifier with each of these common kernels, using default arguments.<\/p>\n The example below demonstrates this using the GridSearchCV class<\/a> with a grid of values we have defined.<\/p>\n Running the example will evaluate each combination of configurations using repeated cross-validation.<\/p>\n Your specific results may vary given the stochastic nature of the learning algorithm. Try running the example a few times.<\/p>\n In this case, we can see that the RationalQuadratic<\/em> kernel achieved a lift in performance with an accuracy of about 91.3 percent as compared to 79.0 percent achieved with the RBF kernel in the previous section.<\/p>\n This section provides more resources on the topic if you are looking to go deeper.<\/p>\n In this tutorial, you discovered the Gaussian Processes Classifier classification machine learning algorithm.<\/p>\n Specifically, you learned:<\/p>\n Do you have any questions?<\/strong> The post Gaussian Processes for Classification With Python<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":" Author: Jason Brownlee The Gaussian Processes Classifier is a classification machine learning algorithm. Gaussian Processes are a generalization of the Gaussian probability distribution and can […] Read More<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":3927,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[24],"tags":[],"_links":{"self":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/3926"}],"collection":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/comments?post=3926"}],"version-history":[{"count":0,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/3926\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media\/3927"}],"wp:attachment":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media?parent=3926"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/categories?post=3926"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/tags?post=3926"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
![\"Gaussian](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/08\/Gaussian-Processes-for-Classification-With-Python.jpg\")
<\/p>\n
Photo by Mark Kao<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
\n
Gaussian Processes for Classification<\/h2>\n
\n
\n
\n
\n
\n
Gaussian Processes With Scikit-Learn<\/h2>\n
...\r\n# define model\r\nmodel = GaussianProcessClassifier(kernel=1*RBF(1.0))<\/pre>\n
...\r\n# define model\r\nmodel = GaussianProcessClassifier(optimizer=None)<\/pre>\n
# test classification dataset\r\nfrom sklearn.datasets import make_classification\r\n# define dataset\r\nX, y = make_classification(n_samples=100, n_features=20, n_informative=15, n_redundant=5, random_state=1)\r\n# summarize the dataset\r\nprint(X.shape, y.shape)<\/pre>\n
(100, 20) (100,)<\/pre>\n
...\r\n# create the model\r\nmodel = GaussianProcessClassifier()<\/pre>\n
# evaluate a gaussian process classifier model on the dataset\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.model_selection import cross_val_score\r\nfrom sklearn.model_selection import RepeatedStratifiedKFold\r\nfrom sklearn.gaussian_process import GaussianProcessClassifier\r\n# define dataset\r\nX, y = make_classification(n_samples=100, n_features=20, n_informative=15, n_redundant=5, random_state=1)\r\n# define model\r\nmodel = GaussianProcessClassifier()\r\n# define model evaluation method\r\ncv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\r\n# evaluate model\r\nscores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\r\n# summarize result\r\nprint('Mean Accuracy: %.3f (%.3f)' % (mean(scores), std(scores)))<\/pre>\nMean Accuracy: 0.790 (0.101)<\/pre>\n
# make a prediction with a gaussian process classifier model on the dataset\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.gaussian_process import GaussianProcessClassifier\r\n# define dataset\r\nX, y = make_classification(n_samples=100, n_features=20, n_informative=15, n_redundant=5, random_state=1)\r\n# define model\r\nmodel = GaussianProcessClassifier()\r\n# fit model\r\nmodel.fit(X, y)\r\n# define new data\r\nrow = [2.47475454,0.40165523,1.68081787,2.88940715,0.91704519,-3.07950644,4.39961206,0.72464273,-4.86563631,-6.06338084,-1.22209949,-0.4699618,1.01222748,-0.6899355,-0.53000581,6.86966784,-3.27211075,-6.59044146,-2.21290585,-3.139579]\r\n# make a prediction\r\nyhat = model.predict([row])\r\n# summarize prediction\r\nprint('Predicted Class: %d' % yhat)<\/pre>\nPredicted Class: 0<\/pre>\n
Tune Gaussian Processes Hyperparameters<\/h2>\n
\n
\n
...\r\n# define grid\r\ngrid = dict()\r\ngrid['kernel'] = [1*RBF(), 1*DotProduct(), 1*Matern(), 1*RationalQuadratic(), 1*WhiteKernel()]<\/pre>\n
# grid search kernel for gaussian process classifier\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.model_selection import GridSearchCV\r\nfrom sklearn.model_selection import RepeatedStratifiedKFold\r\nfrom sklearn.gaussian_process import GaussianProcessClassifier\r\nfrom sklearn.gaussian_process.kernels import RBF\r\nfrom sklearn.gaussian_process.kernels import DotProduct\r\nfrom sklearn.gaussian_process.kernels import Matern\r\nfrom sklearn.gaussian_process.kernels import RationalQuadratic\r\nfrom sklearn.gaussian_process.kernels import WhiteKernel\r\n# define dataset\r\nX, y = make_classification(n_samples=100, n_features=20, n_informative=15, n_redundant=5, random_state=1)\r\n# define model\r\nmodel = GaussianProcessClassifier()\r\n# define model evaluation method\r\ncv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\r\n# define grid\r\ngrid = dict()\r\ngrid['kernel'] = [1*RBF(), 1*DotProduct(), 1*Matern(), 1*RationalQuadratic(), 1*WhiteKernel()]\r\n# define search\r\nsearch = GridSearchCV(model, grid, scoring='accuracy', cv=cv, n_jobs=-1)\r\n# perform the search\r\nresults = search.fit(X, y)\r\n# summarize best\r\nprint('Best Mean Accuracy: %.3f' % results.best_score_)\r\nprint('Best Config: %s' % results.best_params_)\r\n# summarize all\r\nmeans = results.cv_results_['mean_test_score']\r\nparams = results.cv_results_['params']\r\nfor mean, param in zip(means, params):\r\n print(\">%.3f with: %r\" % (mean, param))<\/pre>\nBest Mean Accuracy: 0.913\r\nBest Config: {'kernel': 1**2 * RationalQuadratic(alpha=1, length_scale=1)}\r\n>0.790 with: {'kernel': 1**2 * RBF(length_scale=1)}\r\n>0.800 with: {'kernel': 1**2 * DotProduct(sigma_0=1)}\r\n>0.830 with: {'kernel': 1**2 * Matern(length_scale=1, nu=1.5)}\r\n>0.913 with: {'kernel': 1**2 * RationalQuadratic(alpha=1, length_scale=1)}\r\n>0.510 with: {'kernel': 1**2 * WhiteKernel(noise_level=1)}<\/pre>\n<\/p>\nFurther Reading<\/h2>\n
Books<\/h3>\n
\n
APIs<\/h3>\n
\n
Articles<\/h3>\n
\n
Summary<\/h2>\n
\n
\nAsk your questions in the comments below and I will do my best to answer.<\/p>\n