Semi-supervised learning<\/strong> refers to algorithms that attempt to make use of both labeled and unlabeled training data.<\/p>\n Semi-supervised learning algorithms are unlike supervised learning algorithms that are only able to learn from labeled training data.<\/p>\n A popular approach to semi-supervised learning is to create a graph that connects examples in the training dataset and propagate known labels through the edges of the graph to label unlabeled examples. An example of this approach to semi-supervised learning is the label propagation algorithm<\/strong> for classification predictive modeling.<\/p>\n In this tutorial, you will discover how to apply the label propagation algorithm to a semi-supervised learning classification dataset.<\/p>\n After completing this tutorial, you will know:<\/p>\n Let’s get started.<\/p>\n Semi-Supervised Learning With Label Propagation This tutorial is divided into three parts; they are:<\/p>\n Label Propagation is a semi-supervised learning algorithm.<\/p>\n The algorithm was proposed in the 2002 technical report by Xiaojin Zhu and Zoubin Ghahramani titled “Learning From Labeled And Unlabeled Data With Label Propagation<\/a>.”<\/p>\n The intuition for the algorithm is that a graph is created that connects all examples (rows) in the dataset based on their distance, such as Euclidean distance<\/a>. Nodes in the graph then have label soft labels or label distribution based on the labels or label distributions of examples connected nearby in the graph.<\/p>\n Many semi-supervised learning algorithms rely on the geometry of the data induced by both labeled and unlabeled examples to improve on supervised methods that use only the labeled data. This geometry can be naturally represented by an empirical graph g = (V,E) where nodes V = {1,…,n} represent the training data and edges E represent similarities between them<\/p>\n<\/blockquote>\n — Page 193, Semi-Supervised Learning<\/a>, 2006.<\/p>\n Propagation refers to the iterative nature that labels are assigned to nodes in the graph and propagate along the edges of the graph to connected nodes.<\/p>\n This procedure is sometimes called label propagation, as it “propagates” labels from the labeled vertices (which are fixed) gradually through the edges to all the unlabeled vertices.<\/p>\n<\/blockquote>\n — Page 48, Introduction to Semi-Supervised Learning<\/a>, 2009.<\/p>\n The process is repeated for a fixed number of iterations to strengthen the labels assigned to unlabeled examples.<\/p>\n Starting with nodes 1, 2,…,l labeled with their known label (1 or −1) and nodes l + 1,…,n labeled with 0, each node starts to propagate its label to its neighbors, and the process is repeated until convergence.<\/p>\n<\/blockquote>\n — Page 194, Semi-Supervised Learning<\/a>, 2006.<\/p>\n Now that we are familiar with the Label Propagation algorithm, let’s look at how we might use it on a project. First, we must define a semi-supervised classification dataset.<\/p>\n In this section, we will define a dataset for semis-supervised learning and establish a baseline in performance on the dataset.<\/p>\n First, we can define a synthetic classification dataset using the make_classification() function<\/a>.<\/p>\n We will define the dataset with two classes (binary classification) and two input variables and 1,000 examples.<\/p>\n Next, we will split the dataset into train and test datasets with an equal 50-50 split (e.g. 500 rows in each).<\/p>\n Finally, we will split the training dataset in half again into a portion that will have labels and a portion that we will pretend is unlabeled.<\/p>\n Tying this together, the complete example of preparing the semi-supervised learning dataset is listed below.<\/p>\n Running the example prepares the dataset and then summarizes the shape of each of the three portions.<\/p>\n The results confirm that we have a test dataset of 500 rows, a labeled training dataset of 250 rows, and 250 rows of unlabeled data.<\/p>\n A supervised learning algorithm will only have 250 rows from which to train a model.<\/p>\n A semi-supervised learning algorithm will have the 250 labeled rows as well as the 250 unlabeled rows that could be used in numerous ways to improve the labeled training dataset.<\/p>\n Next, we can establish a baseline in performance on the semi-supervised learning dataset using a supervised learning algorithm fit only on the labeled training data.<\/p>\n This is important because we would expect a semi-supervised learning algorithm to outperform a supervised learning algorithm fit on the labeled data alone. If this is not the case, then the semi-supervised learning algorithm does not have skill.<\/p>\n In this case, we will use a logistic regression algorithm fit on the labeled portion of the training dataset.<\/p>\n The model can then be used to make predictions on the entire hold out test dataset and evaluated using classification accuracy.<\/p>\n Tying this together, the complete example of evaluating a supervised learning algorithm on the semi-supervised learning dataset is listed below.<\/p>\n Running the algorithm fits the model on the labeled training dataset and evaluates it on the holdout dataset and prints the classification accuracy.<\/p>\n Note<\/strong>: Your results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n In this case, we can see that the algorithm achieved a classification accuracy of about 84.8 percent.<\/p>\n We would expect an effective semi-supervised learning algorithm to achieve better accuracy than this.<\/p>\n Next, let’s explore how to apply the label propagation algorithm to the dataset.<\/p>\n The Label Propagation algorithm is available in the scikit-learn Python machine learning library via the LabelPropagation class<\/a>.<\/p>\n The model can be fit just like any other classification model by calling the fit()<\/em> function and used to make predictions for new data via the predict()<\/em> function.<\/p>\n Importantly, the training dataset provided to the fit()<\/em> function must include labeled examples that are integer encoded (as per normal) and unlabeled examples marked with a label of -1.<\/p>\n The model will then determine a label for the unlabeled examples as part of fitting the model.<\/p>\n After the model is fit, the estimated labels for the labeled and unlabeled data in the training dataset is available via the “transduction_<\/em>” attribute on the LabelPropagation<\/em> class.<\/p>\n Now that we are familiar with how to use the Label Propagation algorithm in scikit-learn, let’s look at how we might apply it to our semi-supervised learning dataset.<\/p>\n First, we must prepare the training dataset.<\/p>\n We can concatenate the input data of the training dataset into a single array.<\/p>\n We can then create a list of -1 valued (unlabeled) for each row in the unlabeled portion of the training dataset.<\/p>\n This list can then be concatenated with the labels from the labeled portion of the training dataset to correspond with the input array for the training dataset.<\/p>\n We can now train the LabelPropagation<\/em> model on the entire training dataset.<\/p>\n Next, we can use the model to make predictions on the holdout dataset and evaluate the model using classification accuracy.<\/p>\n Tying this together, the complete example of evaluating label propagation on the semi-supervised learning dataset is listed below.<\/p>\n Running the algorithm fits the model on the entire training dataset and evaluates it on the holdout dataset and prints the classification accuracy.<\/p>\n Note<\/strong>: Your results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n In this case, we can see that the label propagation model achieves a classification accuracy of about 85.6 percent, which is slightly higher than a logistic regression fit only on the labeled training dataset that achieved an accuracy of about 84.8 percent.<\/p>\n So far, so good.<\/p>\n Another approach we can use with the semi-supervised model is to take the estimated labels for the training dataset and fit a supervised learning model.<\/p>\n Recall that we can retrieve the labels for the entire training dataset from the label propagation model as follows:<\/p>\n We can then use these labels along with all of the input data to train and evaluate a supervised learning algorithm, such as a logistic regression model.<\/p>\n The hope is that the supervised learning model fit on the entire training dataset would achieve even better performance than the semi-supervised learning model alone.<\/p>\n Tying this together, the complete example of using the estimated training set labels to train and evaluate a supervised learning model is listed below.<\/p>\n Running the algorithm fits the semi-supervised model on the entire training dataset, then fits a supervised learning model on the entire training dataset with inferred labels and evaluates it on the holdout dataset, printing the classification accuracy.<\/p>\n Note<\/strong>: Your results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n In this case, we can see that this hierarchical approach of the semi-supervised model followed by supervised model achieves a classification accuracy of about 86.2 percent on the holdout dataset, even better than the semi-supervised learning used alone that achieved an accuracy of about 85.6 percent.<\/p>\n Can you achieve better results by tuning the hyperparameters of the LabelPropagation model?<\/strong> This section provides more resources on the topic if you are looking to go deeper.<\/p>\n In this tutorial, you discovered how to apply the label propagation algorithm to a semi-supervised learning classification dataset.<\/p>\n Specifically, you learned:<\/p>\n Do you have any questions?<\/strong> The post Semi-Supervised Learning With Label Propagation<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":" Author: Jason Brownlee Semi-supervised learning refers to algorithms that attempt to make use of both labeled and unlabeled training data. Semi-supervised learning algorithms are unlike […] Read More<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":4245,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[24],"tags":[],"_links":{"self":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/4244"}],"collection":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/comments?post=4244"}],"version-history":[{"count":0,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/4244\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media\/4245"}],"wp:attachment":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media?parent=4244"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/categories?post=4244"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/tags?post=4244"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
![\"Semi-Supervised](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/04\/Semi-Supervised-Learning-With-Label-Propagation.jpg\")
<\/p>\n
Photo by TheBluesDude<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
\n
Label Propagation Algorithm<\/h2>\n
\n
\n
\n
Semi-Supervised Classification Dataset<\/h2>\n
...\r\n# define dataset\r\nX, y = make_classification(n_samples=1000, n_features=2, n_informative=2, n_redundant=0, random_state=1)<\/pre>\n
...\r\n# split into train and test\r\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.50, random_state=1, stratify=y)<\/pre>\n
...\r\n# split train into labeled and unlabeled\r\nX_train_lab, X_test_unlab, y_train_lab, y_test_unlab = train_test_split(X_train, y_train, test_size=0.50, random_state=1, stratify=y_train)<\/pre>\n
# prepare semi-supervised learning dataset\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.model_selection import train_test_split\r\n# define dataset\r\nX, y = make_classification(n_samples=1000, n_features=2, n_informative=2, n_redundant=0, random_state=1)\r\n# split into train and test\r\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.50, random_state=1, stratify=y)\r\n# split train into labeled and unlabeled\r\nX_train_lab, X_test_unlab, y_train_lab, y_test_unlab = train_test_split(X_train, y_train, test_size=0.50, random_state=1, stratify=y_train)\r\n# summarize training set size\r\nprint('Labeled Train Set:', X_train_lab.shape, y_train_lab.shape)\r\nprint('Unlabeled Train Set:', X_test_unlab.shape, y_test_unlab.shape)\r\n# summarize test set size\r\nprint('Test Set:', X_test.shape, y_test.shape)<\/pre>\nLabeled Train Set: (250, 2) (250,)\r\nUnlabeled Train Set: (250, 2) (250,)\r\nTest Set: (500, 2) (500,)<\/pre>\n
...\r\n# define model\r\nmodel = LogisticRegression()\r\n# fit model on labeled dataset\r\nmodel.fit(X_train_lab, y_train_lab)<\/pre>\n
...\r\n# make predictions on hold out test set\r\nyhat = model.predict(X_test)\r\n# calculate score for test set\r\nscore = accuracy_score(y_test, yhat)\r\n# summarize score\r\nprint('Accuracy: %.3f' % (score*100))<\/pre>\n# baseline performance on the semi-supervised learning dataset\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.model_selection import train_test_split\r\nfrom sklearn.metrics import accuracy_score\r\nfrom sklearn.linear_model import LogisticRegression\r\n# define dataset\r\nX, y = make_classification(n_samples=1000, n_features=2, n_informative=2, n_redundant=0, random_state=1)\r\n# split into train and test\r\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.50, random_state=1, stratify=y)\r\n# split train into labeled and unlabeled\r\nX_train_lab, X_test_unlab, y_train_lab, y_test_unlab = train_test_split(X_train, y_train, test_size=0.50, random_state=1, stratify=y_train)\r\n# define model\r\nmodel = LogisticRegression()\r\n# fit model on labeled dataset\r\nmodel.fit(X_train_lab, y_train_lab)\r\n# make predictions on hold out test set\r\nyhat = model.predict(X_test)\r\n# calculate score for test set\r\nscore = accuracy_score(y_test, yhat)\r\n# summarize score\r\nprint('Accuracy: %.3f' % (score*100))<\/pre>\nAccuracy: 84.800<\/pre>\n
Label Propagation for Semi-Supervised Learning<\/h2>\n
...\r\n# define model\r\nmodel = LabelPropagation()\r\n# fit model on training dataset\r\nmodel.fit(..., ...)\r\n# make predictions on hold out test set\r\nyhat = model.predict(...)<\/pre>\n
...\r\n# get labels for entire training dataset data\r\ntran_labels = model.transduction_<\/pre>\n
...\r\n# create the training dataset input\r\nX_train_mixed = concatenate((X_train_lab, X_test_unlab))<\/pre>\n
...\r\n# create \"no label\" for unlabeled data\r\nnolabel = [-1 for _ in range(len(y_test_unlab))]<\/pre>\n
...\r\n# recombine training dataset labels\r\ny_train_mixed = concatenate((y_train_lab, nolabel))<\/pre>\n
...\r\n# define model\r\nmodel = LabelPropagation()\r\n# fit model on training dataset\r\nmodel.fit(X_train_mixed, y_train_mixed)<\/pre>\n
...\r\n# make predictions on hold out test set\r\nyhat = model.predict(X_test)\r\n# calculate score for test set\r\nscore = accuracy_score(y_test, yhat)\r\n# summarize score\r\nprint('Accuracy: %.3f' % (score*100))<\/pre>\n# evaluate label propagation on the semi-supervised learning dataset\r\nfrom numpy import concatenate\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.model_selection import train_test_split\r\nfrom sklearn.metrics import accuracy_score\r\nfrom sklearn.semi_supervised import LabelPropagation\r\n# define dataset\r\nX, y = make_classification(n_samples=1000, n_features=2, n_informative=2, n_redundant=0, random_state=1)\r\n# split into train and test\r\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.50, random_state=1, stratify=y)\r\n# split train into labeled and unlabeled\r\nX_train_lab, X_test_unlab, y_train_lab, y_test_unlab = train_test_split(X_train, y_train, test_size=0.50, random_state=1, stratify=y_train)\r\n# create the training dataset input\r\nX_train_mixed = concatenate((X_train_lab, X_test_unlab))\r\n# create \"no label\" for unlabeled data\r\nnolabel = [-1 for _ in range(len(y_test_unlab))]\r\n# recombine training dataset labels\r\ny_train_mixed = concatenate((y_train_lab, nolabel))\r\n# define model\r\nmodel = LabelPropagation()\r\n# fit model on training dataset\r\nmodel.fit(X_train_mixed, y_train_mixed)\r\n# make predictions on hold out test set\r\nyhat = model.predict(X_test)\r\n# calculate score for test set\r\nscore = accuracy_score(y_test, yhat)\r\n# summarize score\r\nprint('Accuracy: %.3f' % (score*100))<\/pre>\nAccuracy: 85.600<\/pre>\n
...\r\n# get labels for entire training dataset data\r\ntran_labels = model.transduction_<\/pre>\n
...\r\n# define supervised learning model\r\nmodel2 = LogisticRegression()\r\n# fit supervised learning model on entire training dataset\r\nmodel2.fit(X_train_mixed, tran_labels)\r\n# make predictions on hold out test set\r\nyhat = model2.predict(X_test)\r\n# calculate score for test set\r\nscore = accuracy_score(y_test, yhat)\r\n# summarize score\r\nprint('Accuracy: %.3f' % (score*100))<\/pre>\n# evaluate logistic regression fit on label propagation for semi-supervised learning\r\nfrom numpy import concatenate\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.model_selection import train_test_split\r\nfrom sklearn.metrics import accuracy_score\r\nfrom sklearn.semi_supervised import LabelPropagation\r\nfrom sklearn.linear_model import LogisticRegression\r\n# define dataset\r\nX, y = make_classification(n_samples=1000, n_features=2, n_informative=2, n_redundant=0, random_state=1)\r\n# split into train and test\r\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.50, random_state=1, stratify=y)\r\n# split train into labeled and unlabeled\r\nX_train_lab, X_test_unlab, y_train_lab, y_test_unlab = train_test_split(X_train, y_train, test_size=0.50, random_state=1, stratify=y_train)\r\n# create the training dataset input\r\nX_train_mixed = concatenate((X_train_lab, X_test_unlab))\r\n# create \"no label\" for unlabeled data\r\nnolabel = [-1 for _ in range(len(y_test_unlab))]\r\n# recombine training dataset labels\r\ny_train_mixed = concatenate((y_train_lab, nolabel))\r\n# define model\r\nmodel = LabelPropagation()\r\n# fit model on training dataset\r\nmodel.fit(X_train_mixed, y_train_mixed)\r\n# get labels for entire training dataset data\r\ntran_labels = model.transduction_\r\n# define supervised learning model\r\nmodel2 = LogisticRegression()\r\n# fit supervised learning model on entire training dataset\r\nmodel2.fit(X_train_mixed, tran_labels)\r\n# make predictions on hold out test set\r\nyhat = model2.predict(X_test)\r\n# calculate score for test set\r\nscore = accuracy_score(y_test, yhat)\r\n# summarize score\r\nprint('Accuracy: %.3f' % (score*100))<\/pre>\nAccuracy: 86.200<\/pre>\n
\nLet me know what you discover in the comments below.<\/p>\nFurther Reading<\/h2>\n
Books<\/h3>\n
\n
Papers<\/h3>\n
\n
APIs<\/h3>\n
\n
Articles<\/h3>\n
\n
Summary<\/h2>\n
\n
\nAsk your questions in the comments below and I will do my best to answer.<\/p>\n