Maximum likelihood estimation is an approach to density estimation for a dataset by searching across probability distributions and their parameters.<\/p>\n
It is a general and effective approach that underlies many machine learning algorithms, although it requires that the training dataset is complete, e.g. all relevant interacting random variables are present. Maximum likelihood becomes intractable if there are variables that interact with those in the dataset but were hidden or not observed, so-called latent variables.<\/p>\n
The expectation-maximization algorithm is an approach for performing maximum likelihood estimation in the presence of latent variables. It does this by first estimating the values for the latent variables, then optimizing the model, then repeating these two steps until convergence. It is an effective and general approach and is most commonly used for density estimation with missing data, such as clustering algorithms like the Gaussian Mixture Model.<\/p>\n
In this post, you will discover the expectation-maximization algorithm.<\/p>\n
After reading this post, you will know:<\/p>\n
- \n
- Maximum likelihood estimation is challenging on data in the presence of latent variables.<\/li>\n
- Expectation maximization provides an iterative solution to maximum likelihood estimation with latent variables.<\/li>\n
- Gaussian mixture models are an approach to density estimation where the parameters of the distributions are fit using the expectation-maximization algorithm.<\/li>\n<\/ul>\n
Discover bayes opimization, naive bayes, maximum likelihood, distributions, cross entropy, and much more in my new book<\/a>, with 28 step-by-step tutorials and full Python source code.<\/p>\n
Let\u2019s get started.<\/p>\n
![\"A](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/11\/A-Gentle-Introduction-to-Expectation-Maximization-EM-Algorithm.jpg\")
<\/p>\nA Gentle Introduction to Expectation Maximization (EM Algorithm)
Photo by valcker<\/a>, some rights reserved.<\/p>\n<\/div>\nOverview<\/h2>\n
This tutorial is divided into four parts; they are:<\/p>\n
- \n
- Problem of Latent Variables for Maximum Likelihood<\/li>\n
- Expectation-Maximization Algorithm<\/li>\n
- Gaussian Mixture Model and the EM Algorithm<\/li>\n
- Example of Gaussian Mixture Model<\/li>\n<\/ol>\n
Problem of Latent Variables for Maximum Likelihood<\/h2>\n
A common modeling problem involves how to estimate a joint probability distribution for a dataset.<\/p>\n
Density estimation<\/a> involves selecting a probability distribution function and the parameters of that distribution that best explain the joint probability distribution of the observed data.<\/p>\n
There are many techniques for solving this problem, although a common approach is called maximum likelihood estimation, or simply \u201cmaximum likelihood<\/em>.\u201d<\/p>\n
Maximum Likelihood Estimation<\/a> involves treating the problem as an optimization or search problem, where we seek a set of parameters that results in the best fit for the joint probability of the data sample.<\/p>\n
A limitation of maximum likelihood estimation is that it assumes that the dataset is complete, or fully observed. This does not mean that the model has access to all data; instead, it assumes that all variables that are relevant to the problem are present.<\/p>\n
This is not always the case. There may be datasets where only some of the relevant variables can be observed, and some cannot, and although they influence other random variables in the dataset, they remain hidden.<\/p>\n
More generally, these unobserved or hidden variables are referred to as latent variables<\/a>.<\/p>\n
\n
Many real-world problems have hidden variables (sometimes called latent variables), which are not observable in the data that are available for learning.<\/p>\n<\/blockquote>\n
\u2014 Page 816, Artificial Intelligence: A Modern Approach<\/a>, 3rd edition, 2009.<\/p>\n
Conventional maximum likelihood estimation does not work well in the presence of latent variables.<\/p>\n
\n
\u2026 if we have missing data and\/or latent variables, then computing the [maximum likelihood] estimate becomes hard.<\/p>\n<\/blockquote>\n