\u00a0<\/p>\n This post is part of my forthcoming book The Mathematical Foundations of Data Science<\/strong>. Probability is one of the foundations of machine learning (along with linear algebra and optimization). In this post, we discuss the areas where probability theory could apply in machine learning applications. If you want to know more about the book, follow me on Ajit Jaokar linked<\/a> \u00a0<\/p>\n \u00a0<\/p>\n \u00a0<\/p>\n First, we explore some background behind probability theory<\/p>\n \u00a0<\/p>\n Probability is a measure of uncertainty. Probability applies to machine learning because in the real world, we need to make decisions with incomplete information. Hence, we need a mechanism to quantify uncertainty \u2013 which Probability provides us. Using probability, we can model elements of uncertainty such as risk in financial transactions and many other business processes. In contrast, in traditional programming, we work with deterministic problems i.e. the solution is not affected by uncertainty.\u00a0<\/p>\n Probability quantifies the likelihood or belief that an event will occur. Probability theory has three important concepts: Even<\/strong>t – an outcome to which a probability is assigned; The Sample Space<\/strong> which represents the set of possible outcomes for the events and the Probability Function<\/strong> which maps a probability to an event. The probability function indicates the likelihood that the event being a part of the sample space is drawn. The probability distribution<\/strong> represents the shape or distribution of all events in the sample space. The probability of an event can be calculated directly by counting all the occurrences of the event and dividing them by the total possible outcomes of the event. Probability is a fractional value and has a value in the range between 0 and 1, where 0 indicates no probability and 1 represents full probability. \u00a0<\/p>\n There are two ways of interpreting probability: frequentist probability<\/strong> which considers the actual likelihood of an event and the Bayesian probability<\/strong> which considers how strongly we believe that an event will occur. frequentist probability includes techniques like p-values<\/strong> and confidence intervals<\/strong> used in statistical inference and maximum likelihood<\/strong> estimation for parameter estimation<\/strong>.<\/p>\n \u00a0<\/p>\n Frequentist techniques are based on counts and Bayesian techniques are based on beliefs. In the Bayesian approach, probabilities are assigned to events based on evidence and personal belief. The Bayesian techniques are based on the Bayes\u2019 theorem. Bayseian analysis can be used to model events that have not occurred before or occur infrequently. In contrast, frequentist techniques are based on sampling<\/strong> \u2013 hence the frequency of occurrence of an event. \u00a0For example, the p\u00adValue<\/a> indicates a number between 0 and 1. The larger the p-value \u2013 the more the data conforms to the null hypothesis. The smaller the p-value, the more the data conforms to the alternate hypothesis. If p-value is less than 0.05, then we reject the null hypothesis i.e. accept the alternate hypothesis.<\/p>\n \u00a0<\/p>\n With this background, let us explore how probability can apply to machine learning<\/p>\n \u00a0<\/p>\n Probability forms the basis of sampling. In machine learning, uncertainty can arise in many ways \u2013 for example – noise in data. Probability provides a set of tools to model uncertainty. Noise could arise due to variability in the observations, as a measurement error or from other sources. Noise effects both inputs and outputs. \u00a0<\/p>\n Apart from noise in the sample data, we should also cater for the effects of bias. Even when the observations are uniformly sampled i.e. no bias is assumed in the sampling \u2013 other limitations can introduce bias. For example, if we choose a set of participants from a specific region of the country., by definition. the sample is biased to that region. We could expand the sample scope and variance in the data by including more regions in the country. We need to balance the variance and the bias so that the sample chosen is representative of the task we are trying to model.<\/p>\n Typically, we are given a dataset i.e. we do not have control on the creation and sampling process of the dataset. To cater for this lack of control over sampling, we split the data into train and test sets or we use resampling techniques. Hence, probability (through sampling) is involved when we have incomplete coverage of the problem domain.<\/em><\/strong> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/p>\n \u00a0<\/p>\n Pattern recognition is a key part of machine learning. We can approach machine learning as a pattern recognition problem from a Bayesian standpoint. In Pattern Recognition<\/a> \u2013 Christopher Bishop takes a Bayesian view and presents approximate inference algorithms for situations where exact answers are not feasible. For the same reasons listed above, Probability theory is a key part of pattern recognition because it helps to cater for noise \/ uncertainty and for the finite size of the sample and also to apply Bayesian principles to machine learning.<\/p>\n \u00a0<\/p>\n \u00a0<\/strong><\/p>\n \u00a0<\/p>\n Many iterative machine learning techniques like Maximum likelihood estimation<\/a> (MLE) are based on probability theory. MLE is used for training in models like linear regression, logistic regression and artificial neural networks.<\/p>\n \u00a0<\/p>\n Probability forms the basis of specific algorithms like Naive Bayes classifier<\/a> \u00a0<\/p>\n \u00a0<\/p>\n In machine learning models such as neural networks, hyperparameters are tuned through techniques like grid search. Bayesian optimization can be also used for hyperparameter optimization.<\/p>\n \u00a0<\/p>\n In binary classification tasks, we predict a single probability score. Model evaluation techniques require us to summarize the performance of a model based on predicted probabilities. For example \u2013 aggregation measures like log loss<\/a> require the understanding of probability theory<\/p>\n \u00a0<\/p>\n Applied fields of study<\/strong><\/p>\n Probability forms the foundation of many fields such as physics, biology, and computer science where maths is applied.<\/p>\n \u00a0<\/p>\n Inference<\/strong><\/p>\n Probability is a key part of inference – MLE for frequentist and Bayesian inference for Bayesian \u00a0<\/p>\n<\/p>\n As we see above, there are many areas of machine learning where probability concepts apply. Yet, they are not so commonly taught in typical coding programs on machine learning. In the last blog, we discussed this trend in context of correlation vs causation.<\/a> I suspect the same is true i.e. the starting point for most developers is a dataset which they are already provided. In contrast, if you conduct a PhD experiment \/ thesis \u2013 you have to typically build your experiment from scratch. \u00a0<\/p>\n \u00a0<\/p>\n If you want to know more about the book, follow me on Ajit Jaokar linked<\/a> \u00a0<\/p>\n \u00a0<\/p>\n \u00a0<\/p>\n Image source Dice<\/a><\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<\/a><\/p>\n
Background<\/h2>\n
Probability as a measure of uncertainty<\/h3>\n
Probability of an event<\/h3>\n
Two Schools of Probability<\/h3>\n
Applications<\/h2>\n
Sampling – Dealing with non-deterministic processes<\/h3>\n
Pattern recognition<\/h3>\n
Training – \u00a0use in Maximum likelihood estimation<\/h3>\n
Developing specific algorithms<\/h3>\n
Hyperparameter optimization<\/h3>\n
Model evaluation<\/h3>\n
Conclusion<\/h2>\n<\/p>\n