Bayesian Learning #

Bayesian Learning is a probabilistic approach to machine learning.

Instead of only asking, “Which output should the model predict?”, Bayesian Learning asks:

Given the data we have observed, how likely is each hypothesis, class, or parameter value?

This makes Bayesian Learning useful when uncertainty matters.

It is especially important in classification, probabilistic modelling, generative models, and situations where we want to combine prior knowledge with observed data.

Key takeaway:
Bayesian Learning updates belief using evidence.

Prior belief plus observed data gives posterior belief.

• MLE Hypothesis
• MAP Hypothesis
• Bayes Rule
• Optimal Bayes Classifier
• Naive Bayes Classifier
• Probabilistic Generative Classifiers
• Bayesian Linear Regression

Big Picture #

Bayesian Learning is built around this idea:

1. Start with a prior belief.
2. Observe data.
3. Compute how likely the data is under each hypothesis.
4. Update the belief.
5. Choose the most probable class or hypothesis.

flowchart LR
    A["Prior Knowledge"] --> B["Observed Data"]
    B --> C["Likelihood"]
    C --> D["Posterior"]
    D --> E["Prediction / Classification"]

    style A fill:#E1F5FE,stroke:#5b7db1,color:#222
    style B fill:#C8E6C9,stroke:#5f8f6a,color:#222
    style C fill:#FFF9C4,stroke:#b59b3b,color:#222
    style D fill:#EDE7F6,stroke:#8a6fb3,color:#222
    style E fill:#FDE2E4,stroke:#b85c68,color:#222

Bayes Rule ☆ #

Bayes Rule tells us how to update probability when new evidence is observed.

In classification, we often want to find the probability of a class after seeing the input features.

Where:

Bayes Rule in Hypothesis Form #

In Bayesian Learning, we may compare different hypotheses.

A hypothesis could be a model, a class, or a parameter setting.

Where:

The lecturer emphasised the interpretation:

• prior: belief before seeing data
• likelihood: probability of data under a hypothesis
• posterior: belief after seeing the evidence

Prior, Likelihood, Evidence and Posterior ☆ #

Prior #

The prior is what we believe before seeing the current data.

Example:

Before seeing symptoms, we may know that a disease is rare.

Likelihood #

The likelihood tells us how likely the evidence is if a class or hypothesis is true.

Example:

If the patient has flu, how likely are fever and cough?

Evidence #

The evidence is the overall probability of observing the input.

In classification, it acts as a normalising denominator.

Posterior #

The posterior is the updated probability after seeing evidence.

MLE Hypothesis ☆ #

MLE stands for Maximum Likelihood Estimation.

MLE chooses the hypothesis or parameter value that makes the observed data most likely.

It focuses only on the likelihood term.

If the model has parameters \( \theta \) , we write:

MLE Using Log-Likelihood ☆ #

Products of many probabilities can become very small.

So we usually maximise the log-likelihood instead.

If the data points are independent:

Taking log:

So:

MLE is data-driven.

It does not include prior belief. It only asks which parameter makes the observed data most probable.

MAP Hypothesis ☆ #

MAP stands for Maximum A Posteriori.

MAP chooses the hypothesis with the highest posterior probability after observing the data.

Using Bayes Rule:

Since \( P(D) \) is the same for all hypotheses, we can ignore it when comparing hypotheses.

For parameters:

MAP Using Log Form #

This shows the difference clearly:

• MLE uses only the data likelihood.
• MAP uses likelihood plus prior.

MAP can be interpreted as MLE with a prior-based preference.

This is why MAP is often connected to regularisation.

MLE vs MAP #

MAP Classification Rule ☆ #

In classification, the MAP rule chooses the class with the highest posterior probability.

Using Bayes Rule:

The denominator \( P(X) \) is ignored because it is common for all classes.

Probabilistic Generative Classifiers ☆ #

A probabilistic generative classifier models how data is generated for each class.

It learns:

• the prior probability of each class
• the likelihood of features given each class

Then it applies Bayes Rule to predict the class.

flowchart LR
    A["Training Data"] --> B["Estimate Class Prior P(C)"]
    A --> C["Estimate Likelihood P(X | C)"]
    B --> D["Apply Bayes Rule"]
    C --> D
    D --> E["Choose Class with Highest Posterior"]

    style A fill:#E1F5FE,stroke:#5b7db1,color:#222
    style B fill:#C8E6C9,stroke:#5f8f6a,color:#222
    style C fill:#FFF9C4,stroke:#b59b3b,color:#222
    style D fill:#EDE7F6,stroke:#8a6fb3,color:#222
    style E fill:#FDE2E4,stroke:#b85c68,color:#222

Naive Bayes Classifier ☆ #

Naive Bayes is a probabilistic classifier based on Bayes Rule.

It is called naive because it assumes that features are conditionally independent given the class.

This assumption is often not perfectly true in real data, but the method can still work surprisingly well.

Naive Bayes Formula #

For features \( X = (x_1, x_2, \ldots, x_n) \) :

Using the naive independence assumption:

So the classification rule becomes:

Why Naive Bayes Is Useful #

Naive Bayes is commonly used for:

• spam detection
• sentiment classification
• document classification
• medical diagnosis support
• text categorisation

It is fast because it only needs to estimate simple probabilities.

It is especially useful when the number of features is large, such as in text data.

Types of Naive Bayes #

Laplace Smoothing ☆ #

If a feature value never appears with a class in the training data, its probability becomes zero.

That would make the whole product zero.

Laplace smoothing avoids this problem.

Where \( V \) is the number of possible feature values.

Without smoothing, one unseen feature value can force a class probability to become zero.

Optimal Bayes Classifier #

The Bayes optimal classifier chooses the class with the highest posterior probability.

It gives the best possible classification decision if the true probability distributions are known.

In practice, the true distributions are unknown, so methods such as Naive Bayes estimate them from data.

Bayesian Linear Regression #

In ordinary linear regression, we estimate a single best set of weights.

In Bayesian Linear Regression, the weights are treated as random variables.

We place a prior over the weights and update it using observed data.

A common assumption is Gaussian noise:

The likelihood becomes:

If we use a Gaussian prior over weights:

Then Bayes Rule gives the posterior over weights:

Bayesian Linear Regression Intuition #

Ordinary regression says:

These are the best weights.

Bayesian regression says:

These weights are more likely, but there is uncertainty.

This is useful because it can give uncertainty around predictions.

Worked Mini Example: MAP Classification #

Suppose we want to classify an email as spam or not spam.

Let the evidence be:

• contains the word “offer”
• contains the word “free”

We compare:

The predicted class is whichever value is larger.

Notes ☆ #

You should be comfortable with:

1. Identifying prior, likelihood, evidence and posterior from Bayes Rule.
2. Explaining the difference between MLE and MAP.
3. Writing the MAP classification rule.
4. Applying Naive Bayes using class priors and conditional probabilities.
5. Explaining why Naive Bayes assumes conditional independence.
6. Explaining why Laplace smoothing is needed.
7. Comparing discriminative and generative classifiers.

Common Mistakes #

Revision #

Home | Machine Learning