Monte Carlo Methods

Monte Carlo Methods #

Monte Carlo (MC) methods learn value functions and policies by averaging returns observed in complete episodes. Unlike Dynamic Programming, they do not require transition probabilities or a complete model of the environment.

1. The Main Idea ☆ #

Suppose the agent repeatedly reaches a state \( s \) . Each time, it records the return received from that point until the episode ends. The average of these returns estimates the state’s value.

Prediction & Forecasting

Prediction & Forecasting #

Prediction and forecasting use statistical models to estimate unknown or future values.

In this module, the focus is on correlation, regression, and time series forecasting.

Key takeaway:
Prediction estimates a value using a model.

Forecasting is prediction where the order of time matters.

• Correlation
• Regression
• Time series analysis
• Components of time series data
• Moving average and weighted moving average
• AR model
• ARMA model
• ARIMA model
• SARIMA and SARIMAX
• VAR and VARMAX
• Simple exponential smoothing

Prediction vs Forecasting ☆ #

Overall Workflow #

flowchart LR
    A[Data] --> B[Explore Pattern]
    B --> C[Choose Model]
    C --> D[Train or Fit]
    D --> E[Validate]
    E --> F[Predict or Forecast]
    F --> G[Interpret Error]

    style A fill:#E1F5FE
    style B fill:#C8E6C9
    style C fill:#FFF9C4
    style D fill:#EDE7F6
    style E fill:#C8E6C9
    style F fill:#E1F5FE
    style G fill:#FFF9C4

Correlation ☆ #

Correlation measures the direction and strength of linear relationship between two variables.

Clusters, Hadoop, and Spark

Clusters, Hadoop, and Spark #

Scale-out frameworks distribute data and computation across networked machines. Their value comes from aggregate compute and storage, while their main costs are communication, scheduling, serialisation, and repeated synchronisation.

Course coverage:

1. Cluster computing with Hadoop and Spark
2. MapReduce k-means, mini-batch and streaming variants
3. Convergence and communication reduction
4. Distributed convolutional neural networks: data and model parallelism
5. Layer or tensor slicing, mixed precision, pipelining, and distributed execution frameworks

Learning Objectives #

By the end of this page, you should be able to:

Gaussian Mixture Model & Expectation Maximization

Gaussian Mixture Model & Expectation Maximization #

A Gaussian Mixture Model represents data as a weighted combination of multiple Gaussian distributions.

It is commonly used for soft clustering and density estimation.

Key takeaway:
K-means gives hard cluster membership.

GMM gives probabilities of belonging to each cluster.

• Gaussian Mixture Model
• soft clustering
• mixing coefficients
• latent variables
• likelihood and log-likelihood
• Expectation-Maximization algorithm
• E-step and M-step
• responsibilities
• convergence

Motivation ☆ #

Many real datasets are not described well by one Gaussian distribution.

Instance-based Learning

Instance-based Learning #

Instance-based learning is a family of methods that do not build one explicit global model during training. Instead, they store training examples and delay most of the work until a new query arrives.

When a new point must be classified or predicted, the algorithm compares it with previously seen examples, finds the most relevant neighbours, and uses them to produce the answer.

Instance-based Learning covers three linked ideas:

Temporal-Difference Learning I

Temporal-Difference Learning I #

Temporal-Difference (TD) learning updates predictions from one transition at a time. It learns directly from experience like Monte Carlo methods, but it does not need to wait until the episode ends.

Course Content covered in this module:

• Temporal-Difference Learning
• TD(0), SARSA, Q-Learning and Expected SARSA

Learning Objectives #

By the end of this module, you should be able to:

Support Vector Machine

May 8, 2026

Support Vector Machine (SVM) #

Support Vector Machine (SVM) is a supervised machine learning algorithm used for:

• Classification (most common)
• Regression (SVR – Support Vector Regression)

It connects many earlier ideas:

• classification and decision boundaries
• linear classifiers
• margins
• optimisation
• constrained optimisation
• kernels for non-linear data

SVM is a discriminative classifier.

That means it does not try to model how each class is generated.

Instead, it tries to find the best separating boundary between classes.

Distributed Training Strategies

Distributed Training Strategies #

Distributed training combines devices to increase throughput, fit larger models, or shorten time to solution. The central design question is what to partition: examples, model parameters, layers, tensors, or the training schedule.

Course coverage:

1. Data parallelism
2. Model parallelism
3. Pipeline parallelism and micro-batches
4. Gradient checkpointing
5. Mixed-precision training and loss scaling

Learning Objectives #

By the end of this page, you should be able to:

• compare data, model, and pipeline parallelism
• calculate global batch size, speedup, efficiency, and effective throughput
• explain gradient aggregation through all-reduce
• calculate simple pipeline time and utilisation
• quantify the memory–computation trade-off of checkpointing
• explain mixed precision and loss scaling

Big Picture #

flowchart TD
    A["Training Constraint"] --> B["More Data Throughput"]
    A --> C["Model Too Large"]
    A --> D["Activation Memory Too High"]
    B --> E["Data Parallelism"]
    C --> F["Model or Pipeline Parallelism"]
    D --> G["Checkpointing or Mixed Precision"]

    style A fill:#E1F5FE
    style B fill:#C8E6C9
    style C fill:#FFF9C4
    style D fill:#EDE7F6
    style E fill:#E1F5FE
    style F fill:#C8E6C9
    style G fill:#FFF9C4

1. Data Parallelism ☆ #

Each worker holds a complete model replica and processes a different shard of the mini-batch.

Temporal-Difference Learning II and DRL Taxonomy

Temporal-Difference Learning II and DRL Taxonomy #

One-step TD methods bootstrap after a single transition. Multi-step methods instead use several observed rewards before bootstrapping, forming a bridge between TD(0) and Monte Carlo learning.

Course Content covered in this module:

• Temporal-Difference Learning: n-step returns and TD(λ)
• Classification of Reinforcement Learning approaches, algorithms and applications:
  • Model-Based versus Model-Free
  • Value-Based versus Policy-Based
  • On-Policy versus Off-Policy

Learning Objectives #

By the end of this module, you should be able to:

Bayesian Learning

Bayesian Learning #

Bayesian Learning is a probabilistic approach to machine learning.

Instead of only asking, “Which output should the model predict?”, Bayesian Learning asks:

Given the data we have observed, how likely is each hypothesis, class, or parameter value?

This makes Bayesian Learning useful when uncertainty matters.

It is especially important in classification, probabilistic modelling, generative models, and situations where we want to combine prior knowledge with observed data.