ML System Optimization,
MLSysOps,
Multi-Core CPU,
GPGPU,
SIMD,
MIMD,
SIMT,
GPU,
TPU,
AI,
ML Parallel Programming Models
#
Parallel algorithms need hardware that can execute independent work efficiently. Modern systems therefore combine multiple CPU cores, memory hierarchies, threads, instruction pipelines, GPUs, clusters, and specialised matrix processors.
This page covers:
- multi-core CPU organisation
- cache and memory hierarchy
- processes, threads, scheduling, and synchronisation
- instruction pipelining and clock-cycle time
- SIMD, MIMD, and SIMT execution
- GPGPU architecture and GPU memory behaviour
- CPU-only and GPU-accelerated clusters
- Tensor Processing Units and systolic arrays
Learning Objectives
#
By the end of this page, you should be able to:
Linear Regression
#
Linear Regression is a supervised
ML
method used to predict a numerical target by fitting a model that is linear in its parameters.
In
ML
, linear models are a core baseline:
they’re fast, often surprisingly strong, and usually easy to interpret.
Key takeaway:
Linear Regression learns parameters by minimising a squared-error cost.
You can solve it directly (closed form) or iteratively (gradient descent),
and you can extend it using basis functions and regularisation.
February 21, 2026Direct solution method - Ordinary Least Squares and the Line of Best Fit
#
Revision:
OLS is the direct method for linear regression. It finds the best-fit line by minimising the sum of squared residuals without iterative updates.
Direct Method vs Iterative Method ☆
#
Linear regression parameters can be found in two main ways.
| Method | Main idea | When used |
|---|
| Ordinary Least Squares | Compute the best parameters directly | Small or moderate datasets |
| Gradient Descent | Start with parameters and update repeatedly | Large datasets or many features |
flowchart LR
A["Linear Regression"] --> B["Direct Solution<br/>OLS"]
A --> C["Iterative Solution<br/>Gradient Descent"]
B --> B1["Normal Equation"]
B --> B2["No learning rate"]
B --> B3["One-shot solution"]
C --> C1["Learning rate"]
C --> C2["Repeated updates"]
C --> C3["Stops after convergence"]
style A fill:#E1F5FE,stroke:#5b7db1,color:#000
style B fill:#C8E6C9,stroke:#5f8f6a,color:#000
style C fill:#FFF9C4,stroke:#b59b3b,color:#000
style B1 fill:#EDE7F6,stroke:#8a6fb3,color:#000
style B2 fill:#EDE7F6,stroke:#8a6fb3,color:#000
style B3 fill:#EDE7F6,stroke:#8a6fb3,color:#000
style C1 fill:#EDE7F6,stroke:#8a6fb3,color:#000
style C2 fill:#EDE7F6,stroke:#8a6fb3,color:#000
style C3 fill:#EDE7F6,stroke:#8a6fb3,color:#000
Why It Is Called “Least Squares” ☆
#
OLS is called least squares because it chooses parameters that make the squared residual errors as small as possible.
February 21, 2026 Cost Function
#
Revision:
A cost function converts model error into a single number. Training means changing the model parameters until this number becomes as small as possible.
Why Cost Function Matters in ML ☆
#
A machine learning model needs a way to decide whether one set of parameters is better than another.
For linear regression, every possible value of the parameters gives a different line.
The cost function tells us which line is better by measuring how far the predictions are from the true values.
February 21, 2026 Gradient Descent for Linear Regression
#
Revision:
Gradient descent is the step-by-step method for reducing the cost function when a direct closed-form solution is not convenient.
Where Gradient Descent Fits in ML ☆
#
Gradient descent is used when we want the model to learn parameters by repeatedly improving them.
For linear regression, it adjusts the slope and intercept until the prediction error becomes small.
flowchart LR
A["Initial Parameters"] --> B["Make Predictions"]
B --> C["Compute Cost"]
C --> D["Compute Gradient"]
D --> E["Update Parameters"]
E --> B
style A fill:#E1F5FE,stroke:#5b7db1,color:#000
style B fill:#C8E6C9,stroke:#5f8f6a,color:#000
style C fill:#FFF9C4,stroke:#b59b3b,color:#000
style D fill:#EDE7F6,stroke:#8a6fb3,color:#000
style E fill:#C8E6C9,stroke:#5f8f6a,color:#000
Core Idea ☆
#
The gradient tells us the direction in which the cost increases fastest.
Linear models for Classification
#
- categorises data by finding a linear boundary (hyperplane) that separates classes
- calculating a weighted sum of input features plus bias
flowchart TD
T["Linear<br/>classification<br/>models"] --> P["Perceptron"]
T --> LR["Logistic<br/>regression"]
T --> SVM["Linear<br/>SVM"]
P -->|uses| STEP["Step<br/>activation"]
LR -->|uses| SIG["Sigmoid<br/>+ log loss"]
SVM -->|uses| HNG["Hinge<br/>loss"]
style T fill:#90CAF9,stroke:#1E88E5,color:#000
style P fill:#C8E6C9,stroke:#2E7D32,color:#000
style LR fill:#C8E6C9,stroke:#2E7D32,color:#000
style SVM fill:#C8E6C9,stroke:#2E7D32,color:#000
style STEP fill:#CE93D8,stroke:#8E24AA,color:#000
style SIG fill:#CE93D8,stroke:#8E24AA,color:#000
style HNG fill:#CE93D8,stroke:#8E24AA,color:#000
- Discriminant Functions
- Decision Theory
- Probabilistic Discriminative Classifiers
- Logistic Regression
Logistic Regression
#
- Supervised machine learning algorithm
- Binary classification algorithm
- requires data to be linearly separable
- predicts the probability that an input belongs to a specific class
- uses Sigmoid function to convert inputs into a probability value between 0 and 1
Key takeaway:
Logistic regression predicts $P(y=1\mid x)$ using a sigmoid of a linear score $z=w\cdot x+b$,
then learns $w,b$ by maximising likelihood (equivalently minimising log-loss).
Hypothesis Testing
#
Hypothesis testing is a statistical decision-making method used to decide whether sample evidence is strong enough to reject an initial assumption about a population.
It connects probability, sampling distributions, confidence intervals, significance levels, and decision rules.
Key takeaway:
Hypothesis testing is not about proving something with certainty.
It is about asking:
If the null hypothesis were true, how surprising would this sample result be?
Neural Networks and Neural Language Modelling
#
Neural networks learn useful representations and nonlinear relationships directly from data. In language modelling, they replace discrete N-gram identities with learned word embeddings and use these representations to predict the next word.
Learning Objectives
#
- Explain the computation performed by a neural unit.
- Describe why hidden layers and nonlinear activations are needed.
- Explain how feed-forward networks support NLP classification.
- Trace the flow through a feed-forward neural language model.
- Compare N-gram and neural language models.
Big Picture
#
flowchart TD
A["Context Words"] --> B["One-hot Inputs"]
B --> C["Embedding Lookup"]
C --> D["Combined Context"]
D --> E["Hidden Layer"]
E --> F["Softmax"]
F --> G["Next-word Probabilities"]
style A fill:#E1F5FE
style B fill:#C8E6C9
style C fill:#FFF9C4
style D fill:#EDE7F6
style E fill:#E1F5FE
style F fill:#C8E6C9
style G fill:#FFF9C4
1. Neural Network Units ☆
#
A neural unit receives input values, multiplies them by learned weights, adds a bias, and applies an activation function.
Rewards, Returns, Policies and Value Functions
#
An MDP describes how states, actions, rewards and transitions fit together. The next task is to evaluate behaviour: what should the agent try to achieve, how should future rewards be counted, and how good is a state or action over the long term?
Rewards define the objective, returns combine rewards across time, a policy describes behaviour, and value functions predict the long-term quality of that behaviour.
December 14, 2025 Foundation Model
#
AI models trained on massive datasets to perform a wide range of tasks with minimal fine-tuning.
are large deep learning neural networks
are large AI models trained on massive and diverse datasets (text, images, audio, or multiple modalities).
Contain millions or billions of parameters.
designed to perform a broad range of general tasks
designed for general-purpose intelligence, not a single task.
acts as base models for building specialised AI applications