Ordinary Least Squares

February 21, 2026

Direct solution method - Ordinary Least Squares and the Line of Best Fit #

It is possible to compute the best parameters for linear regression in one shot (closed-form), instead of iteratively improving them step-by-step. fileciteturn34file10turn34file6

For linear regression, the direct method is usually Ordinary Least Squares (OLS).

Ordinary Least Squares (OLS) chooses the “best” line by minimising squared prediction errors.

Key takeaway: OLS defines “best fit” as the line that minimises the total squared residual error across all data points.

Cost Function

February 21, 2026

Cost Function #

• also known as an objective function

• how far the predicted values are from the actual ones

• measure of the difference between predicted values and actual values

• quantifies the error between a model’s predicted values and actual values

• measures the model’s error on a group of datapoints

• method used to predict values by drawing the best-fit line through the data

• used to evaluate the accuracy of a model’s predictions