https://arshadhs.github.io/docs/ai/010-maths/020-calculus/06-nonlinear-optimisation/Recent content in Nonlinear Optimisation on Arshad SiddiquiHugoen-ushttps://arshadhs.github.io/docs/ai/010-maths/020-calculus/06-nonlinear-optimisation/optimisation-challenges/Mon, 01 Jan 0001 00:00:00 +0000https://arshadhs.github.io/docs/ai/010-maths/020-calculus/06-nonlinear-optimisation/optimisation-challenges/<h1 id="challenges-in-gradient-based-optimisation"> Challenges in Gradient-Based Optimisation <a class="anchor" href="#challenges-in-gradient-based-optimisation">#</a> </h1> <ul> <li>Local optima and flat regions</li> <li>Differential curvature</li> <li>Difficult topologies (cliffs and valleys)</li> </ul> <hr> <p><a href="https://arshadhs.github.io/">Home</a> | <a href="https://arshadhs.github.io/docs/ai/010-maths/020-calculus/06-nonlinear-optimisation/"> Nonlinear Optimisation </a></p>https://arshadhs.github.io/docs/ai/010-maths/020-calculus/06-nonlinear-optimisation/stochastic-gradient-descent/Mon, 01 Jan 0001 00:00:00 +0000https://arshadhs.github.io/docs/ai/010-maths/020-calculus/06-nonlinear-optimisation/stochastic-gradient-descent/<h1 id="stochastic-gradient-descent-sgd"> Stochastic Gradient Descent (SGD) <a class="anchor" href="#stochastic-gradient-descent-sgd">#</a> </h1> <p>SGD uses mini-batches to trade exact gradients for speed and generalisation.</p> <hr> <p><a href="https://arshadhs.github.io/">Home</a> | <a href="https://arshadhs.github.io/docs/ai/010-maths/020-calculus/06-nonlinear-optimisation/"> Nonlinear Optimisation </a></p>https://arshadhs.github.io/docs/ai/010-maths/020-calculus/06-nonlinear-optimisation/momentum-methods/Mon, 01 Jan 0001 00:00:00 +0000https://arshadhs.github.io/docs/ai/010-maths/020-calculus/06-nonlinear-optimisation/momentum-methods/<h1 id="momentum-based-learning"> Momentum-Based Learning <a class="anchor" href="#momentum-based-learning">#</a> </h1> <p>Momentum smooths updates and helps traverse valleys efficiently.</p> <hr> <p><a href="https://arshadhs.github.io/">Home</a> | <a href="https://arshadhs.github.io/docs/ai/010-maths/020-calculus/06-nonlinear-optimisation/"> Nonlinear Optimisation </a></p>https://arshadhs.github.io/docs/ai/010-maths/020-calculus/06-nonlinear-optimisation/adaptive-methods/Mon, 01 Jan 0001 00:00:00 +0000https://arshadhs.github.io/docs/ai/010-maths/020-calculus/06-nonlinear-optimisation/adaptive-methods/<h1 id="adaptive-methods-adagrad-rmsprop-adam"> Adaptive Methods: AdaGrad, RMSProp, Adam <a class="anchor" href="#adaptive-methods-adagrad-rmsprop-adam">#</a> </h1> <p>Adaptive methods adjust learning rates per-parameter.</p> <hr> <p><a href="https://arshadhs.github.io/">Home</a> | <a href="https://arshadhs.github.io/docs/ai/010-maths/020-calculus/06-nonlinear-optimisation/"> Nonlinear Optimisation </a></p>https://arshadhs.github.io/docs/ai/010-maths/020-calculus/06-nonlinear-optimisation/hyperparameter-tuning/Mon, 01 Jan 0001 00:00:00 +0000https://arshadhs.github.io/docs/ai/010-maths/020-calculus/06-nonlinear-optimisation/hyperparameter-tuning/<h1 id="tuning-hyperparameters-and-preprocessing"> Tuning Hyperparameters and Preprocessing <a class="anchor" href="#tuning-hyperparameters-and-preprocessing">#</a> </h1> <ul> <li>Learning rate schedules</li> <li>Initialisation</li> <li>Tuning hyperparameters</li> <li>Importance of feature preprocessing</li> </ul> <hr> <p><a href="https://arshadhs.github.io/">Home</a> | <a href="https://arshadhs.github.io/docs/ai/010-maths/020-calculus/06-nonlinear-optimisation/"> Nonlinear Optimisation </a></p>