https://arshadhs.github.io/tags/gradient-descent/Recent content in Gradient Descent on Arshad SiddiquiHugoen-usSat, 21 Feb 2026 00:00:00 +0000https://arshadhs.github.io/docs/ai/machine-learning/03-gradient-descent-linear-regression/Sat, 21 Feb 2026 00:00:00 +0000https://arshadhs.github.io/docs/ai/machine-learning/03-gradient-descent-linear-regression/<h1 id="gradient-descent-for-linear-regression"> Gradient Descent for Linear Regression <a class="anchor" href="#gradient-descent-for-linear-regression">#</a> </h1> <p>Gradient descent is an iterative optimisation method used to minimise the regression cost function by repeatedly updating parameters in the direction that reduces error.</p> <ul> <li><strong>Iterative method</strong></li> <li>Types: batch / stochastic / mini-batch</li> </ul> <blockquote class="book-hint info"> <p>Key takeaway: Gradient descent starts with initial parameter values and repeatedly updates them using the gradient until the cost stops decreasing.</p> </blockquote> <script src="https://arshadhs.github.io/mermaid.min.js"></script> <script>mermaid.initialize({ "flowchart": { "useMaxWidth":true }, "theme": "default" } )</script> <pre class="mermaid"> flowchart TD GD[&#34;Gradient&lt;br/&gt;Descent&#34;] --&gt;|minimises| CF[&#34;Cost&lt;br/&gt;function&#34;] GD --&gt;|updates| W[&#34;Parameters&lt;br/&gt;(weights)&#34;] GD --&gt;|uses| GR[&#34;Gradient&lt;br/&gt;(slope)&#34;] GD --&gt; H[&#34;Hyperparameters&#34;] H --&gt; LR[&#34;Learning&lt;br/&gt;rate&#34;] H --&gt; BS[&#34;Batch&lt;br/&gt;size&#34;] H --&gt; EP[&#34;Epochs&#34;] style GD fill:#90CAF9,stroke:#1E88E5,color:#000 style CF fill:#CE93D8,stroke:#8E24AA,color:#000 style W fill:#CE93D8,stroke:#8E24AA,color:#000 style GR fill:#CE93D8,stroke:#8E24AA,color:#000 style H fill:#CE93D8,stroke:#8E24AA,color:#000 style LR fill:#CE93D8,stroke:#8E24AA,color:#000 style BS fill:#CE93D8,stroke:#8E24AA,color:#000 style EP fill:#CE93D8,stroke:#8E24AA,color:#000 </pre> <hr> <h2 id="types-of-gd"> Types of GD <a class="anchor" href="#types-of-gd">#</a> </h2> <pre class="mermaid"> flowchart TD T[&#34;Gradient Descent&lt;br/&gt;types&#34;] --&gt; BGD[&#34;Batch&lt;br/&gt;GD&#34;] T --&gt; SGD[&#34;Stochastic&lt;br/&gt;GD&#34;] T --&gt; MGD[&#34;Mini-batch&lt;br/&gt;GD&#34;] BGD --&gt; ALL[&#34;All data&lt;br/&gt;per step&#34;] BGD --&gt; STB[&#34;Smooth&lt;br/&gt;updates&#34;] SGD --&gt; ONE[&#34;1 sample&lt;br/&gt;per step&#34;] SGD --&gt; FAST[&#34;Quick&lt;br/&gt;progress&#34;] SGD --&gt; NOISE[&#34;Noisy&lt;br/&gt;updates&#34;] MGD --&gt; MB[&#34;Small batch&lt;br/&gt;per step&#34;] MGD --&gt; PRACT[&#34;Practical&lt;br/&gt;default&#34;] style T fill:#90CAF9,stroke:#1E88E5,color:#000 style BGD fill:#C8E6C9,stroke:#2E7D32,color:#000 style SGD fill:#C8E6C9,stroke:#2E7D32,color:#000 style MGD fill:#C8E6C9,stroke:#2E7D32,color:#000 style ALL fill:#CE93D8,stroke:#8E24AA,color:#000 style STB fill:#CE93D8,stroke:#8E24AA,color:#000 style ONE fill:#CE93D8,stroke:#8E24AA,color:#000 style FAST fill:#CE93D8,stroke:#8E24AA,color:#000 style NOISE fill:#CE93D8,stroke:#8E24AA,color:#000 style MB fill:#CE93D8,stroke:#8E24AA,color:#000 style PRACT fill:#CE93D8,stroke:#8E24AA,color:#000 </pre> <h3 id="batch"> Batch <a class="anchor" href="#batch">#</a> </h3> <ul> <li>Use only if you have huge compute and a lot of time to train</li> </ul> <h3 id="sgd"> SGD <a class="anchor" href="#sgd">#</a> </h3> <ul> <li> <p>go-to solution</p>