https://arshadhs.github.io/tags/attention/Recent content in Attention on Arshad SiddiquiHugoen-ushttps://arshadhs.github.io/docs/ai/deep-learning/080-attention-mechanism/Mon, 01 Jan 0001 00:00:00 +0000https://arshadhs.github.io/docs/ai/deep-learning/080-attention-mechanism/<h1 id="attention-mechanism"> Attention Mechanism <a class="anchor" href="#attention-mechanism">#</a> </h1> <p>Attention is a deep learning mechanism that allows a model to focus on the most relevant parts of an input sequence when producing an output.</p> <p>Instead of compressing the whole input into one fixed vector, attention computes a weighted combination of useful information.</p> <blockquote class="book-hint info"> <p><strong>Key takeaway:</strong><br> Attention answers a simple question:</p> <blockquote class='book-hint '> <p>For the current prediction, which input tokens should the model focus on most?</p> </blockquote></blockquote> <ul> <li>Queries, Keys, and Values</li> <li>Attention Pooling by Similarity</li> <li>Attention Pooling via Nadaraya–Watson Regression</li> <li>Attention Scoring Functions</li> <li>Dot Product Attention</li> <li>Convenience Functions</li> <li>Scaled Dot Product Attention</li> <li>Additive Attention</li> <li>Bahdanau Attention Mechanism</li> <li>Multi-Head Attention</li> <li>Self-Attention</li> <li>Positional Encoding</li> </ul> <hr> <h2 id="why-attention-is-needed-"> Why Attention Is Needed ☆ <a class="anchor" href="#why-attention-is-needed-">#</a> </h2> <p>Traditional encoder-decoder RNN models compress the full input sequence into one context vector.</p>