Introduced in “Attention is all you need”, self-attention helps in making static word embeddings into more, richer contextual embeddings. In this blog, we will just focus upon the mathematics behind self-attention and later will implement it in PyTorch.

Table of contents

  1. Heads-Up: Key Terms Ahead
  2. Why do we need Self-Attention?
  3. Self-Attention in Layman terms
    1. Masked Self-Attention
  4. Self-Attention Theory
  5. Self-Attention Implementation (PyTorch)
  6. Appendix
  7. Resources
  8. Contact Me

Heads-Up: Key Terms Ahead

Before we proceed, we need to have a basic understanding of some simple technical terms that we’ll be using quite frequently. If you are doubtful about any, I have attached a simple explanation of each of them in the appendix section.