Introduced in “Attention is all you need”, self-attention helps in making static word embeddings into more, richer contextual embeddings. In this blog, we will just focus upon the mathematics behind self-attention and later will implement it in PyTorch.
Before we proceed, we need to have a basic understanding of some simple technical terms that we’ll be using quite frequently. If you are doubtful about any, I have attached a simple explanation of each of them in the appendix section.