In Language Modeling, (Selective) Attention Matters

With concepts and examples in question-answering tasks

Image by David from Pixabay

Well, it’s hard to argue against paying attention to the right things. Unfortunately, language modeling algorithms were not doing so for a long time.

This all changed about half a decade back. The rest is history. As witnessed most recently with ChatGPT.