Natural Language Processing

Before 2017, NLP relied on RNNs/LSTMs, which processed words sequentially. The (introduced in "Attention Is All You Need") introduced two breakthroughs:

The development of NLP includes three eras, each with a different approach to machine communication: natural language processing

A large model is pretrained on billions of words (e.g., internet text) using self-supervised objectives (masked word prediction, next sentence prediction). Then it is fine-tuned on a small, task-specific dataset. Before 2017, NLP relied on RNNs/LSTMs, which processed