1.Before Transformers: Recurrent Neural Networks
Early NLP models used RNNs and LSTMs, which had trouble with long-range dependencies in text. They processed text sequentially, making parallelization difficult.
2.The Transformer Paper: Attention Is All You Need
Published in 2017, this paper introduced the Transformer architecture, which uses self-attention to process text in parallel, allowing much larger models.
3.Self-Attention Mechanism
Self-attention allows each token to "attend to" every other token in the sequence, capturing relationships regardless of their position.
4.Scaling Laws
LLMs demonstrate predictable scaling behavior: model performance improves with more parameters, more data, and more compute.