Transformer (deep learning architecture)

"Transformer architecture" redirects here. For the design of electrical transformers, see Transformer.

The first Transformer architecture, as described in the 2017 paper Attention Is All You Need.

A transformer is a deep learning architecture developed by Google and based on the multi-head attention mechanism, proposed in a 2017 paper "Attention Is All You Need". Text is converted to numerical representations called tokens, and each token is converted into a vector via looking up from a word embedding table. At each layer, each token is then contextualized within the scope of the context window with other (unmasked) tokens via a parallel multi-head attention mechanism allowing the signal for key tokens to be amplified and less important tokens to be diminished. The transformer paper, published in 2017, is based on the softmax-based attention mechanism proposed by Bahdanau et al. in 2014 for machine translation.

Transformers have the advantage of having no recurrent units, and therefore require less training time than earlier recurrent neural architectures (RNNs) such as long short-term memory (LSTM). Later variations have been widely adopted for training large language models (LLM) on large (language) datasets, such as the Wikipedia corpus and Common Crawl.

Transformers are currently used in large-scale natural language processing, computer vision (vision transformers), audio, multi-modal processing, robotics, and even playing chess. It has also led to the development of pre-trained systems, such as generative pre-trained transformers (GPTs) and BERT (Bidirectional Encoder Representations from Transformers).

History

See also: Attention (machine learning) § History, and Timeline of machine learning

Predecessors

Sequence modelling and generation had been done with plain recurrent neural networks for many years. An early well-cited example was the Elman network (1990). In theory, the information from one token can propagate arbitrarily far down the sequence, but in practice the vanishing-gradient problem leaves the model's state at the end of a long sentence without precise, extractable information about preceding tokens.

One key component of the attention mechanism is to include neurons that multiply the outputs of other neurons. Such neurons were called multiplicative units, and neural networks using multiplicative units were called sigma-pi networks or higher-order networks, but they faced high computational complexity. A key breakthrough was LSTM (1995), which incorporated multiplicative units into a recurrent network, as well as other innovations that prevented the vanishing gradient problem, and allowed efficient learning of long-sequence modelling. It became the standard architecture for long sequence modelling until the 2017 publication of Transformers.

However, LSTM did not solve a general problem that recurrent networks usually have, which is that it cannot operate in parallel over all tokens in a sequence. It must operate one at a time from the first token to the last. The fast weight controller (1992) was an early attempt to bypass the difficulty. It used the fast weights architecture (1987), where one neural network outputs the weights of another neural network. It was later shown to be equivalent to the linear Transformer without normalization. Schmidhuber used the terminology "learning internal spotlights of attention" in 1993, and now claims it was a precursor to what is now known as the attention mechanism, but Geoffrey Hinton disputes this claim of priority.

Attention with seq2seq

Main article: Seq2seq § History

The idea of encoder-decoder sequence transduction had been developed in the early 2010s (see for previous papers). The papers most commonly cited as the originators that produced seq2seq are two papers from 2014.

(Sutskever et al, 2014) , a 380M-parameter seq2seq model for machine translation using two long short-term memory (LSTM). The architecture consists of two parts. The encoder is an LSTM that takes in a sequence of tokens and turns it into a vector. The decoder is another LSTM that converts the vector into a sequence of tokens. Another 130M-parameter model used a gated recurrent units (GRU). Later research showed that GRUs are neither better nor worse than LSTMs for seq2seq.

These early seq2seq models had no attention mechanism, and the state vector is accessible only after the last word of the source text was processed. Although in theory such a vector retains the information about the whole original sentence, in practice the information is poorly preserved, since the input is processed sequentially by one recurrent network into a fixed-size output vector, which was then processed by another recurrent network into an output. If the input is long, then the output vector would not be able to contain all relevant information, and the output quality degrades. As evidence, reversing the input sentence improved seq2seq translation.

(Bahdanau et al, 2014) introduced an attention mechanism to seq2seq for machine translation to solve the bottleneck problem, allowing the model to process long-distance dependencies more easily. They called their model RNNsearch, as it "emulates searching through a source sentence during decoding a translation".

(Luong et al, 2015) compared the relative performance of global (that of (Bahdanau et al, 2014)) and local (sliding window) attention model architectures for machine translation, and found that a mixed attention architecture had higher quality than global attention, while the use of a local attention architecture reduced translation time.

In 2016, Google Translate was revamped to Google Neural Machine Translation, which replaced the previous model based on statistical machine translation. The new model was a seq2seq model where the encoder and the decoder were both 8 layers of bidirectional LSTM. It took nine months to develop, and it achieved a higher level of performance than the statistical approach, which took ten years to develop.

Attention

Seq2seq models with attention still suffered from the same issue with recurrent networks, which is that they are hard to parallelize, which prevented them to be accelerated on GPUs. In 2016, decomposable attention applied attention mechanism to the feedforward network, which are easy to parallelize. One of its authors, Jakob Uszkoreit, suspected that attention without recurrence is sufficient for language translation, thus the title "attention is all you need".

In 2017, the original (100M-sized) encoder-decoder transformer model was proposed in the "Attention is all you need" paper. At the time, the focus of the research was on improving seq2seq for machine translation, by removing its recurrence to processes all tokens in parallel, but preserving its dot-product attention mechanism to keep its text processing performance. Its parallelizability was an important factor to its widespread use in large neural networks.

Transformer boom

Transformers were used in many models that contributed to the AI boom.

In language modelling, ELMo (2018) was a bi-directional LSTM that produces contextualized word embeddings, improving upon the line of research from bag of words and word2vec. It was followed by BERT (2018), an encoder-only Transformer model. In 2019 October, Google started using BERT to process search queries. In 2020, Google Translate replaced the previous RNN-encoder–RNN-decoder model by a Transformer-encoder–RNN-decoder model.

Starting in 2018, the OpenAI GPT series of decoder-only Transformers became state of the art in natural language generation. In 2022, a chatbot based on GPT-3, ChatGPT, became unexpectedly popular, triggering a boom around large language models.

Since 2020, Transformers were applied to more modalities than text, including the vision transformer, speech recognition, robotics, and multimodal. The vision transformer, in turn, stimulated new developments in convolutional neural networks. Image and video generators like DALL-E (2021), Stable Diffusion 3 (2024), and Sora (2024), were based on the Transformer architecture.

Training

Methods for stabilizing training

The plain transformer architecture had difficulty converging. In the original paper the authors recommended using learning rate warmup. That is, the learning rate should linearly scale up from 0 to maximal value for the first part of the training (usually recommended to be 2% of the total number of training steps), before decaying again.

A 2020 paper found that using layer normalization before (instead of after) multiheaded attention and feedforward layers stabilizes training, not requiring learning rate warmup.

Pretrain-finetune

Transformers typically undergo self-supervised learning involving unsupervised pretraining followed by supervised fine-tuning. Pretraining is typically done on a larger dataset than fine-tuning, due to the limited availability of labeled training data. Tasks for pretraining and fine-tuning commonly include:

• language modeling
• next-sentence prediction
• question answering
• reading comprehension
• sentiment analysis
• paraphrasing

The T5 transformer report documents a large number of pretraining tasks. Some examples are:

• restoring corrupted text: Thank you <X> me to your party <Y> week. -> <X> for inviting <Y> last <Z> where the <Z> means "end of output".
• translation: translate English to German: That is good. -> Das ist gut..
• judging the grammatical acceptability of a sentence (CoLA sentence): The course is jumping well. -> not acceptable.

Tasks

See also: Large language model § Evaluation

In general, there are four classes of language modelling tasks: "masked", "autoregressive", and "prefixLM". These classes are independent of architectural choice. One does not need to use a Transformer for training language models to perform these tasks. However, they are often discussed in the context of Transformer.

In a masked task, one or more of the tokens is masked out, and the model would produce a probability distribution predicting what the masked-out tokens are based on the context. The loss function for the task is typically sum of log-perplexities for the masked-out tokens: $${\text{Loss}}=-\sum _{t\in {\text{masked tokens}}}\ln({\text{probability of }}t{\text{ conditional on its context}})$$and the model is trained to minimize this loss function. BERT is trained for two tasks, one of which was masked token prediction.

In an autoregressive task, the entire sequence is masked at first, and the model would produce a probability distribution predicting what the first token is. The first token is then revealed, and the model would predict the second token, and so on. The loss function for the task is still typically the same.

In a prefixLM task, the sequence is divided into two parts. The first part is presented as context, and the model would predict the first token of the second part. Then that would be revealed, and the model would predict the second token, and so on. The loss function for the task is still typically the same.

Note that "masked" as in "masked language modelling" is not "masked" as in "masked attention", and "prefixLM" (prefix language modeling) is not "prefixLM" (prefix language model).

Applications

The transformer has had great success in natural language processing (NLP), for example the tasks of machine translation and time series prediction. Many large language models such as GPT-2, GPT-3, GPT-4, Claude, BERT, XLNet, RoBERTa and ChatGPT demonstrate the ability of transformers to perform a wide variety of such NLP-related tasks, and have the potential to find real-world applications. These may include:

• machine translation
• document summarization
• document generation
• named entity recognition (NER)
• biological sequence analysis
• writing computer code based on requirements expressed in natural language.
• video understanding.

In addition to the NLP applications, it has also been successful in other fields, such as computer vision, or the protein folding applications (such as AlphaFold). It was also applied to evaluating chess board positions. Using static evaluation alone (that is, with no Minimax search) it was able to achieve an Elo of 2895, putting it at grandmaster level.

Implementations

The transformer model has been implemented in standard deep learning frameworks such as TensorFlow and PyTorch.

Transformers is a library produced by Hugging Face that supplies transformer-based architectures and pretrained models.

Architecture

All transformers have the same primary components:

• Tokenizers, which convert text into tokens.
• Embedding layer, which converts tokens and positions of the tokens into vector representations.
• Transformer layers, which carry out repeated transformations on the vector representations, extracting more and more linguistic information. These consist of alternating attention and feedforward layers. There are two major types of transformer layers: encoder layers and decoder layers, with further variants.
• (optional) Un-embedding layer, which converts the final vector representations back to a probability distribution over the tokens.

Tokenization

Main articles: Tokenization and Lexical analysis

As the Transformer architecture natively processes numerical data, not text, there must be a translation between text and tokens. A token is an integer that represents a character, or a short segment of characters. On the input side, the input text is parsed into a token sequence. Similarly, on the output side, the output tokens are paresd back to text. The module doing the conversion between token sequences and texts is a tokenizer.

The set of all tokens is the vocabulary of the tokenizer. When faced with tokens outside the vocabulary, typically a special token is used, written as "[UNK]" for "unknown".

Some commonly used tokenizers are byte pair encoding, WordPiece, and SentencePiece.

Embedding

Further information: Word embedding

Each token is converted into an embedding vector via a lookup table. Then, the positional encoding vectors are added to their respective token embedding vectors, producing the sequence of input vectors.

Positional encoding

A diagram of a sinusoidal positional encoding with parameters $N=10000,d=100$

A positional encoding is a fixed-size vector representation of the relative positions of tokens within a sequence: it provides the transformer model with information about where the words are in the input sequence. Without positional encoding, the model would be unable to process input sequence as more than a bag of words, as for example, both "man bites dog" and "dog bites man" would be processed exactly the same way.

The positional encoding is defined as a function of type $f:\mathbb {R} \to \mathbb {R} ^{d};d\in \mathbb {Z} ,d>0$, where $d$ is a positive even integer. The full positional encoding – as defined in the original paper – is given by the equation:$$(f(t)_{2k},f(t)_{2k 1})=(\sin(\theta ),\cos(\theta ))\quad \forall k\in \{0,1,\ldots ,d/2-1\}$$where $\theta ={\frac {t}{r^{k}}},r=N^{2/d}$.

Here, $N$ is a free parameter that should be significantly larger than the biggest $k$ that would be input into the positional encoding function. In the original paper, the authors chose $N=10000$.

The function is in a simpler form when written as a complex function of type $f:\mathbb {R} \to \mathbb {C} ^{d/2}$$$f(t)=\left(e^{it/r^{k}}\right)_{k=0,1,\ldots ,{\frac {d}{2}}-1}$$where $r=N^{2/d}$.

The main reason the authors chose this as the positional encoding function is that it allows one to perform shifts as linear transformations:$$f(t \Delta t)=\mathrm {diag} (f(\Delta t))f(t)$$where $\Delta t\in \mathbb {R}$ is the distance one wishes to shift. This allows the transformer to take any encoded position, and find the encoding of the position n-steps-ahead or n-steps-behind, by a matrix multiplication.

By taking a linear sum, any convolution can also be implemented as linear transformations:$$\sum _{j}c_{j}f(t \Delta t_{j})=\left(\sum _{j}c_{j}\,\mathrm {diag} (f(\Delta t_{j}))\right)f(t)$$for any constants $c_{j}$. This allows the transformer to take any encoded position and find a linear sum of the encoded locations of its neighbors. This sum of encoded positions, when fed into the attention mechanism, would create attention weights on its neighbors, much like what happens in a convolutional neural network language model. In the author's words, "we hypothesized it would allow the model to easily learn to attend by relative position."

In typical implementations, all operations are done over the real numbers, not the complex numbers, but since complex multiplication can be implemented as real 2-by-2 matrix multiplication, this is a mere notational difference.

Encoder-decoder (overview)

One encoder-decoder block.

A Transformer is composed of stacked encoder layers and decoder layers.

Like earlier seq2seq models, the original transformer model used an encoder-decoder architecture. The encoder consists of encoding layers that process the input tokens iteratively one layer after another, while the decoder consists of decoding layers that iteratively process the encoder's output as well as the decoder output's tokens so far.

The function of each encoder layer is to generate contextualized token representations, where each representation corresponds to a token that "mixes" information from other input tokens via self-attention mechanism. Each decoder layer contains two attention sublayers: (1) cross-attention for incorporating the output of encoder (contextualized input token representations), and (2) self-attention for "mixing" information among the input tokens to the decoder (i.e., the tokens generated so far during inference time).

Both the encoder and decoder layers have a feed-forward neural network for additional processing of the outputs and contain residual connections and layer normalization steps. These feed-forward layers contain a majority of parameters in a Transformer model.

Scaled dot-product attention

Main article: Dot-product attention

Scaled dot-product attention, block diagram.

The transformer building blocks are scaled dot-product attention units. For each attention unit, the transformer model learns three weight matrices: the query weights $W_{Q}$, the key weights $W_{K}$, and the value weights $W_{V}$. For each token $i$, the input token representation $x_{i}$ is multiplied with each of the three weight matrices to produce a query vector $q_{i}=x_{i}W_{Q}$, a key vector $k_{i}=x_{i}W_{K}$, and a value vector $v_{i}=x_{i}W_{V}$. Attention weights are calculated using the query and key vectors: the attention weight $a_{ij}$ from token $i$ to token $j$ is the dot product between $q_{i}$ and $k_{j}$. The attention weights are divided by the square root of the dimension of the key vectors, ${\sqrt {d_{k}}}$, which stabilizes gradients during training, and passed through a softmax which normalizes the weights. The fact that $W_{Q}$ and $W_{K}$ are different matrices allows attention to be non-symmetric: if token $i$ attends to token $j$ (i.e. $q_{i}\cdot k_{j}$ is large), this does not necessarily mean that token $j$ will attend to token $i$ (i.e. $q_{j}\cdot k_{i}$ could be small). The output of the attention unit for token $i$ is the weighted sum of the value vectors of all tokens, weighted by $a_{ij}$, the attention from token $i$ to each token.

The attention calculation for all tokens can be expressed as one large matrix calculation using the softmax function, which is useful for training due to computational matrix operation optimizations that quickly compute matrix operations. The matrices $Q$, $K$ and $V$ are defined as the matrices where the $i$th rows are vectors $q_{i}$, $k_{i}$, and $v_{i}$ respectively. Then we can represent the attention as$${\begin{aligned}{\text{Attention}}(Q,K,V)={\text{softmax}}\left({\frac {QK^{\mathrm {T} }}{\sqrt {d_{k}}}}\right)V\end{aligned}}$$

where the softmax is applied over each of the rows of the matrix.

Multi-head attention

Multiheaded attention, block diagram.

One set of $\left(W_{Q},W_{K},W_{V}\right)$ matrices is called an attention head, and each layer in a transformer model has multiple attention heads. While each attention head attends to the tokens that are relevant to each token, multiple attention heads allow the model to do this for different definitions of "relevance". In addition, the influence field representing relevance can become progressively dilated in successive layers. Many transformer attention heads encode relevance relations that are meaningful to humans. For example, some attention heads can attend mostly to the next word, while others mainly attend from verbs to their direct objects. The computations for each attention head can be performed in parallel, which allows for fast processing. The outputs for the attention layer are concatenated to pass into the feed-forward neural network layers.

Concretely, let the multiple attention heads be indexed by $i$, then we have$${\text{MultiheadedAttention}}(Q,K,V)={\text{Concat}}_{i\in [\#heads]}({\text{Attention}}(XW_{i}^{Q},XW_{i}^{K},XW_{i}^{V}))W^{O}$$ where the matrix $X$ is the concatenation of word embeddings, and the matrices $W_{i}^{Q},W_{i}^{K},W_{i}^{V}$ are "projection matrices" owned by individual attention head $i$, and $W^{O}$ is a final projection matrix owned by the whole multi-headed attention head.

Masked attention

It may be necessary to cut out attention links between some word-pairs. For example, the decoder, when decoding for the token position $t$, should not have access to the token at position $t 1$. This may be accomplished before the softmax stage by adding a mask matrix $M$ that is $-\infty$ at entries where the attention link must be cut, and $0$ at other places:$${\begin{aligned}{\text{MaskedAttention}}(Q,K,V)={\text{softmax}}\left(M {\frac {QK^{\mathrm {T} }}{\sqrt {d_{k}}}}\right)V\end{aligned}}$$For example, the following mask matrix is used in autoregressive modeling:$$M={\begin{bmatrix}0&-\infty &-\infty &\dots &-\infty \\0&0&-\infty &\dots &-\infty \\0&0&0&\dots &-\infty \\\vdots &\vdots &\vdots &\ddots &\vdots \\0&0&0&\dots &0\end{bmatrix}}$$In words, it means that each token can pay attention to itself, and every token before it, but not any after it.

Encoder

One encoder layer.

An encoder consists of an embedding layer, followed by multiple encoder layers.

Each encoder layer consists of two major components: a self-attention mechanism and a feed-forward layer. It takes an input as a sequence of input vectors, applies the self-attention mechanism, to produce an intermediate sequence of vectors, then applies the feed-forward layer for each vector individually. Schematically, we have:${\begin{aligned}{\text{given input vectors }}&h_{0},h_{1},\dots \\{\text{combine them into a matrix }}H&={\begin{bmatrix}h_{0}\\h_{1}\\\vdots \end{bmatrix}}\\{\text{EncoderLayer}}(H)&={\begin{bmatrix}{\text{FFN}}({\text{MultiheadedAttention}}(H,H,H)_{0})\\{\text{FFN}}({\text{MultiheadedAttention}}(H,H,H)_{1})\\\vdots \end{bmatrix}}\\\end{aligned}}$

where ${\text{FFN}}$ stands for "feed-forward network". We can more succinctly write it as$${\text{EncoderLayer}}(H)={\text{FFN}}({\text{MultiheadedAttention}}(H,H,H))$$with the implicit convention that the ${\text{FFN}}$ is applied to each row of the matrix individually.

The encoder layers are stacked. The first encoder layer takes the sequence of input vectors from the embedding layer, producing a sequence of vectors. This sequence of vectors is processed by the second encoder, and so on. The output from the final encoder layer is then used by the decoder.

As the encoder processes the entire input all at once, every token can attend to every other token (all-to-all attention), so there is no need for causal masking.

Decoder

One decoder layer.

A decoder consists of an embedding layer, followed by multiple decoder layers, followed by an un-embedding layer.

Each decoder consists of three major components: a causally masked self-attention mechanism, a cross-attention mechanism, and a feed-forward neural network. The decoder functions in a similar fashion to the encoder, but an additional attention mechanism is inserted which instead draws relevant information from the encodings generated by the encoders. This mechanism can also be called the encoder-decoder attention.

Like the first encoder, the first decoder takes positional information and embeddings of the output sequence as its input, rather than encodings. The transformer must not use the current or future output to predict an output, so the output sequence must be partially masked to prevent this reverse information flow. This allows for autoregressive text generation. For decoding, all-to-all attention is inappropriate, because a token cannot attend to tokens not yet generated. Thus, the self-attention module in the decoder is causally masked.

In contrast, the cross-attention mechanism attends to the output vectors of the encoder, which is computed before the decoder starts decoding. Consequently, there is no need for masking in the cross-attention mechanism.

Schematically, we have:$${\begin{aligned}H'&={\text{FFN}}({\text{MaskedMultiheadedAttention}}(H,H,H))\\{\text{DecoderLayer}}(H)&={\text{FFN}}({\text{MultiheadedAttention}}(H',H^{E},H^{E}))\end{aligned}}$$where $H^{E}$ is the matrix with rows being the output vectors from the encoder.

The last decoder is followed by a final un-embedding layer, that is, a linear transformation followed by a softmax, to produce the output probabilities over the vocabulary. Then, one of the tokens is sampled according to the probability, and the decoder can be run again to produce the next token, etc, autoregressively generating output text.

Full transformer architecture

(a) One encoder layer and one decoder layer. (b) Two encoder layers and two decoder layers. The sublayers are labelled as well.

Block diagram for the full Transformer architecture.

Schematic object hierarchy for the full Transformer architecture, in object-oriented programming style.

The encoder and decoder layers are as described. Multiple encoder and decoder layers are composed into an encoder-decoder configuration, as illustrated.

The final points of detail are the residual connections and layer normalization (LayerNorm, or LN), which while conceptually unnecessary, are necessary for numerical stability and convergence. That is, the output of each sublayer is $$\mathrm {LayerNorm} (x \mathrm {Sublayer} (x))$$where $\mathrm {Sublayer} (x)$ is the function implemented by the sublayer itself.

Each encoder layer contains two sublayers: the self-attention and the feedforward network. Each decoder layer contains four sublayers: the causally masked self-attention, the feedforward network, the cross-attention, and the feedforward network.

Terminology

The Transformer architecture, being modular, allows variations. Several common variations are described here.

An "encoder-only" Transformer applies the encoder to map an input text into a sequence of vectors that represent the input text. This is usually used for text embedding and representation learning for downstream applications. BERT is encoder-only. They are found to be not significantly better than training an encoder-decoder Transformer, then taking just the encoder.

A "decoder-only" Transformer is not literally decoder-only, since without an encoder, the cross-attention mechanism has nothing to attend to. Thus, the decoder layers in a decoder-only Transformer is composed of just two sublayers: the causally masked self-attention, the feedforward network. This is usually used for text generation and instruction following. The models in the GPT series are decoder-only.

An "encoder-decoder" Transformer is generally the same as the original Transformer, with 2 sublayers per encoder layer and 4 sublayers per decoder layer, etc. They might have minor architectural improvements, such as alternative activation functions, changing the location of normalization, etc. This is also usually used for text generation and instruction following. The models in the T5 series are encoder-decoder.

A "prefixLM" (prefix language model) is a decoder-only architecture, but with prefix masking, which is different from causal masking. Specifically, it has mask of the form$$M_{\text{prefixLM}}={\begin{bmatrix}\mathbf {0} &0,-\infty \\\mathbf {0} &M_{\text{causal}}\end{bmatrix}}$$where the first columns correspond to the "prefix", and the subsequent columns correspond to the autoregressively generated text based on the prefix. They resemble encoder-decoder models, but has less "sparsity". Such models are rarely used, though they are cited as theoretical possibilities and benchmarked comparisons.

Subsequent work

Alternative activation functions

The original transformer uses ReLU activation function. Other activation functions were developed, such as SwiGLU.

Alternative positional encodings

Transformers may use other positional encoding methods than sinusoidal.

RoPE

RoPE (rotary positional embedding), is best explained by considering a list of 2-dimensional vectors $[(x_{1}^{(1)},x_{1}^{(2)}),(x_{2}^{(1)},x_{2}^{(2)}),(x_{3}^{(1)},x_{3}^{(2)}),...]$. Now pick some angle $\theta$. Then RoPE encoding is$${\text{RoPE}}{\big (}x_{m}^{(1)},x_{m}^{(2)},m{\big )}={\begin{pmatrix}\cos m\theta &-\sin m\theta \\\sin m\theta &\cos m\theta \end{pmatrix}}{\begin{pmatrix}x_{m}^{(1)}\\x_{m}^{(2)}\\\end{pmatrix}}={\begin{pmatrix}x_{m}^{(1)}\cos m\theta -x_{m}^{(2)}\sin m\theta \\x_{m}^{(2)}\cos m\theta x_{m}^{(1)}\sin m\theta \\\end{pmatrix}}$$Equivalently, if we write the 2-dimensional vectors as complex numbers $z_{m}:=x_{m}^{(1)} ix_{m}^{(2)}$, then RoPE encoding is just multiplication by an angle:$${\text{RoPE}}{\big (}z_{m},m{\big )}=e^{im\theta }z_{m}$$For a list of $2n$-dimensional vectors, a RoPE encoder is defined by a sequence of angles $\theta ^{(1)},...,\theta ^{(n)}$. Then the RoPE encoding is applied to each pair of coordinates.

The benefit of RoPE is that the dot-product between two vectors depends on their relative location only:

$${\text{RoPE}}{\big (}x,m{\big )}^{T}{\text{RoPE}}{\big (}y,n{\big )}={\text{RoPE}}{\big (}x,m k{\big )}^{T}{\text{RoPE}}{\big (}y,n k{\big )}$$ for any integer $k$.

ALiBi

ALiBi (Attention with Linear Biases) is not a replacement for the positional encoder on the original transformer. Instead, it is an additional positional encoder that is directly plugged into the attention mechanism. Specifically, the ALiBi attention mechanism is$${\begin{aligned}{\text{Attention}}(Q,K,V)={\text{softmax}}\left({\frac {QK^{\mathrm {T} }}{\sqrt {d_{k}}}} sB\right)V\end{aligned}}$$Here, $s$ is a real number ("scalar"), and $B$ is the linear bias matrix defined by$$B={\begin{pmatrix}0&1&2&3&\cdots \\-1&0&1&2&\cdots \\-2&-1&0&1&\cdots \\-3&-2&-1&0&\cdots \\\vdots &\vdots &\vdots &\vdots &\ddots \\\end{pmatrix}}$$in other words, $B_{i,j}=j-i$.

ALiBi allows pretraining on short context windows, then finetuning on longer context windows. Since it is directly plugged into the attention mechanism, it can be combined with any positional encoder that is plugged into the "bottom" of the entire network (which is where the sinusoidal encoder on the original transformer, as well as RoPE and many others, are located).

Relative Position Encodings

Relative Position Encodings is similar to ALiBi, but more generic:$${\begin{aligned}{\text{Attention}}(Q,K,V)={\text{softmax}}\left({\frac {QK^{\mathrm {T} }}{\sqrt {d_{k}}}} B\right)V\end{aligned}}$$where $B$ is a Toeplitz matrix, that is, $B_{i,j}=B_{i',j'}$ whenever $i-j=i'-j'$.

Normalization

Transformer encoder with norm-first and norm-last.

Transformer decoder with norm-first and norm-last.

In 2020, difficulties with converging the original transformer were solved by applying layer normalization before (instead of after) multiheaded attention. This is called pre-LN Transformer.

Efficient implementation

FlashAttention

FlashAttention is an algorithm that implements the transformer attention mechanism efficiently on a GPU. It performs matrix multiplications in blocks, such that each block fits within the cache of a GPU, and by careful management of the blocks it minimizes data copying between GPU caches (as data movement is slow).

An improved version, FlashAttention-2, was developed to cater to the rising demand for language models capable of handling longer context lengths. It offers enhancements in work partitioning and parallelism, enabling it to achieve up to 230 TFLOPs/s on A100 GPUs (FP16/BF16), a 2x speed increase over the original FlashAttention.

Key advancements in FlashAttention-2 include the reduction of non-matmul FLOPs, improved parallelism over the sequence length dimension, better work partitioning between GPU warps, and added support for head dimensions up to 256 and multi-query attention (MQA) and grouped-query attention (GQA).

Benchmarks revealed FlashAttention-2 to be up to 2x faster than FlashAttention and up to 9x faster than a standard attention implementation in PyTorch. Future developments include optimization for new hardware like H100 GPUs and new data types like FP8.

Multi-Query Attention

Multi-Query Attention changes the multiheaded attention mechanism. Whereas normally,

$${\text{MultiheadedAttention}}(Q,K,V)={\text{Concat}}_{i\in [\#heads]}\left({\text{Attention}}(XW_{i}^{Q},XW_{i}^{K},XW_{i}^{V})\right)W^{O}$$with Multi-Query Attention, there is just one $W^{K},W^{V}$, thus:

$${\text{MultiQueryAttention}}(Q,K,V)={\text{Concat}}_{i\in [\#heads]}\left({\text{Attention}}(XW_{i}^{Q},XW^{K},XW^{V})\right)W^{O}$$

This has a neutral effect on model quality and training speed, but increases inference speed.

Caching

When an autoregressive transformer is used for inference, such as generating text, the query vector is different at each step, but the already-computed key and value vectors are always the same. The KV caching method saves the computed key and value vectors at each attention block, so that they are not recomputed at each new token. PagedAttention applies memory paging to KV caching.

If a transformer is used with a baked-in prompt, such as ["You are a customer support agent..."], then the key and value vectors can be computed for the prompt, and saved on disk. The saving in compute is significant when the model is used for many short interactions, such as in online chatbots.

Speculative decoding

Transformers are used in large language models for autoregressive sequence generation: generating a stream of text, one token at a time. However, in most settings, decoding from language models is memory-bound, meaning that we have spare compute power available. Speculative decoding uses this spare compute power by computing several tokens in parallel. Similarly to speculative execution in CPUs, future tokens are computed concurrently, by speculating on the value of previous tokens, and are later discarded if it turns out the speculation was incorrect.

Specifically, consider a transformer model like GPT-3 with a context window size of 512. To generate an entire context window autoregressively with greedy decoding, it must be run for 512 times, each time generating a token $x_{1},x_{2},...,x_{512}$. However, if we had some educated guess for the values of these tokens, we could verify all of them in parallel, in one run of the model, by checking that each $x_{t}$ is indeed the token with the largest log-likelihood in the $t$-th output.

In speculative decoding, a smaller model or some other simple heuristic is used to generate a few speculative tokens that are subsequently verified by the larger model. For example, suppose a small model generated four speculative tokens: ${\tilde {x_{1}}},{\tilde {x_{2}}},{\tilde {x_{3}}},{\tilde {x_{4}}}$. These tokens are run through the larger model, and only ${\tilde {x_{1}}}$ and ${\tilde {x_{2}}}$ are accepted. The same run of the large model already generated a new token $x_{3}$ to replace ${\tilde {x_{3}}}$, and ${\tilde {x_{4}}}$ is completely discarded. The process then repeats (starting from the 4th token) until all tokens are generated.

For non-greedy decoding, similar ideas apply, except the speculative tokens are accepted or rejected stochastically, in a way that guarantees the final output distribution is the same as if speculative decoding was not used.

Sub-quadratic transformers

Training transformer-based architectures can be expensive, especially for long inputs. Alternative architectures include the Reformer (which reduces the computational load from $O(N^{2})$ to $O(N\ln N)$), or models like ETC/BigBird (which can reduce it to $O(N)$) where $N$ is the length of the sequence. This is done using locality-sensitive hashing and reversible layers.

Ordinary transformers require a memory size that is quadratic in the size of the context window. Attention-free transformers reduce this to a linear dependence while still retaining the advantages of a transformer by linking the key to the value.

Long Range Arena (2020) is a standard benchmark for comparing the behavior of transformer architectures over long inputs.

Random Feature Attention (2021) uses Fourier random features:$$\varphi (x)={\frac {1}{\sqrt {D}}}[\cos \langle w_{1},x\rangle ,\sin \langle w_{1},x\rangle ,\cdots \cos \langle w_{D},x\rangle ,\sin \langle w_{D},x\rangle ]^{T}$$where $w_{1},...,w_{D}$ are independent samples from the normal distribution $N(0,\sigma ^{2}I)$. This choice of parameters satisfy $\mathbb {E} [\langle \varphi (x),\varphi (y)\rangle ]=e^{-{\frac {\|x-y\|^{2}}{2\sigma ^{2}}}}$, or $$e^{\langle x,y\rangle /\sigma ^{2}}=\mathbb {E} [\langle e^{\|x\|^{2}/2\sigma ^{2}}\varphi (x),e^{\|y\|^{2}/2\sigma ^{2}}\varphi (y)\rangle ]\approx \langle e^{\|x\|^{2}/2\sigma ^{2}}\varphi (x),e^{\|y\|^{2}/2\sigma ^{2}}\varphi (y)\rangle$$Consequently, the one-headed attention, with one query, can be written as $${\text{Attention}}(q,K,V)={\text{softmax}}\left({\frac {qK^{\mathrm {T} }}{\sqrt {d_{k}}}}\right)V\approx {\frac {\varphi (q)^{T}\sum _{i}e^{\|k_{i}\|^{2}/2\sigma ^{2}}\varphi (k_{i})v_{i}^{T}}{\varphi (q)^{T}\sum _{i}e^{\|k_{i}\|^{2}/2\sigma ^{2}}\varphi (k_{i})}}$$where $\sigma =d_{K}^{1/4}$. Similarly for multiple queries, and for multiheaded attention.

This approximation can be computed in linear time, as we can compute the matrix $\varphi (k_{i})v_{i}^{T}$ first, then multiply it with the query. In essence, we have managed to obtain a more precise version of $${\text{Attention}}(Q,K,V)={\text{softmax}}\left({\frac {QK^{\mathrm {T} }}{\sqrt {d_{k}}}}\right)V\approx Q(K^{T}V/{\sqrt {d_{k}}})$$

Performer (2022) uses the same Random Feature Attention, but $w_{1},...,w_{D}$ are first independently sampled from the normal distribution $N(0,\sigma ^{2}I)$, then they are Gram-Schmidt processed.

Multimodality

Transformers can also be used/adapted for modalities (input or output) beyond just text, usually by finding a way to "tokenize" the modality.

Vision transformers adapt the transformer to computer vision by breaking down input images as a series of patches, turning them into vectors, and treating them like tokens in a standard transformer.

Conformer and later Whisper follow the same pattern for speech recognition, first turning the speech signal into a spectrogram, which is then treated like an image, i.e. broken down into a series of patches, turned into vectors and treated like tokens in a standard transformer.

Perceivers by Andrew Jaegle et al. (2021) can learn from large amounts of heterogeneous data.

Regarding image outputs, Peebles et al introduced a diffusion transformer (DiT) which facilitates use of the transformer architecture for diffusion-based image production. Also, Google released a transformer-centric image generator called "Muse" based on parallel decoding and masked generative transformer technology. (Transformers played a less-central role with prior image-producing technologies, albeit still a significant one.)

See also

• seq2seq – Family of machine learning approaches
• Perceiver – Machine learning algorithm for non-textual data
• Vision transformer – Machine learning algorithm for vision processing
• Large language model – Type of artificial neural network
• BERT (language model) – Language model developed by Google
• GPT-2 – 2019 text-generating language model

Further reading

• Alexander Rush, The Annotated transformer Archived 2021-09-22 at the Wayback Machine, Harvard NLP group, 3 April 2018
• Phuong, Mary; Hutter, Marcus (2022). "Formal Algorithms for Transformers". arXiv:2207.09238 [cs.LG].
• Ferrando, Javier; Sarti, Gabriele; Bisazza, Arianna; Costa-jussà, Marta R. (2024-05-01). "A Primer on the Inner Workings of Transformer-based Language Models". arXiv:2405.00208 [cs.CL].
• Hubert Ramsauer et al. (2020), "Hopfield Networks is All You Need" Archived 2021-09-18 at the Wayback Machine, preprint submitted for ICLR 2021. arXiv:2008.02217; see also authors' blog Archived 2021-09-18 at the Wayback Machine

– Discussion of the effect of a transformer layer as equivalent to a Hopfield update, bringing the input closer to one of the fixed points (representable patterns) of a continuous-valued Hopfield network