The Transformer Architecture
Simplified Technical English edition. Term-based models such as BM25 match strings. They do not capture meaning. This week builds the transformer step by step, so that later weeks can use it for retrieval. The notes cover the vector and matrix maths, WordPiece tokenisation, token, position and segment embeddings, and special tokens. They show self-attention with queries, keys and values, with a calculation by hand. They explain the decoder and its output head, encoder-only and decoder-only models, and self-supervised training. They describe BERT (masked language modelling, next sentence prediction, fine-tuning, dense retrieval and cross-encoder reranking), autoregressive decoders such as GPT, and the uses of LLMs in information retrieval. Dr Shuai Wang, UQ ielab.
1. Why BM25 is not enough
Sources: slides 2–6 · Video 1 (Overview of the transformer) · Lecture transcript (ranking history, term mismatch)
Weeks 1–3 built the classical retrieval stack. Week 1 covered search engine architecture and text processing. Week 2 covered evaluation. Week 3 covered term-based lexical models: TF, TF-IDF, VSM, BIM and BM25. BM25 was the standard model for decades. But BM25 has one basic limit: it can only match exact words. It has no model of the meaning of words or sentences.
1.1 Two classic failure modes
| Failure mode | Example | What BM25 does |
|---|---|---|
| Vocabulary mismatch (false negative) | The query is car. The document says automobile. A second example from the lecture: a query with biomedical and a document about medicine or biology. The concepts are related, but the strings are different. | The texts share zero terms. Thus the score is near zero. The system misses the relevant document. |
| Polysemy (false positive) | The query is where is the closest bank? (a financial institution). The document is about a river bank (a geographic feature). The in-class example: "I want to fix my laptop" and "I want to fix this guy". The same word has different meanings in different sentences. | The word bank is in both texts. Thus the document can get a high rank. The system retrieves an irrelevant document. |
Worked example 1A relevant document BM25 cannot retrieve
The query is how to fix laptop. The document title is "Computer repair guide for damaged notebooks" (slide 5). A person sees that the two texts match. Examine the term overlap after stop-word removal:
| Query term | Does a document term match? | Overlap |
|---|---|---|
fix | repair is a synonym, but it is a different string | ✗ |
laptop | computer and notebooks are related concepts, but they are different strings | ✗ |
The BM25 score from Week 3 is a sum over the query terms that are also in the document. Here, this set is empty:
1.2 What a modern retrieval system needs
| Requirement | Meaning | Example |
|---|---|---|
| Semantic matching | The system understands meaning, and also surface strings | car must match automobile. laptop and computer are related concepts. |
| Context awareness | One word can have different meanings in different contexts. The system must understand the query intent. | Apple in a technology context is a company. apple in a recipe is a fruit. |
1.3 Where we are in the history of ranking
The lecture started with a timeline of rankers. Week 4 is at the "Industrial Revolution" (2017–2018): the start of the transformer. All later parts of the course use the transformer.
| Era | Models | Where in this course |
|---|---|---|
| "Jurassic" (1950s–1990s) | Term-based lexical models (Boolean, TF-IDF, BM25) | Week 3 |
| "Human evolution" (2000s–2010s) | Feature-based reranking models (LTR, online LTR; fusion, RF) | Weeks 7, 11, 12 |
| "Stone Age transitioning" (2010s) | Word2vec, RNN, LSTM | Outside the course |
| "Industrial Revolution!" (2017–2018): we are here | Transformer. Encoder: BERT · Decoder: GPT | Week 4 |
| "New inventions" (2018–2022) | Encoder-based rerankers | Week 8 |
| (same period) | Dense retrievers; learnt sparse models | Weeks 5, 6 |
| "AI age" (2022 onwards) | ChatGPT, Mistral, Claude, Deepseek, Qwen | Parts of Weeks 5, 9, 10 |
| "The future is accelerating" | LLM rerankers and retrievers, LLM-assisted ranking, RAG, search agents | Parts of Weeks 5, 9, 10 |
PracticeA user searches jaguar top speed. BM25 returns a page about the dealerships of the Jaguar car brand. BM25 misses a wildlife page that says "the big cat can sprint at 80 km/h". Name the two failure modes.
2. Maths refresher: vectors, matrices and softmax
Sources: Lecture transcript (linear-layer Q&A, Exercise 3 vector addition, self-attention walkthrough) · background for slides 10–19 and 25–29
A transformer uses a small set of simple operations on vectors and matrices. Neural models can only use numbers. The lecturer said: "we need to represent things in numbers, we need to do number calculations. That's how those neural models are formed." This section explains these operations from the start. The workbook Transformer in a Spreadsheet (sheet "1 Maths") calculates each example of this section with live formulas.
2.1 Vectors, vector addition and the dot product
A vector is an ordered list of numbers, for example . The number of entries is the dimension of the vector (here, 4). In a transformer, a vector represents each token. BERT uses vectors with 768 dimensions. Llama uses vectors with 2,048 dimensions.
- two vectors with the same dimension
- the dimension (number of entries)
- a position index, from 1 to
- entry of the sum vector
- the dot product of and : one number
- entry of each vector
- the shared dimension
- add the products for all positions
Worked example 2Add three vectors, then take a dot product
[CLS] in Exercise 3 (Section 15). The dot product is one number.2.2 Matrices and the transpose
A matrix is a rectangular grid of numbers with rows and columns. Its shape is ("n by m"). The rows are always first. A vector is a matrix with one row (a row vector, ) or one column (a column vector, ). For example, is , and is .
- a matrix with shape
- the transpose of , with shape
- the entry of in row , column
- the row and column indices of the transposed matrix
2.3 Matrix multiplication, step by step
- the left matrix,
- the right matrix,
- the product,
- the entry of in row , column
- the inner dimension: the columns of must equal the rows of
- the index over the inner dimension
Worked example 3Every cell of A × B
Multiply the matrix by the matrix from Section 2.2. The inner dimensions are equal (3 = 3). Thus the result is .
| Cell | Row of A · column of B | Value |
|---|---|---|
| row 1, col 1 | 1×7 + 2×9 + 3×11 = 7 + 18 + 33 | 58 |
| row 1, col 2 | 1×8 + 2×10 + 3×12 = 8 + 20 + 36 | 64 |
| row 2, col 1 | 4×7 + 5×9 + 6×11 = 28 + 45 + 66 | 139 |
| row 2, col 2 | 4×8 + 5×10 + 6×12 = 32 + 50 + 72 | 154 |
The special case that transformers use frequently: vector × matrix
A row vector times a matrix gives a row vector . This operation is a linear layer (also called a projection) in a neural network. It changes a vector of one size into a vector of a different size. It uses a matrix of learned weights. The lecturer wrote this on paper in the Q&A. A token vector with dimensions times a learned matrix gives an output vector with dimensions. When a transformer changes one vector into a different vector, it uses this operation.
Worked example 4A tiny linear layer
Project the vector with the weight matrix . Each output entry is the dot product of the vector with one column.
2.4 Softmax: from scores to probabilities
A transformer must often change a vector of raw scores into probabilities. Raw scores (logits) can be any real numbers. Probabilities are numbers between 0 and 1 with a sum of 1. The softmax function does this change.
- a vector of raw scores (logits),
- the score of item
- the exponential of the score (); always positive
- the sum of the exponentials of all scores
- the probability of item
Worked example 5Softmax of three scores
The scores are .
| Operation | Shapes | Use this week |
|---|---|---|
| Vector + vector (element by element) | same length in, same length out | Add the token, segment and position embeddings (Section 5) |
| Vector · vector (dot product) | two vectors in, one number out | Attention scores (Section 7); cosine similarity (Section 12) |
| Vector × matrix | Q, K, V projections (Section 7); the output linear layer (Section 8) | |
| Matrix × matrix | All attention scores at the same time. This is why GPUs are good for transformers. | |
| Softmax | scores in, probabilities out (same length) | Attention weights (Section 7); next-token probabilities (Section 8) |
PracticeA token vector has 768 dimensions. A learned matrix has the shape 768 × 64. What is the shape of the product? What occurs if you multiply the vector by a 64 × 768 matrix?
PracticeCalculate softmax of and of . What does this show?
3. The transformer: overview and the four steps
Sources: slides 7–19 · Video 1 (Overview of the transformer) · Lecture transcript (four steps, block contents)
In 2017, researchers at Google published the paper "Attention Is All You Need" (Vaswani et al.). This paper gave the transformer architecture. The full diagram of the architecture looks difficult. The lecturer said: "it may look quite complicated, and in fact it was actually complicated … and this is already a simplified version". But the main idea is simple.
3.1 Built for translation, used again for search
The original transformer was for machine translation. An English sentence such as "The cat sat on the mat" goes into an encoder–decoder pipeline. The model gives the same sentence in a different language. The slide uses Chinese: 猫坐在垫子上.
3.2 The four processing steps
Slides 13–19 build the architecture in four steps. These steps are the map for Sections 4 to 8.
⟨start⟩ token and the encoded representations. The video says that the decoder asks: "I'm going to generate; encoder, give me the context."⟨end⟩ token (Section 8). This step-by-step generation makes coherent text.Thus the translation of "The cat sat on the mat" is a loop. In each iteration, the decoder gives a probability distribution over the vocabulary. It selects the token with the highest probability:
| Iteration | Decoder input | Predicted token |
|---|---|---|
| 1 | ⟨start⟩ + encoded vectors | 猫 |
| 2 | ⟨start⟩ 猫 + encoded vectors | 坐在 |
| 3 | ⟨start⟩ 猫 坐在 + encoded vectors | 垫子 |
| 4 | ⟨start⟩ 猫 坐在 垫子 + encoded vectors | 上 |
| 5 | ⟨start⟩ 猫 坐在 垫子 上 + encoded vectors | ⟨end⟩: generation stops |
3.3 What is in each block
The lecture showed a hand-drawn diagram of the architecture. Each encoder or decoder block has these parts, from bottom to top: multi-head self-attention → layer norm → multilayer perceptron (MLP) → layer norm. Arrows go around each part (residual connections). Section 7.4 shows the diagram. Remember two facts:
- The dashed loop means "repeat". The output of a block is a set of vectors with the same shape as the input. Thus the output can go into the block again. "We can repeat this 12 times". BERT has 12 layers because it stacks 12 blocks.
- The decoder has a different end. Its last block goes into a linear layer + softmax. This gives probabilities over the vocabulary, and not vectors. This makes generation possible (Section 8).
PracticeIn the translation loop, which steps occur one time for each sentence? Which steps occur one time for each generated token?
4. Step 1a: tokenisation and WordPiece
Sources: slides 30–33 · Video 4 (Tokenisation) · Lecture transcript (WordPiece, token IDs)
Before the embedding step, the model must break the text into tokens. Tokens are the units that the model reads. Each token has an integer ID in a fixed vocabulary. The lecturer's example: the token my can have ID 3500. The model maps these IDs to vectors. Week 1 tokenised text for an inverted index. Here, the tokeniser serves a neural model with a lookup table of fixed size. Thus the design goals are different.
A simple method splits the text at spaces: "my dog is cute" → [my, dog, is, cute]. Slide 31 gives three problems with this method:
- The vocabulary is very large. English has approximately 500,000 unique words. This needs a very large embedding lookup table. With names, typing errors and new words, the vocabulary is "almost infinite".
- Out-of-vocabulary (OOV) words. New words such as coronavirus (new in 2020), typing errors, or random strings such as
ecerfeferf. The model did not see these words in training. Thus it has no embedding for them. - The model loses morphological relationships. run, running and runner become three unrelated tokens. But these words are clearly related.
The opposite method uses single characters as tokens (only a–z, 26 tokens). This method can represent all text. But characters do not show how people combine letters into units with meaning. Also, the sequences become very long. The best method is between these two methods.
## shows a piece that continues a word.| Tokeniser | Tokens for "[CLS] My dog is cute [SEP] he likes playing [SEP]" |
|---|---|
| Word splitting | [CLS], my, dog, is, cute, [SEP], he, likes, playing, [SEP] |
| WordPiece | [CLS], my, dog, is, cute, [SEP], he, likes, play, ##ing, [SEP] |
ing occurs very frequently in English. A separate token for each -ing word makes the vocabulary too large. Thus the tokeniser keeps play and ##ing as two pieces that it can use again.4.1 How WordPiece training works
[a, b, c, …, z].th, ing, tion, …Worked example 6Subword merges on a toy corpus
The training data (illustrative) has four word types with these counts: low ×5, lower ×2, newest ×6, widest ×3.
l o w, l o w e r, n e w e s t, w i d e s t.| Pair | Where it occurs | Count |
|---|---|---|
e s | newest (6) + widest (3) | 9 (most frequent) |
s t | newest (6) + widest (3) | 9 (tied) |
l o | low (5) + lower (2) | 7 |
o w | low (5) + lower (2) | 7 |
e s → es. The words become n e w es t and w i d es t.es t occurs 9 times. Merge it into est.l o (7) → lo. Then merge lo w (7) → low. Continue in this way. The vocabulary grows by one useful piece at each step: es, est, lo, low, …low + ##est. The two pieces are in the vocabulary. The relation to low and widest stays. There is no OOV failure. This example shows merges by frequency. WordPiece gives each possible merge a score, . This score prefers pairs that occur together more frequently than their parts predict. The method is the same.Worked example 7Tokenising the problem words
| Input word | WordPiece output (illustrative) | Why it works |
|---|---|---|
playing | play + ##ing | The morphology stays: play is also in played and player |
coronavirus | corona + ##virus | The word was new in 2020, but the two pieces were in the vocabulary. Thus there is no OOV. |
ecerfeferf | e + ##cer + ##fe + ##fer + ##f | Random strings and typing errors also break into known pieces. In the worst case, the pieces are single characters. |
PracticeWhy does a WordPiece model never have an out-of-vocabulary error? Why does a word-level model have this error?
5. Step 1b: transformer input embeddings
Sources: slides 24–30 · Video 3 (Input embeddings) · Lecture transcript (three embeddings, "I asked him")
After tokenisation, each token must become a vector. A token ID tells what the token is. It does not tell where the token is. But word order changes meaning. The lecturer's example: "I asked him" and "he asked me" use almost the same words in different positions. Their meanings are very different. Thus transformers combine three types of embeddings (slide 25):
| Embedding | Purpose | Required? |
|---|---|---|
| Token embedding | Changes tokens into vectors. It represents word meaning with a learned lookup table. | Always |
| Position embedding | Teaches the model the word order. Each position (0, 1, 2, …) has a learned vector. | Always |
| Segment embedding | Shows the different parts of the input, for example the query and the document. Each segment has a learned vector. | Optional: only some models, for example BERT |
5.1 Token embeddings
The model tokenises the text. Then it finds each token in a learned embedding table. The table has one learned vector for each vocabulary entry. This vector holds the meaning of the token alone. The idea comes from earlier word-vector models such as word2vec. The example from slide 26 (the vectors are shortened):
| Stage | Value |
|---|---|
| Text | "Information Retrieval" |
| Tokenise | ["information", "retrieval"] |
| Lookup | information → [0.1, 0.3, −0.2, 0.9, …]retrieval → [0.4, −0.1, 0.6, 0.2, …] |
5.2 Position embeddings
Word order is important: "information retrieval" ≠ "retrieval information". Each position in the sequence has a learned embedding vector (slide 27): Pos1 → [0.9, 0.4, 0.2, 0.8, …], Pos2 → [0.2, −0.4, 0.4, 0.2, …].
5.3 Segment embeddings (optional)
Models such as BERT use segment embeddings when the input has more than one sequence. An example is a query and a document in one input (the BERT cross-encoder of Section 12). Special tokens separate the segments. [CLS] is at the start. [SEP] is between the segments and at the end. Each token in a segment gets the embedding of that segment (slide 28):
| Stage | Value |
|---|---|
| Text | Information (query) + Retrieval (document) |
| Tokenised | [CLS] information [SEP] retrieval [SEP] |
| Segments | query segment → [0.6, 0.3, 0.7, −0.9, …]doc segment → [0.5, 0.1, 0.8, 0.2, …] |
5.4 Putting it together
- the position of the token in the sequence (0, 1, 2, …)
- the token at position
- the segment of this token (A or B)
- the learned vector for this token, from the token lookup table
- the learned vector for position
- the learned vector for segment (models without segments do not use it)
- the final input vector for position ; it has the same dimension as each part (768 in BERT-base)
[CLS] my dog is cute [SEP] he likes play ##ing [SEP]. The tokens up to the first [SEP] get segment A. The tokens after it get segment B. The positions are 0, 1, 2, … Thus the model knows the position of each token.For a visual version of this addition, column by column, see the explainer How [CLS] becomes [2, 1, 1, 2]. Also see sheet "2 Embeddings" of the Transformer in a Spreadsheet workbook.
Worked example 8One token's final input embedding
Use the token likes from the diagram. The token is likes, the position is 7, and the segment is B. The embeddings have 4 dimensions (illustrative values):
| Component | Vector |
|---|---|
[0.2, −0.1, 0.5, 0.3] | |
[0.0, 0.3, −0.2, 0.1] | |
[0.1, 0.1, 0.0, −0.2] |
Add element by element:
PracticeIn "the bank by the river bank", the token bank is at positions 1 and 5 (one segment). Do the two tokens get the same input vector?
6. Special tokens
Sources: slides 34–39 · Video 5 (Special tokens) · Lecture transcript (CLS for retrieval, PAD for batching, "[FRANKENSTEIN]")
6.1 Encoder special tokens: [CLS], [SEP], [MASK], [PAD]
| Token | Name | Role |
|---|---|---|
[CLS] | Classification | Always at the start of the input. Sentence-level tasks (classification, ranking) use its output vector. The lecturer said: its name is the classification token, but you can fine-tune the model to use this token for any task. Retriever models use it frequently to represent a full sentence as one vector. |
[SEP] | Separator | Separates different sequences. It helps the model to find the parts of the input (for example the query and the document). |
[MASK] | Masked token | Used in training. The model learns to predict the masked words (Sections 10 and 11). |
[PAD] | Padding | Makes all sequences the same length, so that the model can batch them. The GPU processes a batch in parallel only if all sequences have one length. Attention ignores [PAD]. |
Worked example 9A search-task input, token by token
Input: [PAD] [CLS] laptop repair [SEP] Computer maintenance guide for damaged notebooks [SEP]
| Piece | What it does here |
|---|---|
[PAD] | Keeps the same length when the model batches this input with longer inputs. Attention ignores it. |
[CLS] | Shows a classification or ranking task. Its final vector holds the relevance decision. |
laptop repair | Segment A: the query |
[SEP] | Separates the query from the document |
Computer maintenance guide … | Segment B: the document |
[SEP] | Shows the end of the input sequence |
[CLS] vector.6.2 Decoder special tokens: [BOS], [EOS]
| Token | Name | Role |
|---|---|---|
[BOS] | Beginning of sequence | Shows the start of generation. It is the first token that goes into the decoder during inference. |
[EOS] | End of sequence | Shows the end of generation. It tells the decoder when to stop. It controls the length of the output. How does ChatGPT know that its answer is complete? It generates its EOS token. |
Worked example 10BOS and EOS in a generation task
| Part | Tokens |
|---|---|
| Prompt | <BOS> The cat sits on |
| Generated | the mat <EOS> (the model stops here) |
<BOS> shows the start of the prompt. When the model generates <EOS>, the generation is complete. With this simple method, decoder models give output with a correct length and stop at the correct point.[FRANKENSTEIN]. If you train the model with it, the model can learn its meaning.PracticeA batch has an input with 6 tokens and an input with 10 tokens. What does [PAD] do here? Why must attention ignore it?
7. Step 2: self-attention and the encoder
Sources: slides 10–15, 19 · Video 1 (Overview of the transformer) · Lecture transcript (self-attention walkthrough, symmetry question, "fat and tall", Exercise 1)
7.1 The intuition: the attention matrix
For the sentence "The cat sat on the mat", self-attention makes a matrix. Each cell shows how much one word (row) attends to a different word (column). For example, cat gives high attention to sat, because they are subject and verb.
Where do these numbers come from? The in-class lecture gave five steps ("Self-attention: what is it really doing?"). If you know little about neural networks, read the five steps slowly. Each operation is in Section 2. Sheet "3 Attention" of the Transformer in a Spreadsheet workbook does the five steps with live formulas. With the Transformer by Hand worksheet, you calculate one block yourself.
7.2 The five steps of self-attention
Step 1: one input vector for each token
Each token has one input vector . This vector is the sum of its token, segment and position embeddings from Section 5. For the toy sentence the · cat · sat: .
Step 2: project each input vector into q, k and v
The model has three learned weight matrices: , and . Training learns them, as it learns all other weights. The model multiplies the input vector of each token by each matrix. This is the vector × matrix operation of Section 2.3:
- the input vector of token (a row vector)
- learned projection matrices, each ( can have a different output size )
- the query vector of token : what the token looks for
- the key vector of token : what the token offers for matching
- the value vector of token : the content that the token gives
- the input dimension and the query/key dimension
Step 3: what are Q, K and V? (the lecturer's intuitions)
| Vector | Slide definition | Lecturer's words |
|---|---|---|
| Q: query | what this token looks for | "What am I to myself; what am I really looking for?" |
| K: key | what this token offers, what the model can match | "What am I to others?" |
| V: value | the information or content that this token carries | "In the end, how do I represent myself?" |
Step 4: calculate the attention weights
How much must token attend to token ? Calculate the dot product of and . This shows how well "what I look for" matches "what you offer". Scale the result. Then apply softmax to each row. Thus the weights of each token have a sum of 1.
- the dot product of the query of token and the key of token (the raw score)
- the dimension of the query and key vectors
- the scaled score of token for token
- the number of tokens in the sequence
- the index over all tokens in the softmax denominator
- the attention weight: the share of the attention of token that goes to token
- all queries and all keys as rows ()
- the attention weight matrix; row has
Step 5: make the new vector for each token
The new, contextual representation of each token is the weighted sum of the value vectors of all tokens. The attention weights are the weights. The sum includes the value vector of the token itself.
- the new output vector of token
- the attention weight of token on token (from Step 4)
- the value vector of token
- the index over all tokens, also
The five steps together, in matrix form, give the famous formula from Attention Is All You Need:
- the query, key and value matrices (each with rows)
- the transpose of ; thus is
- the scaling factor from the key dimension
- applied to each row separately
- the output matrix with rows; row is
7.3 Self-attention by hand
Worked example 11Self-attention for the · cat · sat, every number calculated
The toy sentence (illustrative) has three tokens. The input vectors have 2 dimensions (): . We select the projection matrices by hand. A real model learns them.
| Token | |||
|---|---|---|---|
the | [1, 0] | [1, 1] | [1, 1] |
cat | [2, 2] | [0, 4] | [0, 2] |
sat | [2, 1] | [1, 3] | [1, 2] |
| k(the) = [1,1] | k(cat) = [0,4] | k(sat) = [1,3] | |
|---|---|---|---|
| q(the) = [1,0] | 1 | 0 | 1 |
| q(cat) = [2,2] | 4 | 8 | 8 |
| q(sat) = [2,1] | 3 | 4 | 5 |
7.4 What makes a real transformer "fat" and "tall"
The five steps describe one attention head in one layer. Real transformers make this larger in two directions. The lecturer used these words:
| Direction | What it means | Effect |
|---|---|---|
| "Fat": multi-head attention | In one layer, the model has many independent copies of . Each copy is an attention head with its own learned weights. Each head calculates its own attention pattern in parallel. For example, one head can find subject–verb links, and a different head can find adjacent words. The model combines their outputs. | Each layer becomes wider |
| "Tall": stacked layers | Each token goes in as a vector and comes out as a vector with the same dimension. Thus the model can apply the full block again: output of layer 1 → input of layer 2 → … BERT-base has 12 layers. Each layer makes the representations better. | The model becomes deeper |
Two more parts follow the attention step in each block:
- Multilayer perceptron (MLP, feed-forward layer): a simple neural network for each token vector. It projects the vector up to a higher dimension and then down again. It uses two vector × matrix multiplications with a non-linear function between them. Much of the "thinking capacity" of the model is here.
- Layer norm and residual connections: these parts keep the numbers stable during training. They let each layer add to the results of the earlier layers. Thus the earlier information stays.
The explainer The encoder backbone, and why it cannot generate shows how a stack of these layers changes one vector for each token. Sheet "4 Multi-Head" of the spreadsheet workbook shows two heads side by side.
7.5 What the encoder produces, and why GPUs are good for it
After the last layer, the encoder gives one encoded vector for each input token. The count and the dimension are the same as the input. Each vector now represents "the semantics of the word with respect to the sentence". For one vector for the full sentence, there are two standard methods. Use the output vector of the [CLS] token (train the model to put a summary of the sentence there). Or use the mean of all token vectors (mean pooling). Retrieval uses the two methods frequently (Section 12 and Week 5).
- the final output vector of the encoder for token
- the number of tokens in the mean (without padding tokens)
- the final output vector of the [CLS] token
- one vector for the full text, with the same dimension as each
Worked example 12A sentence vector by mean pooling
Use the three output vectors from Worked example 11: , , .
PracticeIn Worked example 11, calculate the raw score of sat for the, and the raw score of the for sat. Are the two scores equal?
PracticeWhy can BERT-base repeat the same block 12 times? What breaks if the MLP gives a vector with a different size?
8. Steps 3–4: the decoder and generation
Sources: slides 16–19, 23 · Video 1 (Overview of the transformer) · Video 7 (Autoregressive decoders) · Lecture transcript (decoder, linear-layer Q&A, sampling)
The lecturer said: "Don't worry too much about the decoder; it's almost the same." The decoder uses the same parts as the encoder. It tokenises the input (your prompt), makes input embeddings, and uses stacked blocks of self-attention + MLP. Two things are different. The two differences exist because the task of the decoder is generation.
8.1 Difference 1: masked (causal) attention
In the decoder, each token can attend only to itself and earlier tokens. During generation, the future tokens do not exist yet. When the model generates the first word of its answer, there is no word after it. The lecturer said: "It's not that we don't want it; we don't need it. How could it look at future tokens? They're not yet generated." In practice, the model sets the scores for future positions to before the softmax. This removes the upper-right triangle of the attention matrix (compare the two grids in Section 9).
- the position of the token that attends
- the position that gets attention
- “itself or earlier”: position is position or a position before it
- the usual scaled score from Section 7
- a score that becomes after the exponential
- the masked score
- the attention weight (0 for each future position)
Worked example 13The causal mask on the · cat · sat
Use the scaled scores from Worked example 11. Mask each future position.
8.2 Difference 2: the output head, from vector to next-token probabilities
The encoder ends with vectors. The decoder must end with a selection of the next token. The lecturer wrote the full chain on paper for a student. It is one matrix multiplication and one softmax:
- the final-layer hidden vector of the last token (dimension )
- the learned output projection ("linear layer"),
- the vocabulary size (approximately 30,000 for BERT-style vocabularies)
- the logits: one raw score for each vocabulary token
- the next-token probability distribution
- the probability of vocabulary token
Worked example 14The output head with numbers
The vocabulary (illustrative) has 5 tokens (the, a, cat, pie, …). The logits are .
the (0.56). With greedy decoding, it is the next token. Softmax has no problem with the negative logit of pie: it becomes a small probability (0.03).8.3 The generation loop and sampling
One pass through the model gives one token. To generate text, the model adds the predicted token to the input and does the full process again ("generate, put it back, generate, put it back"). The loop stops when the model gives the [EOS] token. Section 13 calls this autoregressive generation.
PracticeA model has a hidden size of 2,048 and a vocabulary of 32,000 tokens. What is the shape of the output projection? How many numbers does it give at each step? Why does the model use only the vector of the last token?
9. Encoder vs decoder
Sources: slides 20–23 · Video 2 (Encoder vs decoder) · Lecture transcript (using the parts separately, Exercise 1)
In the original transformer, the encoder and the decoder work together as one system. The encoder changes text into vector representations. The decoder uses these vectors and the ⟨start⟩ token to predict the next token. Slides 20–21 give the important point: you can use the two parts separately. The lecturer asked two questions: "If my task is not generation, do I need the decoder? If my task is only generation, do I need the encoder?" The answer "no" to each question gave some of the most important advances in AI:
- BERT uses only the encoder. It is an encoder-only model with a very large effect on search and language understanding (Section 11).
- Modern large language models (GPT, ChatGPT, Claude, …) use only the decoder (Section 13).
| Encoder: "understand and represent" | Decoder: "generate and predict" | |
|---|---|---|
| Purpose | Receives input text. Makes a dense vector representation. Understands context and relationships. | Receives input (optionally with encoded representations). Generates output step by step. Predicts next-token probabilities. |
| Attention | Bidirectional: each token can see all tokens before and after it. The encoder sees the full sentence at the same time. | Masked (unidirectional): a token sees only earlier words |
| Example | Input How to fix laptop? → output dense vector [0.23, −0.45, 0.67, …]. The vector holds the meaning of the sentence. | Input UQ is (+ encoded vectors, optional) → output the → University → of → Queensland → … |
| Applications | Search and classification with embeddings. These tasks need the full meaning of the text. | Translation, text generation, and all modern LLM use |
PracticeYou must embed ten million product descriptions for semantic search. You must also write product summaries. Which part of the transformer is correct for each task? Which attention pattern does each part use?
10. How transformers are trained
Sources: slides 41, 44, 53 · Video 6 (BERT) · Video 7 (Autoregressive decoders) · Lecture transcript (training, data and lawsuits)
A new model with random weights knows nothing. The embedding tables, the matrices in each head of each layer, the MLP weights and the output projection all start as random numbers. Training changes all these weights until the model "understands" language. The practical problem: training needs a very large amount of training signal without human labels. BERT and GPT use the same solution: self-supervised learning. Use the full web as training data, and let the text give the labels.
| Objective | How it works | Lecture example |
|---|---|---|
| Masked language modelling (encoder, BERT-style) | Take web text. Replace some tokens at random with [MASK]. Train the model to predict the masked tokens from all the context around them. The context is on the two sides, thus bidirectional attention is important. Compare the prediction with the real token and measure the error (loss). Change the weights a little to make the loss smaller. Repeat over the full web. Section 11 gives the details and a loss calculation. | "The bank [MASK] loans to clients" → predict makes |
| Next-token prediction (decoder, GPT-style) | This is simpler. Give the first token: can the model predict the second token? Give the first two tokens: predict the third token. Give the first three tokens: predict the fourth token. Each position of each sentence on the web becomes a training example. | If you train a model on your chat history, it learns your next reply ("if you've written bullshit a thousand times, it knows the next token is bullshit"). Nobody teaches grammar to the model. It learns how people write because it predicts their text, billions of times. |
Worked example 15One sentence, two training signals
Use the sentence UQ is the University of Queensland. It has 6 tokens.
UQ → is, UQ is → the, UQ is the → University, … University → of, … of → Queensland. One sentence gives 5 labelled examples.UQ is the [MASK] of Queensland. It predicts University from the two sides.PracticeWhy is next-token prediction a bad pre-training objective for an encoder with bidirectional attention?
11. BERT
Sources: slides 40–48, 50 · Video 6 (BERT) · Lecture transcript (MLM masking rate, scaling)
11.1 The two-stage training approach
11.2 Ingredients and input processing
The success of BERT comes from three pre-training ingredients (slide 42). (1) An encoder-only transformer architecture with many parameters. (2) Very large amounts of text data (Wikipedia, web pages). (3) Very large computing power (cloud TPU pods were necessary at the time).
The input starts with the embedding layers of Section 5: token + segment + position embeddings, added together. Then the input goes through many transformer encoder layers with self-attention (Section 7). The output is a sequence of vector representations, one for each input token. Each token attends to each other token.
11.3 Pre-training task 1: masked language modelling (MLM)
The model masks approximately 15% of the input tokens at random. It learns to predict them from all the context around them (in the two directions). The BERT paper and slide 50 use 15%. In class, the lecturer said 30% from memory. The exact fraction is a design choice. The principle is the important part. The loss measures how well BERT predicts the masked tokens:
- the set of masked positions in the sentence
- the true (original) token at masked position
- the sentence with the masked positions hidden: all the visible context
- the probability that BERT gives to the correct token at position , for this context
- natural logarithm
- the masked language modelling loss for this sentence (lower is better)
Worked example 16MLM loss on the lecture's sentence
The original sentence (slide 47): "The Mongol invasion of Europe in the 13th century was the conquest of much of Europe by the Mongol Empire." BERT masks century and Europe: "… in the 13th [MASK] was the conquest of much of [MASK] by the Mongol Empire." With bidirectional context, it predicts:
| Masked position | Prediction | Confidence | Loss contribution |
|---|---|---|---|
… 13th [MASK] was … | century | 0.94 | |
… much of [MASK] by … | Europe | 0.97 |
11.4 Pre-training task 2: next sentence prediction (NSP)
BERT also trains on pairs of sentences. It learns to predict if the second sentence follows naturally after the first sentence. This is a binary classification on the [CLS] output. It teaches relationships between sentences. These relationships are important for question answering and document ranking.
| Sentence A | Sentence B | Label |
|---|---|---|
| My dog is cute. | He likes playing. | IsNext |
| My dog is cute. | The stock market fell 2% today. | NotNext |
11.5 Model size is important
| Model | Parameters | Configuration |
|---|---|---|
| BERT-base | 110 million | 12 layers |
| BERT-large | 345 million | more layers, larger hidden dimension, more attention heads |
BERT-large is always better than BERT-base, also for ranking and for small tasks. Also, more training steps give higher accuracy (slide 46). This was an early sign of the scaling laws of today's large language models. The lecturer said that models with a billion parameters are now small, and the largest models have trillions. But the industry must now balance size against efficiency ("a user says hello, and answering hi costs me $20; I can't do that").
11.6 Fine-tuning to different tasks
- Classification of one input: give one text. The
[CLS]output is the summary of the sentence for the classifier. - Tasks with two inputs (question answering, ranking): process the two inputs together in different segments (query
[SEP]document). - Sequence labelling: each token gets a prediction from its own output vector.
11.7 The limits of BERT
| Limit | Detail (slide 50) |
|---|---|
| Input length limit | The maximum input is 512 tokens, with the special tokens. BERT cannot process long documents directly. They need chunking or truncation, and some information is lost. |
| No autoregressive generation | MLM pre-training masks approximately 15% of the tokens at random and predicts them with bidirectional context. BERT does not train on next-token prediction. Thus it cannot make coherent sequences token by token, as GPT does. (Section 15 gives a modern solution: diffusion language models generate text with encoder backbones.) |
Decoder-only models such as GPT solve the second limit (Section 13).
PracticeDuring training, BERT gives the correct masked word a probability of 0.5 at one position and 0.05 at a different position. Calculate each loss contribution. Which position has more effect on learning?
12. BERT in information retrieval
Sources: slide 49 · Video 6 (BERT) · Lecture transcript (ranking challenge: [CLS] or mean vectors, ColBERT-style interaction)
IR uses BERT in two main ways (slide 49). The two ways make search quality much better. Week 5 covers dense retrievers. Week 8 covers BERT-based rerankers.
| Dense retriever (bi-encoder) | Reranker (cross-encoder) | |
|---|---|---|
| Input | The model encodes the query and the document separately | [CLS] query [SEP] document [SEP], read together (segments A and B, Section 5) |
| Score | Cosine similarity or dot product of and | Classification score from the [CLS] head |
| Cost | The model calculates the document vectors one time, at indexing time. Search is fast. | One BERT run for each (query, document) pair. Use it to rerank a short list of candidates. |
| Accuracy | Good | Higher: each query token attends to each document token |
| Course | Week 5 | Week 8 (this is the monoBERT style of reranker) |
From the in-class discussion: for the dense retriever, the sentence vector usually comes from the [CLS] output vector or from the mean of all token output vectors (Section 7.5). The relevance score is a cosine similarity or dot product. The lecturer said: "either way, you can have a score, as long as you train it." A student gave a more advanced method. Keep one vector for each token of the query and of the document. Compare each query token with each document token (a full similarity matrix, as in attention). Then combine the similarities into a final score. This is the idea of ColBERT-style late interaction. It is more accurate than one vector, but it must store many vectors for each document (later weeks).
- the query vector and the document vector from BERT
- their dot product (Section 2.1)
- entry of each vector
- the lengths of the vectors (Euclidean norms)
- the cosine similarity, between −1 and 1
Worked example 17Dense retrieval fixes the fix-laptop query
BERT encodes the query and two documents into vectors with 3 dimensions (illustrative values):
| Text | Vector |
|---|---|
| = V("how to fix laptop") | [0.90, 0.30, 0.10] |
| = V("computer repair guide for damaged notebooks") | [0.80, 0.40, 0.00] |
| = V("river bank fishing spots") | [0.10, 0.20, 0.95] |
PracticeA collection has 8 million passages. The system must answer queries in less than 100 ms. Where does each BERT method go in the pipeline, and why?
13. Autoregressive decoders and GPT
Sources: slides 51–54 · Video 7 (Autoregressive decoders)
- the current time step (the position that the model generates)
- all tokens before step : the prompt and all tokens that the model generated before
- the vocabulary
- one candidate token from the vocabulary. The arg max tries each token.
- the probability from the model that is the next token (the softmax output of Section 8)
- "the that gives the largest value"
- the token that the model selects at step (hat = predicted)
- a full sequence of tokens
- the length of the sequence
- the probability of token for the tokens before it
- multiply the step probabilities together
Worked example 18Autoregressive generation of "UQ is …"
The prompt is UQ is. At each step, the decoder gives a distribution over the vocabulary and selects the top token (greedy). The probabilities are illustrative:
| Step | Input sequence | Top candidates | Selected |
|---|---|---|---|
| 1 | UQ is | the 0.62 · a 0.15 · ranked 0.08 · … | the |
| 2 | UQ is the | University 0.87 · best 0.05 · … | University |
| 3 | UQ is the University | of 0.95 · in 0.02 · … | of |
| 4 | UQ is the University of | Queensland 0.98 · Qld 0.01 · … | Queensland |
The probability of the generated text is the product of the step probabilities (chain rule):
UQ is the. It never sees a future token (masked attention). BERT sees the full sentence at the same time, but it cannot generate it.13.1 GPT-3 and ChatGPT
OpenAI released GPT-3 in June 2020 (slide 52). It is a decoder-only transformer (no encoder) with 175 billion parameters. This is approximately 1,000× larger than BERT-base. It uses unidirectional attention: only earlier tokens are visible. The first release had a context window of 2,048 tokens. The example on the slide: the prompt "Recite the first law of robotics" → GPT-3 generates, token by token, "A robot may not injure a human …". In 2022, OpenAI released ChatGPT with a very similar architecture. ChatGPT started the current AI revolution. It made generative models (LLMs) very popular in the world.
13.2 Training: next-token prediction gives in-context learning
GPT models train with simple next-token prediction (Section 10) on very large amounts of text. This simple objective gives in-context learning. The model learns to do tasks only from examples in the prompt: arithmetic, spelling correction, translation. It needs no training for the specific task (Brown et al., 2020, "Language models are few-shot learners", slide 53).
Worked example 19In-context learning with a few-shot prompt
| Role | Text |
|---|---|
| Prompt | sea otter → loutre de merpeppermint → menthe poivréecheese → |
| The model continues | fromage |
fromage. Nobody fine-tuned it for French translation. It finds the pattern in the prompt and continues it. This is only next-token prediction with a good context.13.3 Instruction fine-tuning
To make these models more useful, they get instruction fine-tuning (slide 54). They train on pairs of instructions and responses, for example "Summarize this text: [text]" → [summary]. This teaches the model to follow human instructions and to do tasks on request. It changes a raw language model into a system such as ChatGPT, which can have conversations and help with tasks.
PracticeIn Worked example 18, the model samples "a" at step 1 (probability 0.15). Can the final text still be "the University of Queensland"? What does this show about sampling?
UQ is a. Thus the text changes (for example "a university in Brisbane"). Each token stays fixed after the model generates it. One random selection at the start changes all later probabilities. Thus sampled answers are different in different runs.14. LLMs in information retrieval
Sources: slides 55–56 · Video 7 (Autoregressive decoders) · Lecture transcript (ranking challenge)
Large language models change IR in three main directions (slide 55):
| Direction | Idea | Examples |
|---|---|---|
| Prompting LLMs for retrieval and ranking | Use the LLM directly as a rewriter, retriever, reranker or reader in the search pipeline (LLM-enhanced IR) | RankGPT, PromptReps |
| Retrieval-augmented generation (RAG) | Retrieve relevant documents. Then give them to the LLM as context, so that it generates better and more reliable responses (IR for LLMs). | Self-RAG, BlendFilter |
| Model-based IR | The LLM is the indexer and also the retriever: generative (document) retrieval | DSI (Differentiable Search Index), GR |
The in-class challenge gave more methods for LLM-based ranking: pointwise, pairwise and listwise prompting, query and document expansion, and embeddings from decoders. Section 15 explains all of them.
14.1 The evolutionary trend
PracticeA company chatbot answers questions about internal policy documents. It must cite the documents. Which of the three directions is this? What does the IR part give?
15. In-class exercises, worked
Sources: Lecture transcript (in-class Exercises 1–3 and the challenge questions) · slides 20–23, 29, 49, 55
The lecture ended with three exercises and a challenge. Try each exercise before you open the answer. The lecturer said: "thinking about how you would use these components is exactly how the techniques later in this course were invented."
15.1 Exercise 1: transformer components
PracticeWhich statement about the transformer architecture is correct? A. The encoder uses masked attention to prevent looking at future tokens. B. The decoder uses bidirectional attention to see all tokens in the sequence. C. The encoder processes input tokens in parallel using bidirectional attention. D. Both encoder and decoder use the same type of attention mechanism.
| Option | Result | Why |
|---|---|---|
| A | incorrect | It is the opposite. The encoder looks at future tokens (bidirectional). The decoder uses the mask. Each encoder token must see all tokens near it. For example, the token loans shows the meaning of an earlier bank. |
| B | incorrect | This is also the opposite. The encoder uses bidirectional attention. The decoder only generates. During generation, future tokens do not exist yet. |
| C | correct | The encoder sees the full sequence at the same time (bidirectional). It processes all tokens in parallel. Attention is batched matrix multiplication, and GPUs make this fast. (CPUs calculate in sequence. GPUs process all values at the same time. Thus transformers made GPUs successful, and GPUs made transformers successful.) |
| D | incorrect | They use different attention. The encoder is bidirectional (all positions, earlier and later). The decoder is masked or causal (earlier positions only). |
15.2 Exercise 2: BERT vs GPT
PracticeWhat is the main architectural difference between BERT and GPT? A. BERT uses only the decoder part, while GPT uses only the encoder part. B. BERT uses only the encoder part, while GPT uses only the decoder part. C. BERT has more layers than GPT. D. BERT uses different tokenization techniques than GPT.
Why C is incorrect (and interesting): in reality, BERT has fewer layers than modern GPT models. But this is a choice of size and not a part of the architecture. You can build an encoder with more layers than a GPT. Generation models are usually much bigger, because generation is a more difficult task than representation. The 110M parameters of BERT-base were "sort of already enough" for classification and ranking. But small generative models (GPT-2 era, T5 era) did not work well. Good generation came only with very large parameter counts (the scaling law). Thus the name is large language models.
Why D is incorrect (and interesting): the two models use the same technique: trained subword tokenisation (WordPiece-style, Section 4). The slide asks a second question: "do BERT and GPT-2 have a different number of overall tokens?" The answer is yes. They trained on different collections with different target vocabulary sizes and goals. BERT has approximately 30K pieces. Some newer models have hundreds of thousands. Thus each model tokenises the same text differently. The technique is the same, but the vocabulary is different.
15.3 Exercise 3: constructing input embeddings
PracticeThe sentence "The cat sits" is tokenised as [CLS] the cat sits [SEP], with embedding dimension 4. Token embeddings: [CLS] = [1,0,0,1], the = [0,1,1,0], cat = [1,1,0,0], sits = [0,0,1,1], [SEP] = [1,0,1,0]. Segment embeddings: A = [0,1,0,1], B = [0,0,0,2] (all special tokens belong to segment A). Position embeddings: pos₀ = [1,0,1,0], pos₁ = [0,1,0,1], pos₂ = [1,1,0,0], pos₃ = [0,0,1,1], pos₄ = [1,0,0,1]. Construct the complete input embedding for every token.
Step 2: add the three vectors for each token (element-by-element addition, Section 2.1; one vector for each token):
| Token | Token E | + Segment E (A) | + Position E | = Final input embedding |
|---|---|---|---|---|
[CLS] (pos 0) | [1,0,0,1] | [0,1,0,1] | [1,0,1,0] | [2,1,1,2] |
the (pos 1) | [0,1,1,0] | [0,1,0,1] | [0,1,0,1] | [0,3,1,2] |
cat (pos 2) | [1,1,0,0] | [0,1,0,1] | [1,1,0,0] | [2,3,0,1] |
sits (pos 3) | [0,0,1,1] | [0,1,0,1] | [0,0,1,1] | [0,1,2,3] |
[SEP] (pos 4) | [1,0,1,0] | [0,1,0,1] | [1,0,0,1] | [2,1,1,2] |
Sample calculation for [CLS]: .
Common traps (the two traps occurred in class). (1) Students put the tokens after [SEP] into segment B. This is incorrect here, because [SEP] only ends the single sentence. (2) Students put the three embeddings into a 3×4 matrix. This is incorrect, because the model adds them into one vector with 4 dimensions for each token. The five final vectors, as rows, make the 5×4 input matrix of the encoder. Real models use 768 or more dimensions. The practical of this week calculates them in the same way. The explainer How [CLS] becomes [2, 1, 1, 2] shows the [CLS] row step by step.
15.4 Challenge: encoders and decoders for ranking, encoders for generation
PracticeQ1: How can we use encoders or decoders for ranking? Name at least two ways.
With encoders
| Method | How it works | Related week |
|---|---|---|
| Single-vector dense retrieval (bi-encoder) | Encode the query into one vector ([CLS] or the mean of the token embeddings). Encode each document into one vector. The score is the cosine similarity or the dot product. The model encodes the documents offline at indexing time. Thus search is fast. | Week 5 |
| Multi-vector, late interaction (ColBERT-style) | Keep one vector for each token of the query and of the document. Calculate the similarity of each query token with each document token (a full similarity matrix). Combine the similarities into a final score. This is more precise, but it needs more storage. | Later weeks |
| Cross-encoder reranking | Join query [SEP] document, put it through BERT, and read a relevance score from the [CLS] head. This is the most accurate method, but it needs one model run for each (q, d) pair. | Week 8 |
With decoders (LLMs)
| Method | How it works |
|---|---|
| Prompted ranking (RankGPT-style) | Ask the LLM. Pointwise: "here is a query and a document; give me a relevance score." Pairwise: "document A or document B: which is more relevant?" Listwise: "here are 10 documents; give me an order." (The lecturer: "you may be using this every single day; why can't people think of the easy way?") |
| Query expansion (query2doc-style) | The LLM generates a pseudo-document or an answer for the query (for example, ask Gemini and use its answer). Then a lexical ranker uses this longer text. The generated text contains car, cars, automobile, Tesla, BMW …. Thus it solves the vocabulary mismatch of Section 1. |
| Document expansion (doc2query-style) | This is the opposite method, and it is better for latency. At indexing time, add to each document the queries or terms that it can answer. There is no cost at query time, because the system does this work when it builds the index (as in the BM25 indexing of the Week 3 practical). |
| Decoder hidden states as embeddings | "Every time you see a vector, you can do computations." Stop the model before the output linear layer. Use the last hidden vector as a dense representation for similarity search. Thus decoders can also make embeddings. |
| Output probabilities as representations (PromptReps-style) | Use the next-token probability distribution over the full vocabulary as a representation of the document. This representation is sparse, similar to term statistics. A document about this course gives high probability to information, retrieval, web, search, ranking, University of Queensland …. Rank the documents by a comparison of these distributions. |
PracticeQ2 (challenge): Can we use an encoder backbone for generation?
- What did the encoder learn? Masked language modelling: it predicts [MASK] tokens. This is also a type of token prediction.
- Thus take the prompt and add a batch of [MASK] tokens (for example, 256) at the end. Run the encoder one time. Read the predictions for all masked positions at the same time. An autoregressive decoder predicts one token at a time.
- The problem: the model predicts each masked token independently. Thus the output can be incoherent.
- The solution comes from image diffusion models: iterative refinement. An image model removes noise step by step. In the same way, run the prediction a few times (3–5 passes). Each pass makes the text more coherent.
- The advantage: 256 tokens need approximately 5 passes and not 256 sequential passes. GPUs process parallel tokens easily. Thus generation becomes much faster.
PracticeA student asked after class: which is cheaper for users, document expansion or query expansion with an LLM? Why?
16. Summary and study checklist
16.1 BERT vs GPT at a glance
| BERT (encoder-only) | GPT-3 and LLMs (decoder-only) | |
|---|---|---|
| Released | October 2018 (Google) | June 2020 (OpenAI); ChatGPT 2022 |
| Architecture | Transformer encoder, 12 layers (base) | Transformer decoder |
| Parameters | 110M (base) / 345M (large) | 175B (approximately 1,000× BERT-base) |
| Attention | Bidirectional: sees the full sentence | Unidirectional (masked): sees only the past |
| Pre-training | Masked language modelling (approximately 15% masked) + next sentence prediction | Next-token prediction |
| Output | One contextual vector for each token ([CLS] or mean pooling for the sentence) | Probability distribution over the vocabulary (linear layer + softmax on the last hidden vector) |
| Strength | Understanding and representation | Generation and prediction; in-context learning |
| Best for | Search, classification, ranking | Text generation, translation, conversation |
| Main limits | Maximum input of 512 tokens; no autoregressive generation | No access to future context; expensive at scale |
| IR use | Dense retrieval (Week 5), reranking (Week 8) | RankGPT-style prompting, query and document expansion, RAG, model-based IR |
16.2 Key takeaways
- Term-based models (BM25) fail on vocabulary mismatch and polysemy. They match strings and do not capture meaning. Modern retrieval needs semantic matching and context awareness.
- Section 2 has all the maths of a transformer: vector addition (add embeddings), dot products (similarity scores), vector × matrix multiplication (learned projections; , row · column for each cell) and softmax (scores → probabilities).
- The transformer (2017) gets semantic understanding from self-attention. Project the vector of each token into q (what I look for), k (what I offer) and v (my content). Calculate the score , divide by , apply softmax, then calculate . The attention matrix has a direction (it is not symmetric).
- Real transformers are "fat" (many attention heads in each layer) and "tall" (stacked layers; this is possible because the input and output shapes are equal). MLP and layer-norm steps follow attention.
- The pipeline: WordPiece tokenisation (approximately 30K learned subword pieces; it solves vocabulary size, OOV and morphology) → input embeddings (token + position + segment, added element by element) → encoder and/or decoder. Special tokens ([CLS], [SEP], [MASK], [PAD]; [BOS], [EOS]) are instructions.
- Encoders (BERT) understand. They use bidirectional attention, parallel processing, and one contextual vector for each token. They pre-train with MLM + NSP and fine-tune for tasks. IR uses them as dense retrievers and cross-encoder rerankers.
- Decoders (GPT) generate. They use masked attention. The last hidden vector × vocabulary projection → softmax → next-token probabilities, in a loop until [EOS]. Sampling is the reason why answers change. They train with next-token prediction and show in-context learning. IR uses them for prompted ranking, query and document expansion, RAG and model-based IR.
- The two types train self-supervised on web-scale text. The text gives its own labels (masked words, next words). Scale is important (scaling laws), and efficiency is now also important.
- Next: Week 5, BERT-based dense retrieval; Week 8, BERT-based rerankers. From now on, almost every model in the course is a transformer.
16.3 Study checklist
- Explain vocabulary mismatch and polysemy with one example each. Show why BM25 gives 0 for "how to fix laptop" and "computer repair guide for damaged notebooks".
- Multiply small matrices by hand, give the shape rule, and calculate a softmax of three scores.
- Give the four transformer steps. Tell which steps occur one time and which steps loop during generation.
- Explain why WordPiece is better than word splitting (vocabulary size, OOV, morphology). Describe the merge-training loop.
- Build the input embedding of a token from the token, segment and position vectors (Exercise 3). Give the correct segments.
- Give the role of [CLS], [SEP], [MASK], [PAD], [BOS] and [EOS].
- For a toy example with 3 tokens, calculate q, k, v, the scaled scores, the attention weights and an output vector. Explain and why attention is not symmetric.
- Apply the causal mask to an attention matrix. Explain why decoders need it.
- Follow the decoder output head from the hidden vector to the next-token probability, with the shapes.
- Compare the encoder and the decoder (attention, purpose, examples) and BERT and GPT (Exercise 2).
- Calculate an MLM loss from predicted probabilities. Describe NSP.
- Compare a BERT dense retriever with a cross-encoder reranker. Calculate a cosine similarity.
- Give at least four ways to use encoders or decoders for ranking. Explain diffusion language models in one paragraph.
17. Video index and resources
| Video | Topic | Length | Used in sections |
|---|---|---|---|
| Video 1 | Overview of the transformer (BM25 limits, attention, the four steps) | 13:35 | 1, 3, 7, 8 |
| Video 2 | Encoder vs decoder | 3:15 | 9 |
| Video 3 | Input embeddings | 5:25 | 5 |
| Video 4 | Tokenisation (WordPiece) | 2:58 | 4 |
| Video 5 | Special tokens | 4:00 | 6 |
| Video 6 | BERT | 7:18 | 10, 11, 12 |
| Video 7 | Autoregressive decoders | 4:45 | 8, 10, 13, 14 |
- Slide deck (56 slides): w4 slide-deck.pdf. The video transcripts are in
Pre-class youtube Transcripts/. The transcript of the in-person lecture isINFS7410_S2_2026_STL-Lecture-transcript.txt. - Explainers: How [CLS] becomes [2, 1, 1, 2] (Exercise 3, element-by-element addition) and The encoder backbone, and why it cannot generate (stacked layers, Q2 challenge).
- Spreadsheets: Transformer in a Spreadsheet (live formulas for the maths refresher, embeddings, five-step attention, multi-head attention, the decoder mask and the output head) and Transformer by Hand (calculate one transformer block yourself, with practice, check and reveal modes).
18. References
- Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS 2017.
- Devlin, J., Chang, M.-W., Lee, K. and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019.
- Brown, T. et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020, 33, 1877–1901.
- Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Yin, B. and Hu, X. (2023). Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond. arXiv:2304.13712.