INFS7410 · Information Retrieval and Web Search · Week 4

The Transformer Architecture

Simplified Technical English edition. Term-based models such as BM25 match strings. They do not capture meaning. This week builds the transformer step by step, so that later weeks can use it for retrieval. The notes cover the vector and matrix maths, WordPiece tokenisation, token, position and segment embeddings, and special tokens. They show self-attention with queries, keys and values, with a calculation by hand. They explain the decoder and its output head, encoder-only and decoder-only models, and self-supervised training. They describe BERT (masked language modelling, next sentence prediction, fine-tuning, dense retrieval and cross-encoder reranking), autoregressive decoders such as GPT, and the uses of LLMs in information retrieval. Dr Shuai Wang, UQ ielab.

1. Why BM25 is not enough

Sources: slides 2–6 · Video 1 (Overview of the transformer) · Lecture transcript (ranking history, term mismatch)

Weeks 1–3 built the classical retrieval stack. Week 1 covered search engine architecture and text processing. Week 2 covered evaluation. Week 3 covered term-based lexical models: TF, TF-IDF, VSM, BIM and BM25. BM25 was the standard model for decades. But BM25 has one basic limit: it can only match exact words. It has no model of the meaning of words or sentences.

Key ideaTerm-based lexical models score a document from the terms that it shares with the query. They model relevance with term statistics and term overlap. The lecturer said that term statistics cause the limit. If the query and the document use the same terms, the document gets a high rank. If they use different terms with the same meaning, the document gets a low rank or the system does not retrieve it. The model has no semantic understanding of the query or the documents.

1.1 Two classic failure modes

1 · Vocabulary mismatch (false negative) Query car Document automobile no shared term BM25 ≈ 0 same meaning, zero overlap → relevant document missed 2 · Polysemy (false positive) Query closest bank? Document river bank shares: bank BM25 > 0 shared string, different meaning → irrelevant document ranked high
Term matching fails in two directions. The system misses synonyms that share no string. The system gives a high score to a shared string with a different meaning.
Failure modeExampleWhat BM25 does
Vocabulary mismatch (false negative)The query is car. The document says automobile. A second example from the lecture: a query with biomedical and a document about medicine or biology. The concepts are related, but the strings are different.The texts share zero terms. Thus the score is near zero. The system misses the relevant document.
Polysemy (false positive)The query is where is the closest bank? (a financial institution). The document is about a river bank (a geographic feature). The in-class example: "I want to fix my laptop" and "I want to fix this guy". The same word has different meanings in different sentences.The word bank is in both texts. Thus the document can get a high rank. The system retrieves an irrelevant document.

Worked example 1A relevant document BM25 cannot retrieve

The query is how to fix laptop. The document title is "Computer repair guide for damaged notebooks" (slide 5). A person sees that the two texts match. Examine the term overlap after stop-word removal:

Query termDoes a document term match?Overlap
fixrepair is a synonym, but it is a different string✗
laptopcomputer and notebooks are related concepts, but they are different strings✗

The BM25 score from Week 3 is a sum over the query terms that are also in the document. Here, this set is empty:

BM25(Q,D)=∑t ∈ Q∩DIDF(t)⋅(k1+1) ft,Dk1((1−b)+b∣D∣avdl)+ft,D  =  0because Q∩D=∅ \text{BM25}(Q,D)=\sum_{t\,\in\,Q\cap D}\text{IDF}(t)\cdot\frac{(k_1+1)\,f_{t,D}}{k_1\big((1-b)+b\frac{|D|}{avdl}\big)+f_{t,D}} \;=\; 0 \quad\text{because } Q\cap D=\varnothing
ResultThe document score is 0. BM25 cannot retrieve the document. This is a pure semantic mismatch failure. The slide says: "No overlapping terms; but the document should be relevant. Not retrievable by BM25."

1.2 What a modern retrieval system needs

RequirementMeaningExample
Semantic matchingThe system understands meaning, and also surface stringscar must match automobile. laptop and computer are related concepts.
Context awarenessOne word can have different meanings in different contexts. The system must understand the query intent.Apple in a technology context is a company. apple in a recipe is a fruit.

1.3 Where we are in the history of ranking

The lecture started with a timeline of rankers. Week 4 is at the "Industrial Revolution" (2017–2018): the start of the transformer. All later parts of the course use the transformer.

EraModelsWhere in this course
"Jurassic" (1950s–1990s)Term-based lexical models (Boolean, TF-IDF, BM25)Week 3
"Human evolution" (2000s–2010s)Feature-based reranking models (LTR, online LTR; fusion, RF)Weeks 7, 11, 12
"Stone Age transitioning" (2010s)Word2vec, RNN, LSTMOutside the course
"Industrial Revolution!" (2017–2018): we are hereTransformer. Encoder: BERT · Decoder: GPTWeek 4
"New inventions" (2018–2022)Encoder-based rerankersWeek 8
(same period)Dense retrievers; learnt sparse modelsWeeks 5, 6
"AI age" (2022 onwards)ChatGPT, Mistral, Claude, Deepseek, QwenParts of Weeks 5, 9, 10
"The future is accelerating"LLM rerankers and retrievers, LLM-assisted ranking, RAG, search agentsParts of Weeks 5, 9, 10
Watch outThe field changes fast. Thus the course focuses on core ideas and the most important techniques. The lecturer said that this week is one of the most important weeks of the course. After this week, almost every model in the course is a transformer.
PracticeA user searches jaguar top speed. BM25 returns a page about the dealerships of the Jaguar car brand. BM25 misses a wildlife page that says "the big cat can sprint at 80 km/h". Name the two failure modes.
One failure gives a score to a shared string. The other failure misses a relevant page with different strings.
AnswerThe dealership page is a polysemy false positive. The string jaguar matches, but the meaning is wrong. The wildlife page is a vocabulary mismatch false negative. The words big cat and sprint at 80 km/h express jaguar and top speed, but they share no terms. To fix the two failures, the system needs semantic matching and context awareness.

2. Maths refresher: vectors, matrices and softmax

Sources: Lecture transcript (linear-layer Q&A, Exercise 3 vector addition, self-attention walkthrough) · background for slides 10–19 and 25–29

A transformer uses a small set of simple operations on vectors and matrices. Neural models can only use numbers. The lecturer said: "we need to represent things in numbers, we need to do number calculations. That's how those neural models are formed." This section explains these operations from the start. The workbook Transformer in a Spreadsheet (sheet "1 Maths") calculates each example of this section with live formulas.

2.1 Vectors, vector addition and the dot product

A vector is an ordered list of numbers, for example x=[2, 1, 0, 3] x = [2,\, 1,\, 0,\, 3] . The number of entries is the dimension of the vector (here, 4). In a transformer, a vector represents each token. BERT uses vectors with 768 dimensions. Llama uses vectors with 2,048 dimensions.

(a+b)i  =  ai+bi,i=1,…,d (a+b)_i \;=\; a_i + b_i, \qquad i = 1,\dots,d
Where
a,ba, b
two vectors with the same dimension
dd
the dimension (number of entries)
ii
a position index, from 1 to dd
(a+b)i(a+b)_i
entry ii of the sum vector
Read it aloudEntry i of a plus b equals entry i of a plus entry i of b, for each position i from 1 to d.
Formula in plain wordsVector addition works position by position. Add the first entries, then the second entries, and continue to the end. The two vectors must have the same dimension. The result is a vector with the same dimension. Thus the sum of three vectors is one vector. The lecturer said this in Exercise 3: "you add them together, you do not make them a matrix". Section 5 uses this operation to add the token, segment and position embeddings.
a⋅b  =  ∑i=1dai bi a \cdot b \;=\; \sum_{i=1}^{d} a_i\, b_i
Where
a⋅ba \cdot b
the dot product of aa and bb: one number
ai, bia_i,\ b_i
entry ii of each vector
dd
the shared dimension
∑i=1d\sum_{i=1}^{d}
add the products for all dd positions
Read it alouda dot b equals the sum, over each position i from 1 to d, of a sub i times b sub i.
Formula in plain wordsMultiply the two vectors entry by entry. Then add the products. The result is one number. This number shows how much the two vectors agree. The number is large and positive when large entries align (similar directions). The number is near zero when the vectors are not related. The number is negative when the vectors point in opposite directions. A longer vector gives a larger result. Thus cosine similarity (Section 12, and the VSM from Week 3) divides by the lengths. The dot product is the core similarity operation of this course. Attention scores (Section 7) and dense retrieval scores (Week 5) are dot products.

Worked example 2Add three vectors, then take a dot product

1
Add position by position: [1,0,0,1]+[0,1,0,1]+[1,0,1,0]=[1+0+1,  0+1+0,  0+0+1,  1+1+0] [1,0,0,1] + [0,1,0,1] + [1,0,1,0] = [1{+}0{+}1,\; 0{+}1{+}0,\; 0{+}0{+}1,\; 1{+}1{+}0] .
[1,0,0,1]+[0,1,0,1]+[1,0,1,0]=[2, 1, 1, 2] [1,0,0,1] + [0,1,0,1] + [1,0,1,0] = [2,\,1,\,1,\,2]
2
Dot product of [2,2][2,2] and [1,3][1,3]: multiply the entries and add, 2×1+2×3=2+6 2\times 1 + 2\times 3 = 2 + 6 .
ResultThe sum is the vector [2,1,1,2][2,1,1,2] with 4 dimensions. These are the three embeddings of [CLS] in Exercise 3 (Section 15). The dot product [2,2]⋅[1,3]=8[2,2]\cdot[1,3] = 8 is one number.

2.2 Matrices and the transpose

A matrix is a rectangular grid of numbers with nn rows and mm columns. Its shape is n×mn \times m ("n by m"). The rows are always first. A vector is a matrix with one row (a row vector, 1×m1\times m) or one column (a column vector, n×1n\times 1). For example, A=[123456]A=\begin{bmatrix}1 & 2 & 3\\ 4 & 5 & 6\end{bmatrix} is 2×32\times 3, and B=[789101112]B=\begin{bmatrix}7 & 8\\ 9 & 10\\ 11 & 12\end{bmatrix} is 3×23\times 2.

(A⊤)ij  =  Aji \big(A^{\top}\big)_{ij} \;=\; A_{ji}
Where
AA
a matrix with shape n×mn \times m
A⊤A^{\top}
the transpose of AA, with shape m×nm \times n
AjiA_{ji}
the entry of AA in row jj, column ii
i,ji, j
the row and column indices of the transposed matrix
Read it aloudThe entry in row i, column j of A transpose equals the entry in row j, column i of A.
Formula in plain wordsThe transpose turns the rows of a matrix into columns. Thus the 2×32\times 3 matrix AA becomes the 3×23\times 2 matrix A⊤=[142536]A^{\top}=\begin{bmatrix}1 & 4\\ 2 & 5\\ 3 & 6\end{bmatrix}. The numbers stay the same. Only their positions change. The attention formula QK⊤QK^{\top} (Section 7) uses the transpose. It makes the shapes correct for the multiplication.

2.3 Matrix multiplication, step by step

C=AB,Cij  =  ∑r=1kAir Brj,(n×k) (k×m)=(n×m) C = AB, \qquad C_{ij} \;=\; \sum_{r=1}^{k} A_{ir}\, B_{rj}, \qquad (n\times k)\,(k\times m) = (n\times m)
Where
AA
the left matrix, n×kn \times k
BB
the right matrix, k×mk \times m
CC
the product, n×mn \times m
CijC_{ij}
the entry of CC in row ii, column jj
kk
the inner dimension: the columns of AA must equal the rows of BB
rr
the index over the inner dimension
Read it aloudC equals A times B. The entry of C in row i and column j is the sum, over r from 1 to k, of the entry of A in row i, column r, times the entry of B in row r, column j. An n by k matrix times a k by m matrix gives an n by m matrix.
Formula in plain wordsEach cell of the result is one dot product: row ii of AA with column jj of BB. Thus matrix multiplication is a grid of dot products. The shape rule says that the inner dimensions (kk) must be equal. The sum removes the inner dimension. The outer dimensions give the shape of the result. If the inner dimensions are different, the product is not defined. The order is important: ABAB and BABA are usually different, and can have different shapes.
A (2×3) 1 2 3 4 5 6 row 1 highlighted × B (3×2) 7 9 11 8 10 12 column 1 highlighted = C (2×2) 58 64 139 154 C[row 1, col 1] = (row 1 of A) · (column 1 of B) = 1×7 + 2×9 + 3×11 = 7 + 18 + 33 = 58 inner dimensions match (3 = 3) and are consumed; the result is 2 × 2
Each cell of the product is one dot product. It uses a row of the left matrix and a column of the right matrix.

Worked example 3Every cell of A × B

Multiply the 2×32\times 3 matrix AA by the 3×23\times 2 matrix BB from Section 2.2. The inner dimensions are equal (3 = 3). Thus the result is 2×22\times 2.

CellRow of A · column of BValue
row 1, col 11×7 + 2×9 + 3×11 = 7 + 18 + 3358
row 1, col 21×8 + 2×10 + 3×12 = 8 + 20 + 3664
row 2, col 14×7 + 5×9 + 6×11 = 28 + 45 + 66139
row 2, col 24×8 + 5×10 + 6×12 = 32 + 50 + 72154
[123456][789101112]=[5864139154] \begin{bmatrix}1 & 2 & 3\\ 4 & 5 & 6\end{bmatrix}\begin{bmatrix}7 & 8\\ 9 & 10\\ 11 & 12\end{bmatrix}=\begin{bmatrix}58 & 64\\ 139 & 154\end{bmatrix}
ResultAB=[5864139154]AB = \begin{bmatrix}58 & 64\\ 139 & 154\end{bmatrix}. Watch out: the order of the matrices is important. Here, BABA is 3×33\times 3, which is a different shape.

The special case that transformers use frequently: vector × matrix

A row vector (1×n)(1 \times n) times a matrix (n×m)(n \times m) gives a row vector (1×m)(1 \times m). This operation is a linear layer (also called a projection) in a neural network. It changes a vector of one size into a vector of a different size. It uses a matrix of learned weights. The lecturer wrote this on paper in the Q&A. A token vector with nn dimensions times a learned n×mn \times m matrix gives an output vector with mm dimensions. When a transformer changes one vector into a different vector, it uses this operation.

Worked example 4A tiny linear layer

Project the vector [2,1][2, 1] with the 2×32 \times 3 weight matrix [102311]\begin{bmatrix}1 & 0 & 2\\ 3 & 1 & 1\end{bmatrix}. Each output entry is the dot product of the vector with one column.

[ 2,  1 ]⏟1×2  [102311]⏟2×3=[ 2×1+1×3,    2×0+1×1,    2×2+1×1 ]=[ 5,  1,  5 ]⏟1×3 \begin{aligned} \underbrace{[\,2,\;1\,]}_{1\times 2}\;\underbrace{\begin{bmatrix}1 & 0 & 2\\ 3 & 1 & 1\end{bmatrix}}_{2\times 3} &= [\,2{\times}1{+}1{\times}3,\;\; 2{\times}0{+}1{\times}1,\;\; 2{\times}2{+}1{\times}1\,] \\ &= \underbrace{[\,5,\;1,\;5\,]}_{1\times 3} \end{aligned}
ResultA vector with 2 dimensions goes in. A vector with 3 dimensions, [5,1,5][5, 1, 5], comes out. Section 7 uses this operation to make the query, key and value vectors. Section 8 uses it to project a hidden vector (768 dimensions) onto the full vocabulary (approximately 30,000 dimensions) to select the next word.

2.4 Softmax: from scores to probabilities

A transformer must often change a vector of raw scores into probabilities. Raw scores (logits) can be any real numbers. Probabilities are numbers between 0 and 1 with a sum of 1. The softmax function does this change.

softmax(z)i  =  ezi∑j=1nezj \text{softmax}(z)_i \;=\; \frac{e^{z_i}}{\sum_{j=1}^{n} e^{z_j}}
Where
zz
a vector of nn raw scores (logits), z=[z1,…,zn]z = [z_1, \dots, z_n]
ziz_i
the score of item ii
ezie^{z_i}
the exponential of the score (e≈2.718e \approx 2.718); always positive
∑j=1nezj\sum_{j=1}^{n} e^{z_j}
the sum of the exponentials of all nn scores
softmax(z)i\text{softmax}(z)_i
the probability of item ii
Read it aloudThe softmax probability of item i equals e to the power z sub i, divided by the sum over all items j of e to the power z sub j.
Formula in plain wordsCalculate the exponential of each score. Then divide each result by the total. The exponential makes all values positive. Thus negative logits are not a problem: they become small values. The exponential also makes differences larger. A score that is 1 higher gets approximately 2.7 times more weight. The division makes the sum of the outputs equal to 1. The largest score always gets the largest share. If you add the same constant to all scores, the output stays the same. Softmax occurs two times this week. Attention uses it to change similarity scores into attention weights (Section 7). The decoder output uses it to change vocabulary logits into next-token probabilities (Section 8).
scores z (any real numbers) 2.0 1.0 0.0 e² ≈ 7.39 e¹ ≈ 2.72 e⁰ = 1.00 divide each by the sum 11.11 0.67 0.24 0.09 item 1 item 2 item 3 probabilities in (0, 1) that sum to 1
Softmax calculates the exponential of each score and normalises the results. A difference of 1 in score becomes a ratio of approximately 2.7 in probability.

Worked example 5Softmax of three scores

The scores are z=[2.0,  1.0,  0.0] z = [2.0,\; 1.0,\; 0.0] .

1
Calculate the exponentials: e2.0≈7.39e1.0≈2.72e0.0=1.00 \begin{aligned} e^{2.0} &\approx 7.39 \\ e^{1.0} &\approx 2.72 \\ e^{0.0} &= 1.00 \end{aligned}
2
Calculate the sum: 7.39+2.72+1.00≈11.11 7.39 + 2.72 + 1.00 \approx 11.11 .
3
Divide each value by the sum:
softmax(z)≈[ 7.3911.11,  2.7211.11,  1.0011.11 ]=[ 0.67,  0.24,  0.09 ] \begin{aligned} \text{softmax}(z) &\approx \left[\,\frac{7.39}{11.11},\; \frac{2.72}{11.11},\; \frac{1.00}{11.11}\,\right] \\ &= [\,0.67,\; 0.24,\; 0.09\,] \end{aligned}
ResultThe result is [0.67,0.24,0.09][0.67, 0.24, 0.09]. Each entry is in (0,1)(0,1), and the sum is 1. This is a probability distribution. The largest score gets the largest share, and the differences become larger.
TakeawaysUse this summary for the remaining sections:
OperationShapesUse this week
Vector + vector (element by element)same length in, same length outAdd the token, segment and position embeddings (Section 5)
Vector · vector (dot product)two vectors in, one number outAttention scores (Section 7); cosine similarity (Section 12)
Vector × matrix(1×n)(n×m)=(1×m)(1\times n)(n\times m) = (1\times m)Q, K, V projections (Section 7); the output linear layer (Section 8)
Matrix × matrix(n×k)(k×m)=(n×m)(n\times k)(k\times m) = (n\times m)All attention scores at the same time. This is why GPUs are good for transformers.
Softmaxscores in, probabilities out (same length)Attention weights (Section 7); next-token probabilities (Section 8)
PracticeA token vector has 768 dimensions. A learned matrix has the shape 768 × 64. What is the shape of the product? What occurs if you multiply the vector by a 64 × 768 matrix?
Use the shape rule (1×n)(n×m)(1\times n)(n\times m).
Answer(1×768)(768×64)=(1×64)(1\times 768)(768\times 64) = (1\times 64). The result is a vector with 64 dimensions. With a 64×76864\times 768 matrix, the inner dimensions are 768 and 64. They are different, thus the product is not defined. (One attention head in BERT-base uses this 768 → 64 projection for its query, key and value vectors.)
PracticeCalculate softmax of [1,1,1][1, 1, 1] and of [3,3,3][3, 3, 3]. What does this show?
AnswerThe two results are [13, 13, 13]\left[\frac{1}{3},\ \frac{1}{3},\ \frac{1}{3}\right]. Softmax uses only the differences between the scores. If you add the same constant to all scores, the result stays the same.

3. The transformer: overview and the four steps

Sources: slides 7–19 · Video 1 (Overview of the transformer) · Lecture transcript (four steps, block contents)

In 2017, researchers at Google published the paper "Attention Is All You Need" (Vaswani et al.). This paper gave the transformer architecture. The full diagram of the architecture looks difficult. The lecturer said: "it may look quite complicated, and in fact it was actually complicated … and this is already a simplified version". But the main idea is simple.

Key ideaTransformers can "understand semantics" in language. Slide 8 says it more exactly: they learn to "understand word distribution". They learn how words relate to each other, why words occur together, and what meaning words have in context. Words are then more than strings of characters. They become points in a space that holds meaning and relationships.

The original transformer was for machine translation. An English sentence such as "The cat sat on the mat" goes into an encoder–decoder pipeline. The model gives the same sentence in a different language. The slide uses Chinese: 猫坐在垫子上.

The cat sat on the mat input text (English) Encoder encoded vectors Decoder 猫坐在垫子上 Chinese the complete original transformer = encoder + decoder, trained for translation
The original transformer was for translation. Retrieval uses the same core technology to find semantic relationships.
Watch outTranslation seems far from search. But the same technology maps "cat" to "猫" and maps "laptop" to "notebook computer".

3.2 The four processing steps

Slides 13–19 build the architecture in four steps. These steps are the map for Sections 4 to 8.

STEP 1 tokenise text, build input embeddings STEP 2 · ENCODER self-attention builds contextual vectors STEP 3 · DECODER takes ⟨start⟩ + the encoded representation STEP 4 predict the next token from output probabilities generated token is fed back in; loop until ⟨end⟩ hand-off: encoded vectors
The simplified transformer pipeline. Tokenise and embed, encode, give the result to the decoder, then generate one token at a time. The loop stops at an ⟨end⟩ token.
1
Tokenise and embed. The model tokenises the text and changes it into input embeddings. Words become vectors that the model can use in calculations (Sections 4–5).
2
Encode. The embeddings go through the encoder. The encoder uses self-attention to make a contextual representation of each word (Section 7). At the end, the output vectors of the encoder represent the full input: the words, their relationships and their meaning. This is the hand-off point. The encoder work is complete.
3
The decoder starts. The decoder gets two inputs: a special ⟨start⟩ token and the encoded representations. The video says that the decoder asks: "I'm going to generate; encoder, give me the context."
4
Generate, then loop. The decoder gives output token probabilities and selects the next token. It adds this token to its input, with the encoded representation, and repeats. The loop continues until the decoder gives an ⟨end⟩ token (Section 8). This step-by-step generation makes coherent text.

Thus the translation of "The cat sat on the mat" is a loop. In each iteration, the decoder gives a probability distribution over the vocabulary. It selects the token with the highest probability:

IterationDecoder inputPredicted token
1⟨start⟩ + encoded vectors猫
2⟨start⟩ 猫 + encoded vectors坐在
3⟨start⟩ 猫 坐在 + encoded vectors垫子
4⟨start⟩ 猫 坐在 垫子 + encoded vectors上
5⟨start⟩ 猫 坐在 垫子 上 + encoded vectors⟨end⟩: generation stops

3.3 What is in each block

The lecture showed a hand-drawn diagram of the architecture. Each encoder or decoder block has these parts, from bottom to top: multi-head self-attention → layer norm → multilayer perceptron (MLP) → layer norm. Arrows go around each part (residual connections). Section 7.4 shows the diagram. Remember two facts:

Watch outSlide 19 says that this is a simplified version. Real transformers use multi-head attention: many attention heads work in parallel. They also use feed-forward layers, residual connections and layer normalisation. These notes only show the outline of these parts. At this level, the goal is to understand the basic flow and why it captures language well.
PracticeIn the translation loop, which steps occur one time for each sentence? Which steps occur one time for each generated token?
Look at the start and the end of the dashed loop arrow.
AnswerSteps 1 and 2 occur one time: the model tokenises, embeds and encodes the English input. Steps 3 and 4 occur one time for each output token. The decoder reads ⟨start⟩ and all tokens that it generated before, with the same encoded vectors. It predicts one token and repeats until it gives ⟨end⟩. Five iterations give 猫 坐在 垫子 上 ⟨end⟩.

4. Step 1a: tokenisation and WordPiece

Sources: slides 30–33 · Video 4 (Tokenisation) · Lecture transcript (WordPiece, token IDs)

Before the embedding step, the model must break the text into tokens. Tokens are the units that the model reads. Each token has an integer ID in a fixed vocabulary. The lecturer's example: the token my can have ID 3500. The model maps these IDs to vectors. Week 1 tokenised text for an inverted index. Here, the tokeniser serves a neural model with a lookup table of fixed size. Thus the design goals are different.

A simple method splits the text at spaces: "my dog is cute" → [my, dog, is, cute]. Slide 31 gives three problems with this method:

  1. The vocabulary is very large. English has approximately 500,000 unique words. This needs a very large embedding lookup table. With names, typing errors and new words, the vocabulary is "almost infinite".
  2. Out-of-vocabulary (OOV) words. New words such as coronavirus (new in 2020), typing errors, or random strings such as ecerfeferf. The model did not see these words in training. Thus it has no embedding for them.
  3. The model loses morphological relationships. run, running and runner become three unrelated tokens. But these words are clearly related.

The opposite method uses single characters as tokens (only a–z, 26 tokens). This method can represent all text. But characters do not show how people combine letters into units with meaning. Also, the sequences become very long. The best method is between these two methods.

Characters p · l · a · y · i · n · g vocabulary: 26 letters very long sequences few meaningful units WordPiece subwords play · ##ing vocabulary: about 30K pieces any word can be spelled morphology kept Whole words playing vocabulary: about 500K+ words OOV words break it morphology lost vocabulary size smaller larger
WordPiece is between characters and whole words. Its vocabulary has a fixed size that is easy to manage, and it can spell all words.
Key ideaWordPiece is a trained tokeniser. It breaks words into subword pieces. Thus the vocabulary has a fixed number of pieces (approximately 30,000). WordPiece can write all words as a sequence of pieces, also new words and typing errors. The mark ## shows a piece that continues a word.
TokeniserTokens for "[CLS] My dog is cute [SEP] he likes playing [SEP]"
Word splitting[CLS], my, dog, is, cute, [SEP], he, likes, playing, [SEP]
WordPiece[CLS], my, dog, is, cute, [SEP], he, likes, play, ##ing, [SEP]
Watch outWhy does WordPiece split playing? Think about the training of the tokeniser. The string ing occurs very frequently in English. A separate token for each -ing word makes the vocabulary too large. Thus the tokeniser keeps play and ##ing as two pieces that it can use again.

4.1 How WordPiece training works

no: repeat start with characters a–z count adjacent pairs in web text merge the most frequent pair target size (~30K) reached? yes learned subword vocabulary merges grow pieces such as t + h → th, i + n + g → ing, t + i + o + n → tion
WordPiece training is a counting loop over a very large text collection. Merge the most frequent adjacent pair, then count again. Stop when the vocabulary has the target size.
1
Start with a character vocabulary: [a, b, c, …, z].
2
Analyse a very large amount of (web) text. Find the most frequent adjacent character pairs.
3
Merge these pairs into new subwords: th, ing, tion, …
4
Repeat until the vocabulary has the target size (approximately 30K pieces).
5
The result is a learned subword vocabulary. Almost all modern transformer models use it (BERT, GPT, many LLMs). These pieces can represent all words.
Watch outThis is still term statistics. It uses the same counting over a collection as Week 3, but here it learns a vocabulary. Different companies train tokenisers on different collections, with different target sizes and goals. Thus different models tokenise the same text differently. BERT-style models use approximately 30K pieces. Some recent LLMs use hundreds of thousands of tokens. The technique is the same, but the statistics are different. Exercise 2 (Section 15) tests this.

Worked example 6Subword merges on a toy corpus

The training data (illustrative) has four word types with these counts: low ×5, lower ×2, newest ×6, widest ×3.

0
Start with characters: l o w, l o w e r, n e w e s t, w i d e s t.
1
Count the adjacent pairs. Use the word counts as weights:
PairWhere it occursCount
e snewest (6) + widest (3)9 (most frequent)
s tnewest (6) + widest (3)9 (tied)
l olow (5) + lower (2)7
o wlow (5) + lower (2)7
Merge e s → es. The words become n e w es t and w i d es t.
2
Now es t occurs 9 times. Merge it into est.
3
Merge l o (7) → lo. Then merge lo w (7) → low. Continue in this way. The vocabulary grows by one useful piece at each step: es, est, lo, low, …
ResultAfter training, a new word such as lowest becomes low + ##est. The two pieces are in the vocabulary. The relation to low and widest stays. There is no OOV failure. This example shows merges by frequency. WordPiece gives each possible merge a score, score(a,b)=count(ab)count(a) count(b) \text{score}(a,b)=\frac{\text{count}(ab)}{\text{count}(a)\,\text{count}(b)} . This score prefers pairs that occur together more frequently than their parts predict. The method is the same.

Worked example 7Tokenising the problem words

Input wordWordPiece output (illustrative)Why it works
playingplay + ##ingThe morphology stays: play is also in played and player
coronaviruscorona + ##virusThe word was new in 2020, but the two pieces were in the vocabulary. Thus there is no OOV.
ecerfeferfe + ##cer + ##fe + ##fer + ##fRandom strings and typing errors also break into known pieces. In the worst case, the pieces are single characters.
ResultThe three problem words from slide 31 get valid token sequences. Thus each word gets embeddings. The exact splits depend on the trained vocabulary.
PracticeWhy does a WordPiece model never have an out-of-vocabulary error? Why does a word-level model have this error?
Think about the vocabulary before the first merge.
AnswerWordPiece training starts with all characters. It only adds merged pieces. Thus each single character stays in the vocabulary. The model can spell all strings with pieces. In the worst case, it uses one character at a time. A word-level vocabulary is a fixed list of whole words. A word that is not in the list (a new term or a typing error) has no entry and no embedding.

5. Step 1b: transformer input embeddings

Sources: slides 24–30 · Video 3 (Input embeddings) · Lecture transcript (three embeddings, "I asked him")

After tokenisation, each token must become a vector. A token ID tells what the token is. It does not tell where the token is. But word order changes meaning. The lecturer's example: "I asked him" and "he asked me" use almost the same words in different positions. Their meanings are very different. Thus transformers combine three types of embeddings (slide 25):

EmbeddingPurposeRequired?
Token embeddingChanges tokens into vectors. It represents word meaning with a learned lookup table.Always
Position embeddingTeaches the model the word order. Each position (0, 1, 2, …) has a learned vector.Always
Segment embeddingShows the different parts of the input, for example the query and the document. Each segment has a learned vector.Optional: only some models, for example BERT

5.1 Token embeddings

The model tokenises the text. Then it finds each token in a learned embedding table. The table has one learned vector for each vocabulary entry. This vector holds the meaning of the token alone. The idea comes from earlier word-vector models such as word2vec. The example from slide 26 (the vectors are shortened):

StageValue
Text"Information Retrieval"
Tokenise["information", "retrieval"]
Lookupinformation → [0.1, 0.3, −0.2, 0.9, …]
retrieval → [0.4, −0.1, 0.6, 0.2, …]

5.2 Position embeddings

Word order is important: "information retrieval" ≠ "retrieval information". Each position in the sequence has a learned embedding vector (slide 27): Pos1 → [0.9, 0.4, 0.2, 0.8, …], Pos2 → [0.2, −0.4, 0.4, 0.2, …].

5.3 Segment embeddings (optional)

Models such as BERT use segment embeddings when the input has more than one sequence. An example is a query and a document in one input (the BERT cross-encoder of Section 12). Special tokens separate the segments. [CLS] is at the start. [SEP] is between the segments and at the end. Each token in a segment gets the embedding of that segment (slide 28):

StageValue
TextInformation (query) + Retrieval (document)
Tokenised[CLS] information [SEP] retrieval [SEP]
Segmentsquery segment → [0.6, 0.3, 0.7, −0.9, …]
doc segment → [0.5, 0.1, 0.8, 0.2, …]
Watch outDecoder-only models (GPT, Llama) usually do not use segment embeddings. Modern LLMs use better position methods, for example rotary embeddings ("RoPE") for very long contexts. The basic idea stays the same: tell the model what each token is and where it is.

5.4 Putting it together

Inputi  =  Etoken(xi)  +  Eposition(i)  +  Esegment(si) \text{Input}_i \;=\; E_{\text{token}}(x_i)\;+\;E_{\text{position}}(i)\;+\;E_{\text{segment}}(s_i)
Where
ii
the position of the token in the sequence (0, 1, 2, …)
xix_i
the token at position ii
sis_i
the segment of this token (A or B)
Etoken(xi)E_{\text{token}}(x_i)
the learned vector for this token, from the token lookup table
Eposition(i)E_{\text{position}}(i)
the learned vector for position ii
Esegment(si)E_{\text{segment}}(s_i)
the learned vector for segment sis_i (models without segments do not use it)
Inputi\text{Input}_i
the final input vector for position ii; it has the same dimension as each part (768 in BERT-base)
Read it aloudThe input vector at position i equals the token embedding of the token x sub i, plus the position embedding of position i, plus the segment embedding of its segment s sub i.
Formula in plain wordsThe input of each token is the element-by-element sum of three vectors with the same length. This is the vector addition from Section 2.1. The token part tells what the token is. The position part tells where the token is. The segment part tells which part of the input has the token. The result is one vector for each token, with the same dimension as each part. Thus a sentence with nn tokens becomes an n×dn \times d input matrix, with one row for each token. If one part changes, the input vector changes. For example, the same word at a different position gets a different input vector.
[CLS] my dog is cute [SEP] he likes play ##ing [SEP] Token E[CLS] E my E dog E is E cute E[SEP] E he E likes E play E ##ing E[SEP] + Segment E_A (segment A) for every token E_B (segment B) + Position E pos 0 E pos 1 E pos 2 E pos 3 E pos 4 E pos 5 E pos 6 E pos 7 E pos 8 E pos 9 E pos 10 = sum ↓ final input embeddings fed to the transformer: one vector per token
The example from slide 30, [CLS] my dog is cute [SEP] he likes play ##ing [SEP]. The tokens up to the first [SEP] get segment A. The tokens after it get segment B. The positions are 0, 1, 2, … Thus the model knows the position of each token.

For a visual version of this addition, column by column, see the explainer How [CLS] becomes [2, 1, 1, 2]. Also see sheet "2 Embeddings" of the Transformer in a Spreadsheet workbook.

Worked example 8One token's final input embedding

Use the token likes from the diagram. The token is likes, the position is 7, and the segment is B. The embeddings have 4 dimensions (illustrative values):

ComponentVector
Etoken(likes) E_{\text{token}}(\text{likes}) [0.2, −0.1, 0.5, 0.3]
Eposition(7) E_{\text{position}}(7) [0.0, 0.3, −0.2, 0.1]
Esegment(B) E_{\text{segment}}(B) [0.1, 0.1, 0.0, −0.2]

Add element by element:

Inputlikes=[0.2+0.0+0.1,    −0.1+0.3+0.1,0.5−0.2+0.0,    0.3+0.1−0.2]=[0.3,  0.3,  0.3,  0.2] \begin{aligned} \text{Input}_{\text{likes}} &= [0.2{+}0.0{+}0.1,\;\; -0.1{+}0.3{+}0.1, \\ &\qquad 0.5{-}0.2{+}0.0,\;\; 0.3{+}0.1{-}0.2] \\ &= [0.3,\; 0.3,\; 0.3,\; 0.2] \end{aligned}
ResultInputlikes=[0.3,0.3,0.3,0.2]\text{Input}_{\text{likes}} = [0.3, 0.3, 0.3, 0.2]. Put the same word likes at position 2 in segment A. The token embedding stays the same. The position and segment embeddings change. Thus the final input vector changes. In this way, the model knows where and in which part a word occurs. Section 15 has the exam version of this calculation (in-class Exercise 3).
PracticeIn "the bank by the river bank", the token bank is at positions 1 and 5 (one segment). Do the two tokens get the same input vector?
AnswerThe two tokens have the same token embedding Etoken(bank)E_{\text{token}}(\text{bank}) and the same segment embedding. But their position embeddings are different (Eposition(1)E_{\text{position}}(1) and Eposition(5)E_{\text{position}}(5)). Thus the input vectors are different. But the input embedding has little context. Self-attention in the encoder (Section 7) finds which bank is a financial bank and which bank is a river bank.

6. Special tokens

Sources: slides 34–39 · Video 5 (Special tokens) · Lecture transcript (CLS for retrieval, PAD for batching, "[FRANKENSTEIN]")

Key ideaSpecial tokens are reserved symbols. They are instructions for the transformer. They tell the model where sequences start and stop. They tell the model how to separate the parts of the input and which type of task to do. The tokeniser keeps them. Thus each special token always becomes exactly one token in text.

6.1 Encoder special tokens: [CLS], [SEP], [MASK], [PAD]

TokenNameRole
[CLS]ClassificationAlways at the start of the input. Sentence-level tasks (classification, ranking) use its output vector. The lecturer said: its name is the classification token, but you can fine-tune the model to use this token for any task. Retriever models use it frequently to represent a full sentence as one vector.
[SEP]SeparatorSeparates different sequences. It helps the model to find the parts of the input (for example the query and the document).
[MASK]Masked tokenUsed in training. The model learns to predict the masked words (Sections 10 and 11).
[PAD]PaddingMakes all sequences the same length, so that the model can batch them. The GPU processes a batch in parallel only if all sequences have one length. Attention ignores [PAD].
padding task signal separator end of input [PAD] [CLS] laptop repair [SEP] Computer maintenance guide for damaged notebooks [SEP] segment A: the query segment B: the document [PAD] keeps batch lengths equal and is ignored by attention
The search-task input of slide 37. Special tokens show the task, the boundary between the query and the document, and the end of the input.

Worked example 9A search-task input, token by token

Input: [PAD] [CLS] laptop repair [SEP] Computer maintenance guide for damaged notebooks [SEP]

PieceWhat it does here
[PAD]Keeps the same length when the model batches this input with longer inputs. Attention ignores it.
[CLS]Shows a classification or ranking task. Its final vector holds the relevance decision.
laptop repairSegment A: the query
[SEP]Separates the query from the document
Computer maintenance guide …Segment B: the document
[SEP]Shows the end of the input sequence
ResultA BERT reranker (cross-encoder) uses this query + document format (Section 12). The model reads the two texts together. It gives a relevance score from the [CLS] vector.

6.2 Decoder special tokens: [BOS], [EOS]

TokenNameRole
[BOS]Beginning of sequenceShows the start of generation. It is the first token that goes into the decoder during inference.
[EOS]End of sequenceShows the end of generation. It tells the decoder when to stop. It controls the length of the output. How does ChatGPT know that its answer is complete? It generates its EOS token.

Worked example 10BOS and EOS in a generation task

PartTokens
Prompt<BOS> The cat sits on
Generatedthe mat <EOS> (the model stops here)
Result<BOS> shows the start of the prompt. When the model generates <EOS>, the generation is complete. With this simple method, decoder models give output with a correct length and stop at the correct point.
Watch outEncoder models (BERT) and decoder models (LLMs) use different sets of special tokens. Encoders use [CLS]/[SEP]/[MASK]/[PAD]. Decoders use [BOS]/[EOS] and similar tokens. These are only conventions. The lecturer said that you can make any special token, for example [FRANKENSTEIN]. If you train the model with it, the model can learn its meaning.
PracticeA batch has an input with 6 tokens and an input with 10 tokens. What does [PAD] do here? Why must attention ignore it?
AnswerThe model adds four [PAD] tokens to the shorter input. Then the two inputs have length 10, and the GPU can process them together as one rectangular batch. The pads have no content. Thus attention ignores them. If attention uses them, real tokens get meaningless padding vectors in their representations.

7. Step 2: self-attention and the encoder

Sources: slides 10–15, 19 · Video 1 (Overview of the transformer) · Lecture transcript (self-attention walkthrough, symmetry question, "fat and tall", Exercise 1)

Key ideaIn a transformer, each word in a sentence looks at each other word. It decides how much attention to give to each word (slide 10). Words attend to words in the same sequence. Thus the name is the self-attention mechanism. Self-attention lets the transformer find relationships, context and long-range dependencies. Earlier architectures could not do this well. The lecturer said that attention is "the magic beyond the previous line of deep neural networks". It makes tokens interact with each other.

7.1 The intuition: the attention matrix

For the sentence "The cat sat on the mat", self-attention makes a matrix. Each cell shows how much one word (row) attends to a different word (column). For example, cat gives high attention to sat, because they are subject and verb.

The cat sat on the mat The cat sat on the mat word paying attention (row) word being attended to (column); darker = more attention
An illustrative self-attention pattern for "The cat sat on the mat". The shading is only an example; in a real row, the weights have a sum of 1. Each row shows how one word divides its attention over the sentence. cat attends strongly to sat.

Where do these numbers come from? The in-class lecture gave five steps ("Self-attention: what is it really doing?"). If you know little about neural networks, read the five steps slowly. Each operation is in Section 2. Sheet "3 Attention" of the Transformer in a Spreadsheet workbook does the five steps with live formulas. With the Transformer by Hand worksheet, you calculate one block yourself.

7.2 The five steps of self-attention

STEP 1 one input vector per token xᵢ (Section 5) STEP 2 project xᵢ into three vectors qᵢ, kᵢ, vᵢ STEP 3 q seeks, k offers, v carries content STEP 4 score qᵢ·kⱼ/√dₖ, softmax each row → weights aᵢⱼ STEP 5 weighted sum of value vectors zᵢ = Σⱼ aᵢⱼ vⱼ one token → xᵢ → qᵢ, kᵢ, vᵢ → attention weights (row i) → one new output vector zᵢ
The five steps of self-attention from the lecture slide. Learn the one-line summary below the steps.

Step 1: one input vector for each token

Each token ii has one input vector xix_i. This vector is the sum of its token, segment and position embeddings from Section 5. For the toy sentence the · cat · sat: x1=xthe, x2=xcat, x3=xsatx_1 = x_{\text{the}},\ x_2 = x_{\text{cat}},\ x_3 = x_{\text{sat}}.

Step 2: project each input vector into q, k and v

The model has three learned weight matrices: WQW^Q, WKW^K and WVW^V. Training learns them, as it learns all other weights. The model multiplies the input vector of each token by each matrix. This is the vector × matrix operation of Section 2.3:

qi=xiWQki=xiWKvi=xiWV q_i = x_i W^Q \qquad k_i = x_i W^K \qquad v_i = x_i W^V
Where
xix_i
the input vector of token ii (a 1×d1 \times d row vector)
WQ,WK,WVW^Q, W^K, W^V
learned projection matrices, each d×dkd \times d_k (WVW^V can have a different output size dvd_v)
qiq_i
the query vector of token ii: what the token looks for
kik_i
the key vector of token ii: what the token offers for matching
viv_i
the value vector of token ii: the content that the token gives
d, dkd,\ d_k
the input dimension and the query/key dimension
Read it aloudThe query of token i is its input vector times the query weight matrix. Its key is the input vector times the key weight matrix. Its value is the input vector times the value weight matrix.
Formula in plain wordsOne input vector becomes three new vectors. Each new vector is a different learned "view" of the same token. The three matrices are different. Thus one token can offer one thing (its key), look for a different thing (its query) and give a third thing (its value). Put the qq vectors of all tokens as rows: this gives the matrix QQ. KK and VV are the same (each n×dkn \times d_k for nn tokens). Lowercase qi,ki,viq_i, k_i, v_i are vectors for one token. Uppercase Q,K,VQ, K, V are the matrices of all tokens. Training changes the matrices. At run time, the matrices are fixed, and the model applies them to each token in the same way.
x_cat × WQ (learned) × WK (learned) × WV (learned) q_cat k_cat v_cat what I am looking for what I offer to others how I represent myself
Step 2 for one token. Three learned projections change one input vector into a query, a key and a value.

Step 3: what are Q, K and V? (the lecturer's intuitions)

VectorSlide definitionLecturer's words
Q: querywhat this token looks for"What am I to myself; what am I really looking for?"
K: keywhat this token offers, what the model can match"What am I to others?"
V: valuethe information or content that this token carries"In the end, how do I represent myself?"
Watch outThis is abstract. The lecturer said so too ("that's also how the machine learning people are"). Why three roles? Because the model learns them. Separate control over "what I look for", "what I offer" and "what I give" works well in practice. The KV cache in LLM serving keeps these K and V vectors.

Step 4: calculate the attention weights

How much must token ii attend to token jj? Calculate the dot product of qiq_i and kjk_j. This shows how well "what I look for" matches "what you offer". Scale the result. Then apply softmax to each row. Thus the weights of each token have a sum of 1.

sij=qi⋅kjdk,aij=esij∑l=1nesil⟺A=softmax ⁣(QK⊤dk) s_{ij} = \frac{q_i \cdot k_j}{\sqrt{d_k}}, \qquad a_{ij} = \frac{e^{s_{ij}}}{\sum_{l=1}^{n} e^{s_{il}}} \qquad\Longleftrightarrow\qquad A = \text{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)
Where
qi⋅kjq_i \cdot k_j
the dot product of the query of token ii and the key of token jj (the raw score)
dkd_k
the dimension of the query and key vectors
sijs_{ij}
the scaled score of token ii for token jj
nn
the number of tokens in the sequence
ll
the index over all tokens in the softmax denominator
aija_{ij}
the attention weight: the share of the attention of token ii that goes to token jj
Q,KQ, K
all queries and all keys as rows (n×dkn \times d_k)
AA
the n×nn \times n attention weight matrix; row ii has ai1,…,aina_{i1}, \dots, a_{in}
Read it aloudThe score s i j is q i dot k j divided by the square root of d k. The weight a i j is e to the s i j, divided by the sum over each token l of e to the s i l. In matrix form, A is the row-wise softmax of Q times K transpose, divided by the square root of d k.
Formula in plain wordsThe dot product shows how well the query of token ii matches the key of token jj. One matrix multiplication, QK⊤QK^{\top}, calculates this for all pairs at the same time. The result is an n×nn\times n grid. The transpose only makes the shapes correct. The division by dk\sqrt{d_k} is a normalisation. Dot products become larger when the dimension increases. Without the scaling, softmax gives almost all weight to one token. Softmax then changes each row into weights between 0 and 1 with a sum of 1. A higher score gets a larger share, and the other tokens in the row get less. AA is the shaded grid of Section 7.1. The model does not use the values VV yet. To decide "how much do I care about you", the model needs only queries and keys.

Step 5: make the new vector for each token

The new, contextual representation of each token is the weighted sum of the value vectors of all tokens. The attention weights are the weights. The sum includes the value vector of the token itself.

zi  =  ∑j=1naij vj z_i \;=\; \sum_{j=1}^{n} a_{ij}\, v_j
Where
ziz_i
the new output vector of token ii
aija_{ij}
the attention weight of token ii on token jj (from Step 4)
vjv_j
the value vector of token jj
j, nj,\ n
the index over all nn tokens, also j=ij = i
Read it aloudz i equals the sum, over each token j, of the attention weight a i j times the value vector v j.
Formula in plain wordsThe lecturer said: "when I express myself, I take how much I attend to each token, times how that token represents itself, and add it all up." The weights have a sum of 1. Thus ziz_i is a weighted average of the value vectors. Tokens with a high weight move ziz_i toward their values. Tokens with a weight near 0 have almost no effect. The result is a vector with the same size. Thus it can go into the next layer. In this way, bank in "river bank" and bank in "closest bank" get different vectors: they attend to different neighbours.

The five steps together, in matrix form, give the famous formula from Attention Is All You Need:

Attention(Q,K,V)=softmax ⁣(QK⊤dk)V \text{Attention}(Q,K,V)=\text{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V
Where
Q,K,VQ, K, V
the query, key and value matrices (each with nn rows)
K⊤K^{\top}
the transpose of KK; thus QK⊤QK^{\top} is n×nn \times n
dk\sqrt{d_k}
the scaling factor from the key dimension
softmax(⋅)\text{softmax}(\cdot)
applied to each row separately
Attention(Q,K,V)\text{Attention}(Q,K,V)
the output matrix with nn rows; row ii is ziz_i
Read it aloudAttention of Q, K and V equals the softmax of Q times K transpose over the square root of d k, multiplied by V.
Formula in plain wordsThis line does Steps 4 and 5 for all tokens at the same time. The part in the softmax compares each query with each key. The softmax changes each row into weights. The multiplication by VV mixes the value vectors with these weights. The result is one new row ziz_i for each token. All operations are matrix multiplications. Thus a GPU can process the full sequence in parallel. A change to WQW^Q or WKW^K changes which tokens attend to which tokens. A change to WVW^V changes the content that the tokens give.

7.3 Self-attention by hand

Worked example 11Self-attention for the · cat · sat, every number calculated

The toy sentence (illustrative) has three tokens. The input vectors have 2 dimensions (dk=2d_k = 2): xthe=[1, 0], xcat=[0, 2], xsat=[1, 1] x_{\text{the}}=[1,\,0],\ x_{\text{cat}}=[0,\,2],\ x_{\text{sat}}=[1,\,1] . We select the projection matrices by hand. A real model learns them.

WQ=[1011]WK=[1102]WV=[1101] W^Q=\begin{bmatrix}1 & 0\\ 1 & 1\end{bmatrix}\qquad W^K=\begin{bmatrix}1 & 1\\ 0 & 2\end{bmatrix}\qquad W^V=\begin{bmatrix}1 & 1\\ 0 & 1\end{bmatrix}
1
Project. Each qi=xiWQq_i = x_i W^Q is a (1×2)(2×2)(1\times 2)(2\times 2) product (row · column, as in Section 2.3). For cat: qcat=[0,2] WQ=[0⋅1+2⋅1,    0⋅0+2⋅1]=[2, 2] q_{\text{cat}} = [0,2]\,W^Q = [0{\cdot}1 + 2{\cdot}1,\;\; 0{\cdot}0 + 2{\cdot}1] = [2,\,2] . All nine projections:
Tokenqi=xiWQ q_i = x_iW^Q ki=xiWK k_i = x_iW^K vi=xiWV v_i = x_iW^V
the[1, 0][1, 1][1, 1]
cat[2, 2][0, 4][0, 2]
sat[2, 1][1, 3][1, 2]
2
Raw scores QK⊤QK^{\top}: the dot product of each query with each key. The result is a 3×33\times 3 matrix.
qi⋅kj q_i \cdot k_j k(the) = [1,1]k(cat) = [0,4]k(sat) = [1,3]
q(the) = [1,0]101
q(cat) = [2,2]488
q(sat) = [2,1]345
For example, row cat, column sat: qcat⋅ksat=2×1+2×3=8 q_{\text{cat}}\cdot k_{\text{sat}} = 2{\times}1 + 2{\times}3 = 8 .
3
Scale by dk=2≈1.414 \sqrt{d_k} = \sqrt{2} \approx 1.414 . Row cat becomes [2.83,  5.66,  5.66] [2.83,\; 5.66,\; 5.66] .
4
Softmax of row cat:e2.83≈16.9e5.66≈286.4e5.66≈286.4sum≈589.7acat≈[ 0.03,  0.485,  0.485 ] \begin{aligned} e^{2.83} &\approx 16.9 \\ e^{5.66} &\approx 286.4 \\ e^{5.66} &\approx 286.4 \\ \text{sum} &\approx 589.7 \\ a_{\text{cat}} &\approx [\,0.03,\; 0.485,\; 0.485\,] \end{aligned} Do the same for the other rows. This gives the full attention matrix (each row has a sum of 1):
A≈[0.400.200.400.030.4850.4850.140.280.58]←the←cat←sat A \approx \begin{bmatrix}0.40 & 0.20 & 0.40\\ 0.03 & 0.485 & 0.485\\ 0.14 & 0.28 & 0.58\end{bmatrix}\begin{matrix}\leftarrow \text{the}\\ \leftarrow \text{cat}\\ \leftarrow \text{sat}\end{matrix}
keys (attended to) the cat sat q: the q: cat q: sat 0.400.200.40 0.030.4850.485 0.140.280.58 row = query token; each row sums to 1
The calculated attention weights. cat gives almost all its attention to cat and sat, and almost none to the.
5
Weighted sum of the values for cat:
zcat=0.03 [1,1]+0.485 [0,2]+0.485 [1,2]≈[ 0.52,  1.97 ] z_{\text{cat}} = 0.03\,[1,1] + 0.485\,[0,2] + 0.485\,[1,2] \approx [\,0.52,\; 1.97\,]
Resultzcat≈[0.52,1.97]z_{\text{cat}} \approx [0.52, 1.97]. The new vector of cat mixes mostly information from cat and sat. Thus its representation includes its context. Do the same for each token: zthe≈[0.80,1.60] z_{\text{the}} \approx [0.80, 1.60] and zsat≈[0.72,1.86] z_{\text{sat}} \approx [0.72, 1.86] . Now each token has information about the words near it.
Common trapA student asked in the lecture: "Is the attention matrix symmetric?" The answer is no. In the raw scores, the → cat is 0, but cat → the is 4. Attention shows pairwise relationships with a direction. The lecturer said: "I care about that guy; that guy doesn't care about me." Each token calculates its own distribution over all tokens. Formally, qi⋅kj≠qj⋅ki q_i \cdot k_j \neq q_j \cdot k_i usually, because WQ≠WK W^Q \neq W^K .

7.4 What makes a real transformer "fat" and "tall"

The five steps describe one attention head in one layer. Real transformers make this larger in two directions. The lecturer used these words:

DirectionWhat it meansEffect
"Fat": multi-head attentionIn one layer, the model has many independent copies of WQ,WK,WVW^Q, W^K, W^V. Each copy is an attention head with its own learned weights. Each head calculates its own attention pattern in parallel. For example, one head can find subject–verb links, and a different head can find adjacent words. The model combines their outputs.Each layer becomes wider
"Tall": stacked layersEach token goes in as a vector and comes out as a vector with the same dimension. Thus the model can apply the full block again: output of layer 1 → input of layer 2 → … BERT-base has 12 layers. Each layer makes the representations better.The model becomes deeper
x₁ x₂ x₃ input embeddings one encoder block, repeated ×N (BERT-base: N = 12) multi-head self-attention each head: own Q, K, V weights ("fat") add & layer norm (residual) multilayer perceptron (MLP) project up, then back down add & layer norm z₁ z₂ z₃ same count, same size → stackable ("tall") fed back in as the next layer's input
One encoder block, from the hand-drawn lecture slides: multi-head self-attention → add & layer norm → MLP → add & layer norm. The input shape is equal to the output shape. Thus the blocks can stack.

Two more parts follow the attention step in each block:

The explainer The encoder backbone, and why it cannot generate shows how a stack of these layers changes one vector for each token. Sheet "4 Multi-Head" of the spreadsheet workbook shows two heads side by side.

7.5 What the encoder produces, and why GPUs are good for it

After the last layer, the encoder gives one encoded vector for each input token. The count and the dimension are the same as the input. Each vector now represents "the semantics of the word with respect to the sentence". For one vector for the full sentence, there are two standard methods. Use the output vector of the [CLS] token (train the model to put a summary of the sentence there). Or use the mean of all token vectors (mean pooling). Retrieval uses the two methods frequently (Section 12 and Week 5).

encoder output: one vector per token z[CLS] z the z cat z sat z[SEP] [CLS] pooling: keep z[CLS] mean pooling: average all z
Two standard sentence vectors: the [CLS] output vector, or the mean of all token output vectors.
vsentence  =  1n∑i=1nzi(mean pooling)orvsentence=z[CLS] v_{\text{sentence}} \;=\; \frac{1}{n}\sum_{i=1}^{n} z_i \qquad\text{(mean pooling)}\qquad\text{or}\qquad v_{\text{sentence}} = z_{[\text{CLS}]}
Where
ziz_i
the final output vector of the encoder for token ii
nn
the number of tokens in the mean (without padding tokens)
z[CLS]z_{[\text{CLS}]}
the final output vector of the [CLS] token
vsentencev_{\text{sentence}}
one vector for the full text, with the same dimension as each ziz_i
Read it aloudThe sentence vector is one over n times the sum of all n token output vectors (mean pooling). Or, it is the output vector of the CLS token.
Formula in plain wordsMean pooling adds the token vectors position by position and divides by the count. Thus each token has the same effect, and the result is at the centre of the token vectors of the sentence. [CLS] pooling uses one special token. [CLS] attends to each token in each layer. Thus training can teach it to be a summary. The two methods give one vector for the sentence. The model can compare this vector with the vector of a different text with a dot product or cosine similarity. The better method depends on the training of the model.

Worked example 12A sentence vector by mean pooling

Use the three output vectors from Worked example 11: zthe≈[0.80,1.60]z_{\text{the}} \approx [0.80, 1.60], zcat≈[0.52,1.97]z_{\text{cat}} \approx [0.52, 1.97], zsat≈[0.72,1.86]z_{\text{sat}} \approx [0.72, 1.86].

v=13([0.80,1.60]+[0.52,1.97]+[0.72,1.86])=13[2.04, 5.43]≈[0.68, 1.81] \begin{aligned} v &= \frac{1}{3}\big([0.80,1.60] + [0.52,1.97] + [0.72,1.86]\big) \\ &= \frac{1}{3}[2.04,\ 5.43] \\ &\approx [0.68,\ 1.81] \end{aligned}
ResultThe single vector [0.68,1.81][0.68, 1.81] represents the sentence "the cat sat". The model can compare it with a query vector.
Key ideaParallelism (in-class Exercise 1): the encoder processes all input tokens at the same time. The model puts all dot products into large matrix multiplications. GPUs are designed for this work. A CPU calculates in sequence. A GPU does thousands of small calculations at the same time. Thus transformers and GPUs helped each other to succeed.
PracticeIn Worked example 11, calculate the raw score of sat for the, and the raw score of the for sat. Are the two scores equal?
Use qsat=[2,1]q_{\text{sat}} = [2,1], kthe=[1,1]k_{\text{the}} = [1,1], qthe=[1,0]q_{\text{the}} = [1,0], ksat=[1,3]k_{\text{sat}} = [1,3].
Answerqsat⋅kthe=2+1=3qthe⋅ksat=1+0=1 \begin{aligned} q_{\text{sat}}\cdot k_{\text{the}} &= 2 + 1 = 3 \\ q_{\text{the}}\cdot k_{\text{sat}} &= 1 + 0 = 1 \end{aligned} The scores are different. This shows that attention has a direction: queries and keys come from different learned matrices.
PracticeWhy can BERT-base repeat the same block 12 times? What breaks if the MLP gives a vector with a different size?
AnswerEach block receives nn vectors with 768 dimensions and gives nn vectors with 768 dimensions. Thus the output of each block is a valid input for the next block. If the MLP changes the size, the residual addition and the WQ,WK,WVW^Q, W^K, W^V of the next layer (which need inputs with 768 dimensions) do not fit the shapes. Then the stack breaks.

8. Steps 3–4: the decoder and generation

Sources: slides 16–19, 23 · Video 1 (Overview of the transformer) · Video 7 (Autoregressive decoders) · Lecture transcript (decoder, linear-layer Q&A, sampling)

The lecturer said: "Don't worry too much about the decoder; it's almost the same." The decoder uses the same parts as the encoder. It tokenises the input (your prompt), makes input embeddings, and uses stacked blocks of self-attention + MLP. Two things are different. The two differences exist because the task of the decoder is generation.

8.1 Difference 1: masked (causal) attention

In the decoder, each token can attend only to itself and earlier tokens. During generation, the future tokens do not exist yet. When the model generates the first word of its answer, there is no word after it. The lecturer said: "It's not that we don't want it; we don't need it. How could it look at future tokens? They're not yet generated." In practice, the model sets the scores for future positions to −∞-\infty before the softmax. This removes the upper-right triangle of the attention matrix (compare the two grids in Section 9).

sijmask={qi⋅kjdkif j≤i−∞if j>iaij=esijmask∑l=1nesilmask s^{\text{mask}}_{ij} = \begin{cases} \dfrac{q_i\cdot k_j}{\sqrt{d_k}} & \text{if } j \le i \\[6pt] -\infty & \text{if } j > i \end{cases} \qquad a_{ij} = \frac{e^{s^{\text{mask}}_{ij}}}{\sum_{l=1}^{n} e^{s^{\text{mask}}_{il}}}
Where
ii
the position of the token that attends
jj
the position that gets attention
j≤ij \le i
“itself or earlier”: position jj is position ii or a position before it
qi⋅kj/dkq_i\cdot k_j/\sqrt{d_k}
the usual scaled score from Section 7
−∞-\infty
a score that becomes e−∞=0e^{-\infty} = 0 after the exponential
sijmasks^{\text{mask}}_{ij}
the masked score
aija_{ij}
the attention weight (0 for each future position)
Read it aloudThe masked score s i j equals q i dot k j over the square root of d k when j is at or before i. It equals minus infinity when j is after i. The weight a i j is the softmax of the masked scores across row i.
Formula in plain wordsFor past and present tokens, the mask keeps the usual attention calculation. For each future token, the mask makes the weight exactly 0, because e−∞=0e^{-\infty} = 0. The softmax normalises the other weights in each row. Thus their sum is still 1. The first token can only attend to itself (weight 1). The last token sees the full sequence. Thus its row is the same as the unmasked row of the encoder. With this mask, training teaches the model to predict each next token from the past only. This is the same situation as in generation.

Worked example 13The causal mask on the · cat · sat

Use the scaled scores from Worked example 11. Mask each future position.

1
Row the: [0.71,−∞,−∞][0.71, -\infty, -\infty] → softmax [1,0,0][1, 0, 0]. The first token attends only to itself.
2
Row cat: [2.83,5.66,−∞][2.83, 5.66, -\infty]. Apply softmax to the two unmasked scores:e2.83≈16.9e5.66≈286.4sum≈303.3acat≈[0.056,0.944,0] \begin{aligned} e^{2.83} &\approx 16.9 \\ e^{5.66} &\approx 286.4 \\ \text{sum} &\approx 303.3 \\ a_{\text{cat}} &\approx [0.056, 0.944, 0] \end{aligned}
3
Row sat: the mask removes nothing (it is the last token). Thus the row stays [0.14,0.28,0.58][0.14, 0.28, 0.58].
keys (attended to) the cat sat q: the q: cat q: sat 1 masked masked 0.056 0.944 masked 0.14 0.28 0.58 masked = future token, weight exactly 0
With the causal mask, each token divides its attention over itself and earlier tokens only.
4
New vectors:zthe=1⋅vthe=[1,1]zcat=0.056 [1,1]+0.944 [0,2]≈[0.06,1.94]zsat≈[0.72,1.86](as before) \begin{aligned} z_{\text{the}} &= 1\cdot v_{\text{the}} = [1, 1] \\ z_{\text{cat}} &= 0.056\,[1,1] + 0.944\,[0,2] \approx [0.06, 1.94] \\ z_{\text{sat}} &\approx [0.72, 1.86] \quad \text{(as before)} \end{aligned}
ResultThe masked attention matrix is [1000.0560.94400.140.280.58]\begin{bmatrix}1 & 0 & 0\\ 0.056 & 0.944 & 0\\ 0.14 & 0.28 & 0.58\end{bmatrix}. Only the rows of earlier tokens change. cat cannot use sat now. Thus its vector [0.06,1.94][0.06, 1.94] shows only "the cat".

8.2 Difference 2: the output head, from vector to next-token probabilities

The encoder ends with vectors. The decoder must end with a selection of the next token. The lecturer wrote the full chain on paper for a student. It is one matrix multiplication and one softmax:

1
After the last block, take the last-layer vector of the last token, hh. Its dimension is nn (BERT: 768; Llama: 2,048).
2
Multiply hh by a learned linear projection layer WW with the shape n×mn \times m. Here, mm is the vocabulary size (approximately 30,000): (1×n)(n×m)=(1×m)(1\times n)(n \times m) = (1\times m). The result is one raw score (a logit) for each token in the vocabulary.
3
Apply softmax. This changes the logits into probabilities between 0 and 1 with a sum of 1.
4
Select the next token from this distribution.
p  =  softmax(h W),h⏟1×n  W⏟n×m=ℓ⏟1×m p \;=\; \text{softmax}\big(h\,W\big), \qquad \underbrace{h}_{1\times n}\;\underbrace{W}_{n\times m} = \underbrace{\ell}_{1\times m}
Where
hh
the final-layer hidden vector of the last token (dimension nn)
WW
the learned output projection ("linear layer"), n×mn \times m
mm
the vocabulary size (approximately 30,000 for BERT-style vocabularies)
ℓ=hW\ell = hW
the logits: one raw score for each vocabulary token
pp
the next-token probability distribution
pwp_w
the probability of vocabulary token ww
Read it aloudp equals the softmax of h times W. A one by n vector h times an n by m matrix W gives a one by m vector of logits.
Formula in plain wordsEach column of WW is a learned "template" for one vocabulary token. The dot product of hh with this column shows how well the context fits this token as the next token. Softmax changes the mm scores into a probability distribution. A token with a column in the same direction as hh gets a high logit and a large probability. Negative logits become small probabilities. This is the vector × matrix operation of Worked example 4, but with approximately 30,000 outputs and not 3.
h (1 × n) 0.9 −0.2 0.4 last token's last-layer vector × W (n × m) learned linear projection m = vocab (~30K) = logits (1 × m): one score per token 2.0the 1.0a 0.5cat −1.0pie 0.0… softmax probabilities (sum to 1) 0.56 0.21 0.13 0.03 0.08 the a cat pie … ↑ picked (greedy)
The output head of the decoder. One vector × matrix multiplication maps the hidden vector onto the full vocabulary. Softmax changes the scores into next-token probabilities.

Worked example 14The output head with numbers

The vocabulary (illustrative) has 5 tokens (the, a, cat, pie, …). The logits are [2.0, 1.0, 0.5, −1.0, 0.0][2.0,\ 1.0,\ 0.5,\ -1.0,\ 0.0].

1
Calculate the exponentials: e2.0=7.39e1.0=2.72e0.5=1.65e−1.0=0.37e0.0=1.00sum=13.13 \begin{aligned} e^{2.0} &= 7.39 \\ e^{1.0} &= 2.72 \\ e^{0.5} &= 1.65 \\ e^{-1.0} &= 0.37 \\ e^{0.0} &= 1.00 \\ \text{sum} &= 13.13 \end{aligned}
2
Divide by the sum: P=[ 0.56,  0.21,  0.13,  0.03,  0.08 ] P = [\,0.56,\; 0.21,\; 0.13,\; 0.03,\; 0.08\,] .
ResultThe token with the highest probability is the (0.56). With greedy decoding, it is the next token. Softmax has no problem with the negative logit of pie: it becomes a small probability (0.03).

8.3 The generation loop and sampling

One pass through the model gives one token. To generate text, the model adds the predicted token to the input and does the full process again ("generate, put it back, generate, put it back"). The loop stops when the model gives the [EOS] token. Section 13 calls this autoregressive generation.

Watch outFrom the lecture Q&A: if the model always selects the most probable token (greedy decoding), problems can occur. For example, the text can repeat. Real systems use sampling strategies. For example, they keep only the top tokens above a probability threshold and select one of them at random. This random selection changes the context for all next tokens. Thus ChatGPT gives a different answer each time to the same question.
PracticeA model has a hidden size of 2,048 and a vocabulary of 32,000 tokens. What is the shape of the output projection? How many numbers does it give at each step? Why does the model use only the vector of the last token?
AnswerWW is 2048×320002048 \times 32000. hWhW gives 32,000 logits (one for each vocabulary token). Softmax changes them into 32,000 probabilities. The model uses only the vector of the last token for two reasons. With the causal mask, only this position attends to the full prompt. Also, the next token comes after the last token.

9. Encoder vs decoder

Sources: slides 20–23 · Video 2 (Encoder vs decoder) · Lecture transcript (using the parts separately, Exercise 1)

In the original transformer, the encoder and the decoder work together as one system. The encoder changes text into vector representations. The decoder uses these vectors and the ⟨start⟩ token to predict the next token. Slides 20–21 give the important point: you can use the two parts separately. The lecturer asked two questions: "If my task is not generation, do I need the decoder? If my task is only generation, do I need the encoder?" The answer "no" to each question gave some of the most important advances in AI:

Encoder: "understand and represent"Decoder: "generate and predict"
PurposeReceives input text. Makes a dense vector representation. Understands context and relationships.Receives input (optionally with encoded representations). Generates output step by step. Predicts next-token probabilities.
AttentionBidirectional: each token can see all tokens before and after it. The encoder sees the full sentence at the same time.Masked (unidirectional): a token sees only earlier words
ExampleInput How to fix laptop? → output dense vector [0.23, −0.45, 0.67, …]. The vector holds the meaning of the sentence.Input UQ is (+ encoded vectors, optional) → output the → University → of → Queensland → …
ApplicationsSearch and classification with embeddings. These tasks need the full meaning of the text.Translation, text generation, and all modern LLM use
Encoder: bidirectional Decoder: masked (causal) howtofixlaptop? UQistheUni…of howtofixlaptop? UQistheUni…of every token sees every token (before and after) grey = masked future tokens
Attention patterns. Left: an encoder token attends to the full sequence (bidirectional). Right: a decoder token attends only to itself and earlier tokens (masked, or causal, attention).
Key ideaEncoders are good for search and classification. They make one deep representation of the full text at the same time. To find the meaning of a sentence, each token must see the tokens before it and after it. In "the bank makes loans to clients", the later token loans shows which bank the sentence means. Decoders are good for generation. They predict the next token, one token at a time. The core technology is the same, but it serves different purposes.
PracticeYou must embed ten million product descriptions for semantic search. You must also write product summaries. Which part of the transformer is correct for each task? Which attention pattern does each part use?
AnswerEmbedding for search is an understanding task. Use an encoder. It uses bidirectional attention: each token sees the full description, and the model processes all tokens in parallel. Summary writing is generation. Use a decoder. It uses masked attention and generates one token at a time until [EOS].

10. How transformers are trained

Sources: slides 41, 44, 53 · Video 6 (BERT) · Video 7 (Autoregressive decoders) · Lecture transcript (training, data and lawsuits)

A new model with random weights knows nothing. The embedding tables, the WQ,WK,WVW^Q, W^K, W^V matrices in each head of each layer, the MLP weights and the output projection all start as random numbers. Training changes all these weights until the model "understands" language. The practical problem: training needs a very large amount of training signal without human labels. BERT and GPT use the same solution: self-supervised learning. Use the full web as training data, and let the text give the labels.

Encoder (BERT): masked LM The bank [MASK] loans to clients predict: makes uses context on both sides Decoder (GPT): next-token prediction UQ→ is UQ is→ the UQ is the→ University UQ is the University→ of uses only the tokens before
The two objectives change raw web text into labelled examples at no cost. The encoder hides a word and predicts it. The decoder predicts each next word from the words before it.
ObjectiveHow it worksLecture example
Masked language modelling (encoder, BERT-style)Take web text. Replace some tokens at random with [MASK]. Train the model to predict the masked tokens from all the context around them. The context is on the two sides, thus bidirectional attention is important. Compare the prediction with the real token and measure the error (loss). Change the weights a little to make the loss smaller. Repeat over the full web. Section 11 gives the details and a loss calculation."The bank [MASK] loans to clients" → predict makes
Next-token prediction (decoder, GPT-style)This is simpler. Give the first token: can the model predict the second token? Give the first two tokens: predict the third token. Give the first three tokens: predict the fourth token. Each position of each sentence on the web becomes a training example.If you train a model on your chat history, it learns your next reply ("if you've written bullshit a thousand times, it knows the next token is bullshit"). Nobody teaches grammar to the model. It learns how people write because it predicts their text, billions of times.

Worked example 15One sentence, two training signals

Use the sentence UQ is the University of Queensland. It has 6 tokens.

1
Next-token prediction: each prefix predicts the next token: UQ → is, UQ is → the, UQ is the → University, … University → of, … of → Queensland. One sentence gives 5 labelled examples.
2
Masked LM at 15%: 0.15×6=0.90.15 \times 6 = 0.9. Thus the model masks approximately one token, for example UQ is the [MASK] of Queensland. It predicts University from the two sides.
ResultThe two signals need no human annotator. The text itself gives the labels. Next-token prediction gets a training target at each position. Masked LM gets targets only at masked positions, but each prediction uses context from the two directions.
Watch outThe lecture gave a side effect of this need for data. Model quality increases with training data. Thus companies collect all the data that they can find. Because of this, publishers sued some LLM providers about copyrighted training data. High-quality data, and not only more data, is one reason for the success of these models.
Exam angleThe two methods are unsupervised for labels. No person wrote relevance judgements or translation pairs. But the weights after training hold meaning, synonymy and context. This is the semantic understanding that BM25 does not have (Section 1). Weeks 5–8 transfer this understanding to ranking with a small amount of labelled data (fine-tuning).
PracticeWhy is next-token prediction a bad pre-training objective for an encoder with bidirectional attention?
What can a bidirectional token see when it tries to predict the next token?
AnswerWith bidirectional attention, each position can already see the next token. Thus the "prediction" is only a copy, and the model learns nothing useful. Masking hides the target from all positions. Thus the model must use the context. Decoders solve the problem in the opposite way: the causal mask hides the future. Thus next-token prediction stays a real prediction task.

11. BERT

Sources: slides 40–48, 50 · Video 6 (BERT) · Lecture transcript (MLM masking rate, scaling)

Key ideaBERT = Bidirectional Encoder Representations from Transformers. Google released it in October 2018 (Devlin, Chang, Lee and Toutanova, NAACL 2019). It is an encoder-only transformer (no decoder) with 12 layers (BERT-base) and bidirectional attention. It sees the full sentence at the same time. BERT changed 11 NLP tasks. It gave state-of-the-art effectiveness on many of them, also on information retrieval.

11.1 The two-stage training approach

STAGE 1 · Pre-training self-supervised, on huge amounts of UNLABELLED text (Wikipedia, web) learns general language understanding STAGE 2 · Fine-tuning supervised, on SMALL amounts of LABELLED task examples specialises: classification, ranking, QA …
First, the model learns general language understanding from unlabelled text. Then it adapts to a specific task with a small number of labelled examples. This approach was new at the time.

11.2 Ingredients and input processing

The success of BERT comes from three pre-training ingredients (slide 42). (1) An encoder-only transformer architecture with many parameters. (2) Very large amounts of text data (Wikipedia, web pages). (3) Very large computing power (cloud TPU pods were necessary at the time).

The input starts with the embedding layers of Section 5: token + segment + position embeddings, added together. Then the input goes through many transformer encoder layers with self-attention (Section 7). The output is a sequence of vector representations, one for each input token. Each token attends to each other token.

11.3 Pre-training task 1: masked language modelling (MLM)

The model masks approximately 15% of the input tokens at random. It learns to predict them from all the context around them (in the two directions). The BERT paper and slide 50 use 15%. In class, the lecturer said 30% from memory. The exact fraction is a design choice. The principle is the important part. The loss measures how well BERT predicts the masked tokens:

LMLM  =  −∑i ∈ Mlog⁡P(xi ∣ x∖M) \mathcal{L}_{\text{MLM}} \;=\; -\sum_{i\,\in\,M}\log P\big(x_i \,\big|\, x_{\setminus M}\big)
Where
MM
the set of masked positions in the sentence
xix_i
the true (original) token at masked position ii
x∖Mx_{\setminus M}
the sentence with the masked positions hidden: all the visible context
P(xi∣x∖M)P(x_i \mid x_{\setminus M})
the probability that BERT gives to the correct token at position ii, for this context
log⁡\log
natural logarithm
LMLM\mathcal{L}_{\text{MLM}}
the masked language modelling loss for this sentence (lower is better)
Read it aloudThe MLM loss equals minus the sum, over each masked position i, of the log of the probability of the true token x i for the unmasked rest of the sentence.
Formula in plain wordsFor each hidden word, find the probability that the model gave to the correct word. Take minus the log of this probability. A correct prediction with high confidence (probability near 1) gives a very small loss. A bad prediction (probability near 0) gives a large loss, because −log⁡p-\log p increases quickly when pp decreases. The sum over the masked positions gives the loss for the sentence. Training changes the weights to make this loss smaller over the full web. Thus the model must learn a deep understanding of language and context.

Worked example 16MLM loss on the lecture's sentence

The original sentence (slide 47): "The Mongol invasion of Europe in the 13th century was the conquest of much of Europe by the Mongol Empire." BERT masks century and Europe: "… in the 13th [MASK] was the conquest of much of [MASK] by the Mongol Empire." With bidirectional context, it predicts:

Masked positionPredictionConfidence PPLoss contribution −log⁡P-\log P
… 13th [MASK] was …century0.94−ln⁡0.94≈0.062 -\ln 0.94 \approx 0.062
… much of [MASK] by …Europe0.97−ln⁡0.97≈0.030 -\ln 0.97 \approx 0.030
LMLM≈0.062+0.030=0.092 \mathcal{L}_{\text{MLM}} \approx 0.062 + 0.030 = 0.092
ResultThe total loss is approximately 0.092. This is very low, because the model is confident and correct. At the start of training, the model can give the correct word only P=0.10P = 0.10. This gives −ln⁡0.10≈2.30-\ln 0.10 \approx 2.30 for each mask. This large loss makes the weights better. The high confidences (94%, 97%) show that the trained model knows the historical and geographical relationships.

11.4 Pre-training task 2: next sentence prediction (NSP)

BERT also trains on pairs of sentences. It learns to predict if the second sentence follows naturally after the first sentence. This is a binary classification on the [CLS] output. It teaches relationships between sentences. These relationships are important for question answering and document ranking.

Sentence ASentence BLabel
My dog is cute.He likes playing.IsNext
My dog is cute.The stock market fell 2% today.NotNext

11.5 Model size is important

ModelParametersConfiguration
BERT-base110 million12 layers
BERT-large345 millionmore layers, larger hidden dimension, more attention heads

BERT-large is always better than BERT-base, also for ranking and for small tasks. Also, more training steps give higher accuracy (slide 46). This was an early sign of the scaling laws of today's large language models. The lecturer said that models with a billion parameters are now small, and the largest models have trillions. But the industry must now balance size against efficiency ("a user says hello, and answering hi costs me $20; I can't do that").

11.6 Fine-tuning to different tasks

One input: classify [CLS] one text [SEP] BERT class label from [CLS] Two inputs: QA, ranking [CLS] q [SEP] d [SEP] BERT relevance score from [CLS] Per-token labelling [CLS] w₁ w₂ w₃ [SEP] BERT tag tag tag
The fine-tuning patterns of slide 48. Sentence-level tasks read the [CLS] output. Tasks with two inputs put the texts in different segments. Sequence labelling reads the output of each token.

11.7 The limits of BERT

LimitDetail (slide 50)
Input length limitThe maximum input is 512 tokens, with the special tokens. BERT cannot process long documents directly. They need chunking or truncation, and some information is lost.
No autoregressive generationMLM pre-training masks approximately 15% of the tokens at random and predicts them with bidirectional context. BERT does not train on next-token prediction. Thus it cannot make coherent sequences token by token, as GPT does. (Section 15 gives a modern solution: diffusion language models generate text with encoder backbones.)

Decoder-only models such as GPT solve the second limit (Section 13).

PracticeDuring training, BERT gives the correct masked word a probability of 0.5 at one position and 0.05 at a different position. Calculate each loss contribution. Which position has more effect on learning?
Answer−ln⁡0.5≈0.69−ln⁡0.05≈3.00 \begin{aligned} -\ln 0.5 &\approx 0.69 \\ -\ln 0.05 &\approx 3.00 \end{aligned} The second position gives more than four times the loss. Thus its error gives the larger weight changes. The loss gives a much larger penalty to a confident failure than to small uncertainty.

12. BERT in information retrieval

Sources: slide 49 · Video 6 (BERT) · Lecture transcript (ranking challenge: [CLS] or mean vectors, ColBERT-style interaction)

IR uses BERT in two main ways (slide 49). The two ways make search quality much better. Week 5 covers dense retrievers. Week 8 covers BERT-based rerankers.

Dense retriever (Week 5) Query Document BERT BERT V(query) V(doc) similarity: cos(V(q), V(d)) encoded separately: fast, indexable Reranker, cross-encoder (Week 8) Query + [SEP] + Document BERT relevance score from [CLS] read together: more accurate, but one BERT run per (q, d) pair
Two ways to use BERT for search. Left: the dense retriever (bi-encoder) encodes the query and the documents separately and compares the vectors. Right: the reranker (cross-encoder) joins the query and the document with [SEP]. The classification head of BERT then predicts relevance directly.
Dense retriever (bi-encoder)Reranker (cross-encoder)
InputThe model encodes the query and the document separately[CLS] query [SEP] document [SEP], read together (segments A and B, Section 5)
ScoreCosine similarity or dot product of V(q)V(q) and V(d)V(d)Classification score from the [CLS] head
CostThe model calculates the document vectors one time, at indexing time. Search is fast.One BERT run for each (query, document) pair. Use it to rerank a short list of candidates.
AccuracyGoodHigher: each query token attends to each document token
CourseWeek 5Week 8 (this is the monoBERT style of reranker)

From the in-class discussion: for the dense retriever, the sentence vector V(⋅)V(\cdot) usually comes from the [CLS] output vector or from the mean of all token output vectors (Section 7.5). The relevance score is a cosine similarity or dot product. The lecturer said: "either way, you can have a score, as long as you train it." A student gave a more advanced method. Keep one vector for each token of the query and of the document. Compare each query token with each document token (a full similarity matrix, as in attention). Then combine the similarities into a final score. This is the idea of ColBERT-style late interaction. It is more accurate than one vector, but it must store many vectors for each document (later weeks).

cos⁡(q,d)=q⋅d∥q∥ ∥d∥=∑iqidi∑iqi2 ∑idi2 \cos(q,d)=\frac{q\cdot d}{\lVert q\rVert\,\lVert d\rVert} = \frac{\sum_{i} q_i d_i}{\sqrt{\sum_i q_i^2}\,\sqrt{\sum_i d_i^2}}
Where
q,dq, d
the query vector V(q)V(q) and the document vector V(d)V(d) from BERT
q⋅dq\cdot d
their dot product (Section 2.1)
qi,diq_i, d_i
entry ii of each vector
∥q∥,∥d∥\lVert q\rVert, \lVert d\rVert
the lengths of the vectors (Euclidean norms)
cos⁡(q,d)\cos(q,d)
the cosine similarity, between −1 and 1
Read it aloudThe cosine of q and d equals q dot d divided by the length of q times the length of d. This is the sum of q i times d i, over the square root of the sum of q i squared times the square root of the sum of d i squared.
Formula in plain wordsThis is the cosine of the vector space model in Week 3. Here, it uses learned dense vectors and not term-weight vectors. The numerator gives a high value when the large entries of the two vectors align. The division by the two lengths removes the effect of the vector size. Thus only the direction is important. A value near 1 means that the texts have the same direction in the meaning space. A value near 0 means that they are not related. BERT puts synonyms near each other. Thus two texts can get a high score with no shared words. If all vectors have length 1, cosine and dot product give the same ranking.

Worked example 17Dense retrieval fixes the fix-laptop query

BERT encodes the query and two documents into vectors with 3 dimensions (illustrative values):

TextVector
qq = V("how to fix laptop")[0.90, 0.30, 0.10]
d1d_1 = V("computer repair guide for damaged notebooks")[0.80, 0.40, 0.00]
d2d_2 = V("river bank fishing spots")[0.10, 0.20, 0.95]
1
Document 1:q⋅d1=0.9(0.8)+0.3(0.4)+0.1(0)=0.84∥q∥=0.91≈0.954∥d1∥=0.80≈0.894cos⁡(q,d1)=0.840.954×0.894≈0.985 \begin{aligned} q\cdot d_1 &= 0.9(0.8)+0.3(0.4)+0.1(0) = 0.84 \\ \lVert q\rVert &= \sqrt{0.91}\approx 0.954 \\ \lVert d_1\rVert &= \sqrt{0.80}\approx 0.894 \\ \cos(q,d_1) &= \frac{0.84}{0.954\times 0.894}\approx 0.985 \end{aligned}
2
Document 2:q⋅d2=0.09+0.06+0.095=0.245∥d2∥=0.9525≈0.976cos⁡(q,d2)=0.2450.954×0.976≈0.263 \begin{aligned} q\cdot d_2 &= 0.09+0.06+0.095 = 0.245 \\ \lVert d_2\rVert &= \sqrt{0.9525}\approx 0.976 \\ \cos(q,d_2) &= \frac{0.245}{0.954\times 0.976}\approx 0.263 \end{aligned}
ResultThe ranking is d1≻d2d_1 \succ d_2, with a large difference (0.985 and 0.263). But d1d_1 shares zero terms with the query. The BERT embeddings put fix/repair and laptop/notebook near each other. Thus BERT does the semantic matching that BM25 cannot do (Worked example 1). Week 5 (dense retrieval) uses this idea.
PracticeA collection has 8 million passages. The system must answer queries in less than 100 ms. Where does each BERT method go in the pipeline, and why?
AnswerUse the dense retriever (or BM25) for the first-stage retrieval. The system calculates the passage vectors one time, at indexing time. Thus a query needs one BERT pass and a fast vector search. Use the cross-encoder reranker only on the top candidates (for example, 100). It needs one BERT pass for each (query, passage) pair. This is much too slow for 8 million passages. But it reads the two texts together, and thus it gives a more accurate final order.

13. Autoregressive decoders and GPT

Sources: slides 51–54 · Video 7 (Autoregressive decoders)

Key ideaAn autoregressive text-generation model works in time steps. At each time step tt, the model takes the sequence of earlier tokens y<ty_{<t} as input. It gives a new token y^t\hat{y}_t as output (slide 51). Generation occurs step by step. The model generates one token, adds it to the sequence, and uses the longer sequence to generate the next token. Today's LLMs use this loop to make coherent text. Each step is the linear layer + softmax of Section 8.
y^t=arg⁡max⁡y∈V  P(y ∣ y<t) \hat{y}_t=\arg\max_{y \in \mathcal{V}}\;P\big(y \,\big|\, y_{<t}\big)
Where
tt
the current time step (the position that the model generates)
y<ty_{<t}
all tokens before step tt: the prompt and all tokens that the model generated before
V\mathcal{V}
the vocabulary
yy
one candidate token from the vocabulary. The arg max tries each token.
P(y∣y<t)P(y \mid y_{<t})
the probability from the model that yy is the next token (the softmax output of Section 8)
arg⁡max⁡\arg\max
"the yy that gives the largest value"
y^t\hat{y}_t
the token that the model selects at step tt (hat = predicted)
Read it aloudy hat t is the vocabulary token y that gives the largest probability of y for all the tokens before t.
Formula in plain wordsThis is greedy decoding. At each step, the model selects the most probable next token for the text before it. The selected token then becomes part of y<t+1y_{<t+1} for the next step. Thus an early selection has an effect on all later tokens. Random sampling from the top candidates (Section 8.3) can replace the argmax. Then the outputs are more varied, but not always the same.
P(y1,…,yT)=∏t=1TP(yt ∣ y<t) P(y_1,\dots,y_T)=\prod_{t=1}^{T}P\big(y_t \,\big|\, y_{<t}\big)
Where
y1,…,yTy_1,\dots,y_T
a full sequence of TT tokens
TT
the length of the sequence
P(yt∣y<t)P(y_t \mid y_{<t})
the probability of token tt for the tokens before it
∏t=1T\prod_{t=1}^{T}
multiply the TT step probabilities together
Read it aloudThe probability of the full sequence y 1 to y T equals the product, over t from 1 to T, of the probability of y t for all the tokens before it.
Formula in plain wordsThis is the chain rule of probability. The probability of a full sentence is the probability of the first token, times the probability of the second token after the first, and so on. Thus next-token prediction is sufficient to model full texts. Each factor is 1 or less. Thus longer sequences get smaller probabilities, and one improbable token makes the full product small.
y₁ … yₜ₋₁ (previous tokens) context so far Generative LLM decoder-only transformer ŷₜ (next token) ŷₜ is appended to the context and the loop repeats (until [EOS])
The autoregressive loop. Each generated token becomes part of the input for the next token.

Worked example 18Autoregressive generation of "UQ is …"

The prompt is UQ is. At each step, the decoder gives a distribution over the vocabulary and selects the top token (greedy). The probabilities are illustrative:

StepInput sequenceTop candidates P(yt∣y<t) P(y_t\mid y_{<t}) Selected
1UQ isthe 0.62 · a 0.15 · ranked 0.08 · …the
2UQ is theUniversity 0.87 · best 0.05 · …University
3UQ is the Universityof 0.95 · in 0.02 · …of
4UQ is the University ofQueensland 0.98 · Qld 0.01 · …Queensland

The probability of the generated text is the product of the step probabilities (chain rule):

P(the, University, of, Queensland∣UQ is)=0.62×0.87×0.95×0.98≈0.502 P(\text{the, University, of, Queensland}\mid \text{UQ is}) = 0.62\times 0.87\times 0.95\times 0.98 \approx 0.502
ResultThe text "the University of Queensland" has a probability of approximately 0.50. Compare with BERT. At step 2, the model sees only UQ is the. It never sees a future token (masked attention). BERT sees the full sentence at the same time, but it cannot generate it.

13.1 GPT-3 and ChatGPT

OpenAI released GPT-3 in June 2020 (slide 52). It is a decoder-only transformer (no encoder) with 175 billion parameters. This is approximately 1,000× larger than BERT-base. It uses unidirectional attention: only earlier tokens are visible. The first release had a context window of 2,048 tokens. The example on the slide: the prompt "Recite the first law of robotics" → GPT-3 generates, token by token, "A robot may not injure a human …". In 2022, OpenAI released ChatGPT with a very similar architecture. ChatGPT started the current AI revolution. It made generative models (LLMs) very popular in the world.

13.2 Training: next-token prediction gives in-context learning

GPT models train with simple next-token prediction (Section 10) on very large amounts of text. This simple objective gives in-context learning. The model learns to do tasks only from examples in the prompt: arithmetic, spelling correction, translation. It needs no training for the specific task (Brown et al., 2020, "Language models are few-shot learners", slide 53).

Worked example 19In-context learning with a few-shot prompt

RoleText
Promptsea otter → loutre de mer
peppermint → menthe poivrée
cheese →
The model continuesfromage
ResultThe model gives fromage. Nobody fine-tuned it for French translation. It finds the pattern in the prompt and continues it. This is only next-token prediction with a good context.

13.3 Instruction fine-tuning

To make these models more useful, they get instruction fine-tuning (slide 54). They train on pairs of instructions and responses, for example "Summarize this text: [text]" → [summary]. This teaches the model to follow human instructions and to do tasks on request. It changes a raw language model into a system such as ChatGPT, which can have conversations and help with tasks.

PracticeIn Worked example 18, the model samples "a" at step 1 (probability 0.15). Can the final text still be "the University of Queensland"? What does this show about sampling?
AnswerNo. After the model adds "a", each later step uses UQ is a. Thus the text changes (for example "a university in Brisbane"). Each token stays fixed after the model generates it. One random selection at the start changes all later probabilities. Thus sampled answers are different in different runs.

14. LLMs in information retrieval

Sources: slides 55–56 · Video 7 (Autoregressive decoders) · Lecture transcript (ranking challenge)

Large language models change IR in three main directions (slide 55):

Prompting LLMs for retrieval and ranking LLM as rewriter, retriever, reranker or reader e.g. RankGPT, PromptReps Retrieval-augmented generation (RAG) retrieve documents, LLM answers with them (IR for LLMs) e.g. Self-RAG, BlendFilter Model-based IR (generative retrieval) the LLM is both the indexer and the retriever e.g. Differentiable Search Index (DSI)
Three directions from slide 55. LLMs work in the search pipeline, retrieval gives information to LLMs, and LLMs replace the index.
DirectionIdeaExamples
Prompting LLMs for retrieval and rankingUse the LLM directly as a rewriter, retriever, reranker or reader in the search pipeline (LLM-enhanced IR)RankGPT, PromptReps
Retrieval-augmented generation (RAG)Retrieve relevant documents. Then give them to the LLM as context, so that it generates better and more reliable responses (IR for LLMs).Self-RAG, BlendFilter
Model-based IRThe LLM is the indexer and also the retriever: generative (document) retrievalDSI (Differentiable Search Index), GR

The in-class challenge gave more methods for LLM-based ranking: pointwise, pairwise and listwise prompting, query and document expansion, and embeddings from decoders. Section 15 explains all of them.

14.1 The evolutionary trend

2017 2018 2020 2022 now Transformer BERT GPT-3 ChatGPT GPT-4, Claude, … "Attention isall you need" explosion ofencoder models rise ofdecoder models AI revolutionsparked new modelsconstantly
From the 2017 transformer paper, through encoder models (BERT, 2018), to decoder LLMs (GPT-3 in 2020, ChatGPT in 2022), and the fast changes of today (slide 56, Yang et al., 2023).
PracticeA company chatbot answers questions about internal policy documents. It must cite the documents. Which of the three directions is this? What does the IR part give?
AnswerThis is retrieval-augmented generation. A retriever (BM25, a dense retriever, or the two) finds the relevant policy passages. Then the LLM generates the answer with these passages as context. Retrieval gives current evidence that you can check. The LLM did not see this evidence in training. Thus the answers are more reliable and you can cite them.

15. In-class exercises, worked

Sources: Lecture transcript (in-class Exercises 1–3 and the challenge questions) · slides 20–23, 29, 49, 55

The lecture ended with three exercises and a challenge. Try each exercise before you open the answer. The lecturer said: "thinking about how you would use these components is exactly how the techniques later in this course were invented."

15.1 Exercise 1: transformer components

PracticeWhich statement about the transformer architecture is correct? A. The encoder uses masked attention to prevent looking at future tokens. B. The decoder uses bidirectional attention to see all tokens in the sequence. C. The encoder processes input tokens in parallel using bidirectional attention. D. Both encoder and decoder use the same type of attention mechanism.
Two of the incorrect options change the encoder and the decoder.
AnswerC.
OptionResultWhy
AincorrectIt is the opposite. The encoder looks at future tokens (bidirectional). The decoder uses the mask. Each encoder token must see all tokens near it. For example, the token loans shows the meaning of an earlier bank.
BincorrectThis is also the opposite. The encoder uses bidirectional attention. The decoder only generates. During generation, future tokens do not exist yet.
CcorrectThe encoder sees the full sequence at the same time (bidirectional). It processes all tokens in parallel. Attention is batched matrix multiplication, and GPUs make this fast. (CPUs calculate in sequence. GPUs process all values at the same time. Thus transformers made GPUs successful, and GPUs made transformers successful.)
DincorrectThey use different attention. The encoder is bidirectional (all positions, earlier and later). The decoder is masked or causal (earlier positions only).

15.2 Exercise 2: BERT vs GPT

PracticeWhat is the main architectural difference between BERT and GPT? A. BERT uses only the decoder part, while GPT uses only the encoder part. B. BERT uses only the encoder part, while GPT uses only the decoder part. C. BERT has more layers than GPT. D. BERT uses different tokenization techniques than GPT.
AnswerB. BERT is encoder-only. GPT is decoder-only. A is the opposite.

Why C is incorrect (and interesting): in reality, BERT has fewer layers than modern GPT models. But this is a choice of size and not a part of the architecture. You can build an encoder with more layers than a GPT. Generation models are usually much bigger, because generation is a more difficult task than representation. The 110M parameters of BERT-base were "sort of already enough" for classification and ranking. But small generative models (GPT-2 era, T5 era) did not work well. Good generation came only with very large parameter counts (the scaling law). Thus the name is large language models.

Why D is incorrect (and interesting): the two models use the same technique: trained subword tokenisation (WordPiece-style, Section 4). The slide asks a second question: "do BERT and GPT-2 have a different number of overall tokens?" The answer is yes. They trained on different collections with different target vocabulary sizes and goals. BERT has approximately 30K pieces. Some newer models have hundreds of thousands. Thus each model tokenises the same text differently. The technique is the same, but the vocabulary is different.

15.3 Exercise 3: constructing input embeddings

PracticeThe sentence "The cat sits" is tokenised as [CLS] the cat sits [SEP], with embedding dimension 4. Token embeddings: [CLS] = [1,0,0,1], the = [0,1,1,0], cat = [1,1,0,0], sits = [0,0,1,1], [SEP] = [1,0,1,0]. Segment embeddings: A = [0,1,0,1], B = [0,0,0,2] (all special tokens belong to segment A). Position embeddings: pos₀ = [1,0,1,0], pos₁ = [0,1,0,1], pos₂ = [1,1,0,0], pos₃ = [0,0,1,1], pos₄ = [1,0,0,1]. Construct the complete input embedding for every token.
First, find the segment of each token. Then use the formula of Section 5.4.
AnswerStep 1: find the parts. There is only one sentence. There is no second segment after a [SEP] in the middle. Thus each token, also [CLS] and [SEP], is in segment A and gets [0,1,0,1]. The model does not use the embedding of segment B. The positions are 0–4 from left to right.

Step 2: add the three vectors for each token (element-by-element addition, Section 2.1; one vector for each token):

TokenToken E+ Segment E (A)+ Position E= Final input embedding
[CLS] (pos 0)[1,0,0,1][0,1,0,1][1,0,1,0][2,1,1,2]
the (pos 1)[0,1,1,0][0,1,0,1][0,1,0,1][0,3,1,2]
cat (pos 2)[1,1,0,0][0,1,0,1][1,1,0,0][2,3,0,1]
sits (pos 3)[0,0,1,1][0,1,0,1][0,0,1,1][0,1,2,3]
[SEP] (pos 4)[1,0,1,0][0,1,0,1][1,0,0,1][2,1,1,2]

Sample calculation for [CLS]: [1+0+1,  0+1+0,  0+0+1,  1+1+0]=[2,1,1,2] [1{+}0{+}1,\; 0{+}1{+}0,\; 0{+}0{+}1,\; 1{+}1{+}0] = [2,1,1,2] .

Common traps (the two traps occurred in class). (1) Students put the tokens after [SEP] into segment B. This is incorrect here, because [SEP] only ends the single sentence. (2) Students put the three embeddings into a 3×4 matrix. This is incorrect, because the model adds them into one vector with 4 dimensions for each token. The five final vectors, as rows, make the 5×4 input matrix of the encoder. Real models use 768 or more dimensions. The practical of this week calculates them in the same way. The explainer How [CLS] becomes [2, 1, 1, 2] shows the [CLS] row step by step.

15.4 Challenge: encoders and decoders for ranking, encoders for generation

PracticeQ1: How can we use encoders or decoders for ranking? Name at least two ways.
Think about each place where a vector or a probability occurs in Sections 7 and 8.
AnswerThe full list from the class.

With encoders

MethodHow it worksRelated week
Single-vector dense retrieval (bi-encoder)Encode the query into one vector ([CLS] or the mean of the token embeddings). Encode each document into one vector. The score is the cosine similarity or the dot product. The model encodes the documents offline at indexing time. Thus search is fast.Week 5
Multi-vector, late interaction (ColBERT-style)Keep one vector for each token of the query and of the document. Calculate the similarity of each query token with each document token (a full similarity matrix). Combine the similarities into a final score. This is more precise, but it needs more storage.Later weeks
Cross-encoder rerankingJoin query [SEP] document, put it through BERT, and read a relevance score from the [CLS] head. This is the most accurate method, but it needs one model run for each (q, d) pair.Week 8

With decoders (LLMs)

MethodHow it works
Prompted ranking (RankGPT-style)Ask the LLM. Pointwise: "here is a query and a document; give me a relevance score." Pairwise: "document A or document B: which is more relevant?" Listwise: "here are 10 documents; give me an order." (The lecturer: "you may be using this every single day; why can't people think of the easy way?")
Query expansion (query2doc-style)The LLM generates a pseudo-document or an answer for the query (for example, ask Gemini and use its answer). Then a lexical ranker uses this longer text. The generated text contains car, cars, automobile, Tesla, BMW …. Thus it solves the vocabulary mismatch of Section 1.
Document expansion (doc2query-style)This is the opposite method, and it is better for latency. At indexing time, add to each document the queries or terms that it can answer. There is no cost at query time, because the system does this work when it builds the index (as in the BM25 indexing of the Week 3 practical).
Decoder hidden states as embeddings"Every time you see a vector, you can do computations." Stop the model before the output linear layer. Use the last hidden vector as a dense representation for similarity search. Thus decoders can also make embeddings.
Output probabilities as representations (PromptReps-style)Use the next-token probability distribution over the full vocabulary as a representation of the document. This representation is sparse, similar to term statistics. A document about this course gives high probability to information, retrieval, web, search, ranking, University of Queensland …. Rank the documents by a comparison of these distributions.
PracticeQ2 (challenge): Can we use an encoder backbone for generation?
What did the encoder learn to predict?
AnswerYes. This is one of the newest ideas in the field: diffusion language models. The steps of the argument from the lecture:
  1. What did the encoder learn? Masked language modelling: it predicts [MASK] tokens. This is also a type of token prediction.
  2. Thus take the prompt and add a batch of [MASK] tokens (for example, 256) at the end. Run the encoder one time. Read the predictions for all masked positions at the same time. An autoregressive decoder predicts one token at a time.
  3. The problem: the model predicts each masked token independently. Thus the output can be incoherent.
  4. The solution comes from image diffusion models: iterative refinement. An image model removes noise step by step. In the same way, run the prediction a few times (3–5 passes). Each pass makes the text more coherent.
  5. The advantage: 256 tokens need approximately 5 passes and not 256 sequential passes. GPUs process parallel tokens easily. Thus generation becomes much faster.
The lecturer said that his recent paper uses diffusion language models for retrieval. The main lesson: these techniques combine the building blocks of this week in new ways, and you can understand them. "Keep your imagination." For the backbone diagram, see The encoder backbone, and why it cannot generate.
PracticeA student asked after class: which is cheaper for users, document expansion or query expansion with an LLM? Why?
AnswerDocument expansion. It runs the expensive LLM one time for each document at indexing time, before users send queries. Thus the query latency stays the same. Query expansion runs the LLM while the user waits. The two methods add terms to the lexical representation. Thus exact-match rankers such as BM25 can solve vocabulary mismatch.

16. Summary and study checklist

16.1 BERT vs GPT at a glance

BERT (encoder-only)GPT-3 and LLMs (decoder-only)
ReleasedOctober 2018 (Google)June 2020 (OpenAI); ChatGPT 2022
ArchitectureTransformer encoder, 12 layers (base)Transformer decoder
Parameters110M (base) / 345M (large)175B (approximately 1,000× BERT-base)
AttentionBidirectional: sees the full sentenceUnidirectional (masked): sees only the past
Pre-trainingMasked language modelling (approximately 15% masked) + next sentence predictionNext-token prediction
OutputOne contextual vector for each token ([CLS] or mean pooling for the sentence)Probability distribution over the vocabulary (linear layer + softmax on the last hidden vector)
StrengthUnderstanding and representationGeneration and prediction; in-context learning
Best forSearch, classification, rankingText generation, translation, conversation
Main limitsMaximum input of 512 tokens; no autoregressive generationNo access to future context; expensive at scale
IR useDense retrieval (Week 5), reranking (Week 8)RankGPT-style prompting, query and document expansion, RAG, model-based IR

16.2 Key takeaways

  1. Term-based models (BM25) fail on vocabulary mismatch and polysemy. They match strings and do not capture meaning. Modern retrieval needs semantic matching and context awareness.
  2. Section 2 has all the maths of a transformer: vector addition (add embeddings), dot products (similarity scores), vector × matrix multiplication (learned projections; (1×n)(n×m)=(1×m)(1{\times}n)(n{\times}m) = (1{\times}m), row · column for each cell) and softmax (scores → probabilities).
  3. The transformer (2017) gets semantic understanding from self-attention. Project the vector of each token into q (what I look for), k (what I offer) and v (my content). Calculate the score qi⋅kjq_i \cdot k_j, divide by dk\sqrt{d_k}, apply softmax, then calculate zi=∑jaijvjz_i = \sum_j a_{ij} v_j. The attention matrix has a direction (it is not symmetric).
  4. Real transformers are "fat" (many attention heads in each layer) and "tall" (stacked layers; this is possible because the input and output shapes are equal). MLP and layer-norm steps follow attention.
  5. The pipeline: WordPiece tokenisation (approximately 30K learned subword pieces; it solves vocabulary size, OOV and morphology) → input embeddings (token + position + segment, added element by element) → encoder and/or decoder. Special tokens ([CLS], [SEP], [MASK], [PAD]; [BOS], [EOS]) are instructions.
  6. Encoders (BERT) understand. They use bidirectional attention, parallel processing, and one contextual vector for each token. They pre-train with MLM + NSP and fine-tune for tasks. IR uses them as dense retrievers and cross-encoder rerankers.
  7. Decoders (GPT) generate. They use masked attention. The last hidden vector × vocabulary projection → softmax → next-token probabilities, in a loop until [EOS]. Sampling is the reason why answers change. They train with next-token prediction and show in-context learning. IR uses them for prompted ranking, query and document expansion, RAG and model-based IR.
  8. The two types train self-supervised on web-scale text. The text gives its own labels (masked words, next words). Scale is important (scaling laws), and efficiency is now also important.
  9. Next: Week 5, BERT-based dense retrieval; Week 8, BERT-based rerankers. From now on, almost every model in the course is a transformer.

16.3 Study checklist

17. Video index and resources

VideoTopicLengthUsed in sections
Video 1Overview of the transformer (BM25 limits, attention, the four steps)13:351, 3, 7, 8
Video 2Encoder vs decoder3:159
Video 3Input embeddings5:255
Video 4Tokenisation (WordPiece)2:584
Video 5Special tokens4:006
Video 6BERT7:1810, 11, 12
Video 7Autoregressive decoders4:458, 10, 13, 14

18. References