INFS7410 · Information Retrieval and Web Search · Week 4

The Transformer Architecture

Term-based models such as BM25 match strings and miss meaning. This week builds the transformer from the ground up so that later weeks can use it for retrieval: the vector and matrix maths it needs, WordPiece tokenisation, token, position and segment embeddings, special tokens, self-attention with queries, keys and values (worked by hand), the decoder and its next-token output head, the split into encoder-only and decoder-only models, self-supervised training, BERT (masked language modelling, next sentence prediction, fine-tuning, dense retrieval and cross-encoder reranking) and autoregressive decoders such as GPT, and the main ways LLMs enter information retrieval. Dr Shuai Wang, UQ ielab.

1. Why BM25 is not enough

Sources: slides 2–6 · Video 1 (Overview of the transformer) · Lecture transcript (ranking history, term mismatch)

Weeks 1–3 built the classical retrieval stack: search engine architecture and text processing (Week 1), evaluation (Week 2), and term-based lexical models such as TF, TF-IDF, VSM, BIM and BM25 (Week 3). BM25 has been the gold standard for decades, and it has one fundamental blind spot: it can only do exact word matching. It has no model of what words or sentences mean.

Key ideaTerm-based lexical models score a document by counting the terms it shares with the query: they model relevance by term statistics and term overlap. The lecturer's framing: term statistics are what cause the limitation. When query and document use the same terms, the document ranks high; when they use different terms, even terms with identical meaning, the document ranks low or is never retrieved. The model has no semantic understanding of the query or the documents.

1.1 Two classic failure modes

1 · Vocabulary mismatch (false negative) Query car Document automobile no shared term BM25 ≈ 0 same meaning, zero overlap → relevant document missed 2 · Polysemy (false positive) Query closest bank? Document river bank shares: bank BM25 > 0 shared string, different meaning → irrelevant document ranked high
Term matching fails in both directions: synonyms with no shared string are missed, and a shared string with a different sense is wrongly rewarded.
Failure modeExampleWhat BM25 does
Vocabulary mismatch (false negative)Query car; document says automobile. The lecturer's second example: a query with biomedical against a document about medicine or biology, clearly related concepts written as different strings.Zero overlapping terms, so a score of (near) zero. The relevant document is missed.
Polysemy (false positive)Query where is the closest bank? (a financial institution); document about a river bank (a geographic feature). The in-class version: "I want to fix my laptop" against "I want to fix this guy": the same word means completely different things in different sentences.bank appears in both, so the document may rank highly. An irrelevant document is retrieved.

Worked example 1A relevant document BM25 cannot retrieve

Query: how to fix laptop. Document title: "Computer repair guide for damaged notebooks" (slide 5). A human sees at once that they match. Check the term overlap after stop-word removal:

Query termMatching document term?Overlap
fixrepair: a synonym written as a different string✗
laptopcomputer, notebooks: related concepts, different strings✗

The BM25 score from Week 3 is a sum over the query terms that also occur in the document. Here that set is empty:

BM25(Q,D)=∑t ∈ Q∩DIDF(t)⋅(k1+1) ft,Dk1((1−b)+b∣D∣avdl)+ft,D  =  0because Q∩D=∅ \text{BM25}(Q,D)=\sum_{t\,\in\,Q\cap D}\text{IDF}(t)\cdot\frac{(k_1+1)\,f_{t,D}}{k_1\big((1-b)+b\frac{|D|}{avdl}\big)+f_{t,D}} \;=\; 0 \quad\text{because } Q\cap D=\varnothing
ResultThe document scores 0 and BM25 cannot retrieve it: a pure semantic mismatch failure. The slide's verdict: "No overlapping terms; but the document should be relevant. Not retrievable by BM25."

1.2 What a modern retrieval system needs

RequirementMeaningExample
Semantic matchingUnderstand meaning as well as surface stringscar should match automobile; laptop and computer are related concepts
Context awarenessThe same word can mean different things in different contexts; the system must understand query intent beyond surface wordsApple in a technology context is a company; apple in a recipe is a fruit

1.3 Where we are in the history of ranking

The lecture opened with a "ranker from the history point of view" timeline. Week 4 sits at the "Industrial Revolution" (2017–2018): the arrival of the transformer. Everything after this point in the course builds on it.

EraModelsWhere in this course
"Jurassic" (1950s–1990s)Term-based lexical models (Boolean, TF-IDF, BM25)Week 3
"Human evolution" (2000s–2010s)Feature-based reranking models (LTR, online LTR; fusion, RF)Weeks 7, 11, 12
"Stone Age transitioning" (2010s)Word2vec, RNN, LSTMOutside the course
"Industrial Revolution!" (2017–2018): we are hereTransformer. Encoder: BERT · Decoder: GPTWeek 4
"New inventions" (2018–2022)Encoder-based rerankersWeek 8
(same period)Dense retrievers; learnt sparse modelsWeeks 5, 6
"AI age" (2022 onwards)ChatGPT, Mistral, Claude, Deepseek, QwenParts of Weeks 5, 9, 10
"The future is accelerating"LLM rerankers and retrievers, LLM-assisted ranking, RAG, search agentsParts of Weeks 5, 9, 10
Watch outThe field moves very fast, so the course focuses on core ideas and the most impactful techniques. The lecturer called this one of the most important weeks of the course: after it, essentially every model you meet is built on the transformer.
PracticeA user searches jaguar top speed and BM25 returns a page about the Jaguar car brand's dealerships, while a wildlife page saying "the big cat can sprint at 80 km/h" is missed. Name both failure modes involved.
One problem rewards a shared string; the other misses a relevant page with different strings.
AnswerThe dealership page is a polysemy false positive: jaguar matches as a string with the wrong sense. The wildlife page is a vocabulary mismatch false negative: big cat and sprint at 80 km/h express jaguar and top speed with no shared terms. Fixing both needs semantic matching and context awareness.

2. Maths refresher: vectors, matrices and softmax

Sources: Lecture transcript (linear-layer Q&A, Exercise 3 vector addition, self-attention walkthrough) · background for slides 10–19 and 25–29

Everything a transformer does is built from a handful of simple operations on vectors and matrices. Neural models read numbers only; as the lecturer put it, "we need to represent things in numbers, we need to do number calculations. That's how those neural models are formed." This section rebuilds those operations from scratch so the rest of the notes can use them freely. The companion workbook Transformer in a Spreadsheet (sheet "1 Maths") recomputes every example here with live formulas.

2.1 Vectors, vector addition and the dot product

A vector is an ordered list of numbers, for example x=[2, 1, 0, 3] x = [2,\, 1,\, 0,\, 3] . The number of entries is its dimension (here 4). In transformers every token is represented as a vector: BERT uses 768-dimensional vectors, Llama 2,048-dimensional ones, and so on.

(a+b)i  =  ai+bi,i=1,…,d (a+b)_i \;=\; a_i + b_i, \qquad i = 1,\dots,d
Where
a,ba, b
two vectors of the same dimension
dd
their dimension (number of entries)
ii
a position index, from 1 to dd
(a+b)i(a+b)_i
entry ii of the sum vector
Read it aloudEntry i of a plus b equals entry i of a plus entry i of b, for every position i from 1 to d.
Formula in plain wordsVector addition works position by position: first entry plus first entry, second plus second, and so on. Both vectors must have the same dimension, and the answer is a vector of that same dimension. Adding three vectors therefore still gives one vector (a point the lecturer stressed in Exercise 3: "you add them together, you do not make them a matrix"). This is how token, segment and position embeddings combine in Section 5.
a⋅b  =  ∑i=1dai bi a \cdot b \;=\; \sum_{i=1}^{d} a_i\, b_i
Where
a⋅ba \cdot b
the dot product of aa and bb: a single number
ai, bia_i,\ b_i
entry ii of each vector
dd
the shared dimension
∑i=1d\sum_{i=1}^{d}
add the products over all dd positions
Read it alouda dot b equals the sum, over every position i from 1 to d, of a sub i times b sub i.
Formula in plain wordsMultiply the two vectors entry by entry and add the products. The result is one number that measures how much the vectors agree: it is large and positive when big entries line up with big entries (similar directions), near zero when the vectors are unrelated, and negative when they point in opposite directions. Making either vector longer scales the result up, which is why cosine similarity (Section 12, and the VSM from Week 3) divides by the lengths. The dot product is the core similarity operation of this course: attention scores (Section 7) and dense retrieval scores (Week 5) are both dot products.

Worked example 2Add three vectors, then take a dot product

1
Add position by position: [1,0,0,1]+[0,1,0,1]+[1,0,1,0]=[1+0+1,  0+1+0,  0+0+1,  1+1+0] [1,0,0,1] + [0,1,0,1] + [1,0,1,0] = [1{+}0{+}1,\; 0{+}1{+}0,\; 0{+}0{+}1,\; 1{+}1{+}0] .
[1,0,0,1]+[0,1,0,1]+[1,0,1,0]=[2, 1, 1, 2] [1,0,0,1] + [0,1,0,1] + [1,0,1,0] = [2,\,1,\,1,\,2]
2
Dot product of [2,2][2,2] and [1,3][1,3]: multiply entries and add, 2×1+2×3=2+6 2\times 1 + 2\times 3 = 2 + 6 .
ResultThe sum is the 4-dimensional vector [2,1,1,2][2,1,1,2] (these are exactly the three embeddings of [CLS] in Exercise 3, Section 15). The dot product [2,2]⋅[1,3]=8[2,2]\cdot[1,3] = 8 is a single number.

2.2 Matrices and the transpose

A matrix is a rectangular grid of numbers with nn rows and mm columns, written n×mn \times m ("n by m", always rows first). A vector is a matrix with one row (a row vector, 1×m1\times m) or one column (a column vector, n×1n\times 1). For example A=[123456]A=\begin{bmatrix}1 & 2 & 3\\ 4 & 5 & 6\end{bmatrix} is 2×32\times 3 and B=[789101112]B=\begin{bmatrix}7 & 8\\ 9 & 10\\ 11 & 12\end{bmatrix} is 3×23\times 2.

(A⊤)ij  =  Aji \big(A^{\top}\big)_{ij} \;=\; A_{ji}
Where
AA
an n×mn \times m matrix
A⊤A^{\top}
its transpose, an m×nm \times n matrix
AjiA_{ji}
the entry of AA in row jj, column ii
i,ji, j
row and column indices of the transposed matrix
Read it aloudThe entry in row i, column j of A transpose equals the entry in row j, column i of A.
Formula in plain wordsTransposing flips a matrix over its diagonal: rows become columns. So the 2×32\times 3 matrix AA above becomes the 3×23\times 2 matrix A⊤=[142536]A^{\top}=\begin{bmatrix}1 & 4\\ 2 & 5\\ 3 & 6\end{bmatrix}. No numbers change, only their positions. You will meet it in the attention formula QK⊤QK^{\top} (Section 7), where it is there purely so the shapes line up for multiplication.

2.3 Matrix multiplication, step by step

C=AB,Cij  =  ∑r=1kAir Brj,(n×k) (k×m)=(n×m) C = AB, \qquad C_{ij} \;=\; \sum_{r=1}^{k} A_{ir}\, B_{rj}, \qquad (n\times k)\,(k\times m) = (n\times m)
Where
AA
the left matrix, n×kn \times k
BB
the right matrix, k×mk \times m
CC
the product, n×mn \times m
CijC_{ij}
the entry of CC in row ii, column jj
kk
the inner dimension: columns of AA, which must equal rows of BB
rr
the running index over the inner dimension
Read it aloudC equals A times B, where the entry of C in row i and column j is the sum, over r from 1 to k, of A's entry in row i, column r times B's entry in row r, column j. An n by k matrix times a k by m matrix gives an n by m matrix.
Formula in plain wordsEach cell of the answer is one dot product: row ii of AA with column jj of BB. Matrix multiplication is many dot products laid out in a grid. The shape rule says the inner dimensions (kk) must match; they are "consumed" by the sum, and the outer dimensions survive as the shape of the answer. If the inner dimensions differ, the product is undefined. Order matters: ABAB and BABA generally differ, and may even have different shapes.
A (2×3) 1 2 3 4 5 6 row 1 highlighted × B (3×2) 7 9 11 8 10 12 column 1 highlighted = C (2×2) 58 64 139 154 C[row 1, col 1] = (row 1 of A) · (column 1 of B) = 1×7 + 2×9 + 3×11 = 7 + 18 + 33 = 58 inner dimensions match (3 = 3) and are consumed; the result is 2 × 2
Each cell of the product is one dot product: a row of the left matrix with a column of the right matrix.

Worked example 3Every cell of A × B

Multiply the 2×32\times 3 matrix AA by the 3×23\times 2 matrix BB from Section 2.2. The inner dimensions match (3 = 3), so the answer is 2×22\times 2.

CellRow of A · column of BValue
row 1, col 11×7 + 2×9 + 3×11 = 7 + 18 + 3358
row 1, col 21×8 + 2×10 + 3×12 = 8 + 20 + 3664
row 2, col 14×7 + 5×9 + 6×11 = 28 + 45 + 66139
row 2, col 24×8 + 5×10 + 6×12 = 32 + 50 + 72154
[123456][789101112]=[5864139154] \begin{bmatrix}1 & 2 & 3\\ 4 & 5 & 6\end{bmatrix}\begin{bmatrix}7 & 8\\ 9 & 10\\ 11 & 12\end{bmatrix}=\begin{bmatrix}58 & 64\\ 139 & 154\end{bmatrix}
ResultAB=[5864139154]AB = \begin{bmatrix}58 & 64\\ 139 & 154\end{bmatrix}. Watch out: matrix multiplication is order-sensitive. Here BABA would be 3×33\times 3, a different shape entirely.

The special case transformers use constantly: vector × matrix

A row vector (1×n)(1 \times n) times a matrix (n×m)(n \times m) gives a row vector (1×m)(1 \times m). This is exactly what a neural network linear layer (also called a projection) is: it converts a vector of one size into a vector of another size using a matrix of learned weights. The lecturer derived this on paper during the Q&A: an nn-dimensional token vector times a learned n×mn \times m projection matrix gives an mm-dimensional output vector. Any time a transformer "transforms" one vector into another, it does this.

Worked example 4A tiny linear layer

Project the 2-dimensional vector [2,1][2, 1] with the 2×32 \times 3 weight matrix [102311]\begin{bmatrix}1 & 0 & 2\\ 3 & 1 & 1\end{bmatrix}. Each output entry is the vector dotted with one column.

[ 2,  1 ]⏟1×2  [102311]⏟2×3=[ 2×1+1×3,    2×0+1×1,    2×2+1×1 ]=[ 5,  1,  5 ]⏟1×3 \begin{aligned} \underbrace{[\,2,\;1\,]}_{1\times 2}\;\underbrace{\begin{bmatrix}1 & 0 & 2\\ 3 & 1 & 1\end{bmatrix}}_{2\times 3} &= [\,2{\times}1{+}1{\times}3,\;\; 2{\times}0{+}1{\times}1,\;\; 2{\times}2{+}1{\times}1\,] \\ &= \underbrace{[\,5,\;1,\;5\,]}_{1\times 3} \end{aligned}
ResultA 2-dimensional vector went in and a 3-dimensional vector [5,1,5][5, 1, 5] came out. Section 7 uses this operation to create query, key and value vectors, and Section 8 uses it to project a hidden vector (dimension 768) onto the whole vocabulary (dimension about 30,000) to choose the next word.

2.4 Softmax: turning scores into probabilities

Transformers repeatedly need to turn a vector of raw scores (any real numbers, called logits) into probabilities: numbers between 0 and 1 that sum to 1. The softmax function does this.

softmax(z)i  =  ezi∑j=1nezj \text{softmax}(z)_i \;=\; \frac{e^{z_i}}{\sum_{j=1}^{n} e^{z_j}}
Where
zz
a vector of nn raw scores (logits), z=[z1,…,zn]z = [z_1, \dots, z_n]
ziz_i
the score of item ii
ezie^{z_i}
the exponential of that score (e≈2.718e \approx 2.718); always positive
∑j=1nezj\sum_{j=1}^{n} e^{z_j}
the sum of the exponentials of all nn scores
softmax(z)i\text{softmax}(z)_i
the probability given to item ii
Read it aloudThe softmax probability of item i equals e to the power z sub i, divided by the sum over all items j of e to the power z sub j.
Formula in plain wordsExponentiate every score, then divide each by the total. Exponentiating makes every value positive (so negative logits are fine; they just become small) and exaggerates gaps: a score 1 higher gets about 2.7 times the weight. Dividing by the total forces the outputs to sum to 1. The largest score always gets the largest share, and adding the same constant to every score leaves the output unchanged. Softmax appears twice this week: inside attention (similarity scores to attention weights, Section 7) and at the decoder output (vocabulary logits to next-token probabilities, Section 8).
scores z (any real numbers) 2.0 1.0 0.0 e² ≈ 7.39 e¹ ≈ 2.72 e⁰ = 1.00 divide each by the sum 11.11 0.67 0.24 0.09 item 1 item 2 item 3 probabilities in (0, 1) that sum to 1
Softmax exponentiates the scores and normalises them: a gap of 1 in score becomes a ratio of about 2.7 in probability.

Worked example 5Softmax of three scores

Scores z=[2.0,  1.0,  0.0] z = [2.0,\; 1.0,\; 0.0] .

1
Exponentiate: e2.0≈7.39e1.0≈2.72e0.0=1.00 \begin{aligned} e^{2.0} &\approx 7.39 \\ e^{1.0} &\approx 2.72 \\ e^{0.0} &= 1.00 \end{aligned}
2
Sum: 7.39+2.72+1.00≈11.11 7.39 + 2.72 + 1.00 \approx 11.11 .
3
Divide each by the sum:
softmax(z)≈[ 7.3911.11,  2.7211.11,  1.0011.11 ]=[ 0.67,  0.24,  0.09 ] \begin{aligned} \text{softmax}(z) &\approx \left[\,\frac{7.39}{11.11},\; \frac{2.72}{11.11},\; \frac{1.00}{11.11}\,\right] \\ &= [\,0.67,\; 0.24,\; 0.09\,] \end{aligned}
Result[0.67,0.24,0.09][0.67, 0.24, 0.09]: every entry lies in (0,1)(0,1) and they sum to 1, a probability distribution. The biggest score gets the biggest share, and the gaps are exaggerated.
TakeawaysThe cheat sheet for the rest of these notes:
OperationShapesWhere it appears this week
Vector + vector (element-wise)same length in, same length outSumming token, segment and position embeddings (Section 5)
Vector · vector (dot product)two vectors in, one number outAttention scores (Section 7); cosine similarity (Section 12)
Vector × matrix(1×n)(n×m)=(1×m)(1\times n)(n\times m) = (1\times m)Q, K, V projections (Section 7); the output linear layer (Section 8)
Matrix × matrix(n×k)(k×m)=(n×m)(n\times k)(k\times m) = (n\times m)All attention scores at once, in parallel; this is why GPUs suit transformers
Softmaxscores in, probabilities out (same length)Attention weights (Section 7); next-token probabilities (Section 8)
PracticeA token vector has dimension 768 and a learned matrix has shape 768 × 64. What is the shape of the product, and what happens if you try to multiply the vector by a 64 × 768 matrix instead?
Apply the shape rule (1×n)(n×m)(1\times n)(n\times m).
Answer(1×768)(768×64)=(1×64)(1\times 768)(768\times 64) = (1\times 64): a 64-dimensional vector. With a 64×76864\times 768 matrix the inner dimensions are 768 and 64, which differ, so the product is undefined. (This 768 → 64 projection is the size of one BERT-base attention head's query, key and value vectors.)
PracticeCompute softmax of [1,1,1][1, 1, 1] and of [3,3,3][3, 3, 3]. What does this show?
AnswerBoth give [13, 13, 13]\left[\frac{1}{3},\ \frac{1}{3},\ \frac{1}{3}\right]. Softmax depends only on the differences between scores; adding the same constant to every score changes nothing.

3. The transformer: overview and the four steps

Sources: slides 7–19 · Video 1 (Overview of the transformer) · Lecture transcript (four steps, block contents)

In 2017 researchers at Google published "Attention Is All You Need" (Vaswani et al.), introducing the transformer architecture. The full architecture diagram looks intimidating. The lecturer's honest assessment: "it may look quite complicated, and in fact it was actually complicated … and this is already a simplified version". The key insight is simple to state.

Key ideaTransformers can "understand semantics" in language. More precisely (slide 8), they learn to "understand word distribution": how words relate to each other, why they appear together, and what meaning they convey in context. Words stop being mere strings of characters and become points in a space where meaning and relationships are captured.

The original transformer was designed for machine translation: feed an English sentence such as "The cat sat on the mat" into an encoder–decoder pipeline, and the model outputs the same sentence in another language (the slide uses Chinese, 猫坐在垫子上).

The cat sat on the mat input text (English) Encoder encoded vectors Decoder 猫坐在垫子上 Chinese the complete original transformer = encoder + decoder, trained for translation
The original transformer was built for translation; the same core technology gives retrieval its grasp of semantic relationships.
Watch outTranslation looks far from search, yet the technology that lets a model map "cat" to "猫" is the same technology that lets a retrieval model map "laptop" to "notebook computer".

3.2 The four processing steps

Slides 13–19 build the architecture up in four steps. This is the map for Sections 4 to 8.

STEP 1 tokenise text, build input embeddings STEP 2 · ENCODER self-attention builds contextual vectors STEP 3 · DECODER takes ⟨start⟩ + the encoded representation STEP 4 predict the next token from output probabilities generated token is fed back in; loop until ⟨end⟩ hand-off: encoded vectors
The (simplified) transformer pipeline: tokenise and embed, encode, hand off to the decoder, then generate one token at a time in a loop until an ⟨end⟩ token appears.
1
Tokenise and embed. The text is tokenised and converted into input embeddings: words become vectors the model can compute with (Sections 4–5).
2
Encode. The embeddings pass through the encoder, which applies self-attention to build a rich contextual representation of each word (Section 7). When the encoder finishes, its output vectors represent the complete understanding of the input: words, relationships and semantic meaning. This is the hand-off point: the encoder steps aside.
3
Decoder takes over. The decoder starts with two inputs: a special ⟨start⟩ token and the encoded representations. In the video's words, the decoder says "I'm going to generate; encoder, give me the context."
4
Generate, then loop. The decoder produces output token probabilities, picks the next token, feeds it back in with the encoded representation, and repeats (next token, next token, next token) until it produces an ⟨end⟩ token (Section 8). This step-by-step generation is how the decoder writes coherent text.

The translation of "The cat sat on the mat" therefore runs as a loop. At each iteration the decoder outputs a probability distribution over the vocabulary and picks the highest-probability token:

IterationDecoder inputPredicted token
1⟨start⟩ + encoded vectors猫
2⟨start⟩ 猫 + encoded vectors坐在
3⟨start⟩ 猫 坐在 + encoded vectors垫子
4⟨start⟩ 猫 坐在 垫子 + encoded vectors上
5⟨start⟩ 猫 坐在 垫子 上 + encoded vectors⟨end⟩: generation stops

3.3 What is inside each block

The hand-drawn architecture shown in the lecture reveals what each encoder or decoder block contains, bottom to top: multi-head self-attention → layer norm → multilayer perceptron (MLP) → layer norm, with arrows that skip around each part (residual connections). Section 7.4 draws it. Two facts to hold on to:

Watch outSlide 19 stresses that this is a simplified version. Real transformers use multi-head attention (several attention heads working in parallel) plus feed-forward layers, residual connections and layer normalisation, which these notes only sketch. The goal at this level is the fundamental flow and why it captures language so well.
PracticeIn the translation loop, which of the four steps run once per sentence and which run once per generated token?
Look at where the dashed loop arrow starts and ends.
AnswerSteps 1 and 2 (tokenise, embed and encode the English input) run once. Steps 3 and 4 run once per output token: the decoder reads ⟨start⟩ plus everything generated so far, together with the same encoded vectors, predicts one token, and loops until it emits ⟨end⟩. Five iterations produce 猫 坐在 垫子 上 ⟨end⟩.

4. Step 1a: tokenisation and WordPiece

Sources: slides 30–33 · Video 4 (Tokenisation) · Lecture transcript (WordPiece, token IDs)

Before any embedding happens, text must be broken into tokens: the units the model actually reads. Each token corresponds to an integer ID in a fixed vocabulary (the lecturer's example: the token my might be ID 3500), and these IDs are what get mapped to vectors. Week 1 tokenised text for an inverted index; here the tokeniser serves a neural model with a fixed-size lookup table, which changes the design goals.

The obvious approach splits on spaces: "my dog is cute" → [my, dog, is, cute]. Slide 31 lists three serious problems with simple word splitting:

  1. Huge vocabulary. English has about 500,000 unique words, which would need a massive embedding lookup table; once names, typos and new words are included, the vocabulary space is "almost infinite".
  2. Out-of-vocabulary (OOV) words. New words such as coronavirus (new in 2020), typos, or garbage strings like ecerfeferf: the model has never seen them, so it has no embedding for them.
  3. Lost morphological relationships. run, running and runner become completely unrelated tokens even though they are clearly related words.

The opposite extreme uses single characters as tokens (just a–z, 26 tokens). It can represent anything, but characters alone do not capture how people combine letters into meaningful units, and sequences become very long. The sweet spot lies in between.

Characters p · l · a · y · i · n · g vocabulary: 26 letters very long sequences few meaningful units WordPiece subwords play · ##ing vocabulary: about 30K pieces any word can be spelled morphology kept Whole words playing vocabulary: about 500K+ words OOV words break it morphology lost vocabulary size smaller larger
WordPiece sits between characters and whole words: a fixed, manageable vocabulary that can still spell any word.
Key ideaWordPiece is a trained tokeniser that breaks words into subword pieces, so the vocabulary has a fixed, manageable number of pieces (about 30,000) and any word, including new words and typos, can be written as a sequence of pieces. Continuation pieces are marked with ##.
TokeniserTokens for "[CLS] My dog is cute [SEP] he likes playing [SEP]"
Word splitting[CLS], my, dog, is, cute, [SEP], he, likes, playing, [SEP]
WordPiece[CLS], my, dog, is, cute, [SEP], he, likes, play, ##ing, [SEP]
Watch outWhy does playing split? Think about how the tokeniser was trained: ing appears extremely often in English. Keeping every -ing word as its own token would blow up the vocabulary, so the tokeniser keeps play and ##ing as separate reusable pieces.

4.1 How WordPiece training works

no: repeat start with characters a–z count adjacent pairs in web text merge the most frequent pair target size (~30K) reached? yes learned subword vocabulary merges grow pieces such as t + h → th, i + n + g → ing, t + i + o + n → tion
WordPiece training is a counting loop over a huge text collection: merge the most frequent adjacent pair, then count again, until the vocabulary reaches its target size.
1
Start with a character vocabulary: [a, b, c, …, z].
2
Analyse a huge amount of (web) text and find the most frequent adjacent character pairs.
3
Merge them into new subwords: th, ing, tion, …
4
Repeat until the target vocabulary size (about 30K pieces) is reached.
5
Result: a learned subword vocabulary, used by almost all modern transformer models (BERT, GPT, many LLMs). Any word can now be represented with these pieces.
Watch outThis is still term statistics: the same counting-over-a-collection thinking as Week 3, now used to learn a vocabulary. Different companies train tokenisers on different collections, with different target sizes and goals, so the same text tokenises differently in different models. BERT-style models use about 30K pieces, while some recent LLMs use vocabularies of hundreds of thousands of tokens. Same technique, different statistics; Exercise 2 (Section 15) tests exactly this.

Worked example 6Subword merges on a toy corpus

Illustrative training data with four word types and their counts: low ×5, lower ×2, newest ×6, widest ×3.

0
Start with characters: l o w, l o w e r, n e w e s t, w i d e s t.
1
Count adjacent pairs, weighted by word counts:
PairWhere it occursCount
e snewest (6) + widest (3)9 (most frequent)
s tnewest (6) + widest (3)9 (tied)
l olow (5) + lower (2)7
o wlow (5) + lower (2)7
Merge e s → es. The words become n e w es t and w i d es t.
2
Now es t occurs 9 times → merge into est.
3
l o (7) → lo; then lo w (7) → low; and so on, growing the vocabulary one useful piece at a time: es, est, lo, low, …
ResultAfter training, a never-seen word such as lowest tokenises cleanly into low + ##est: both pieces are in the vocabulary, the link to low and widest is preserved, and there is no OOV failure. This illustrates frequency-based merging. WordPiece itself scores a candidate merge by how much it improves the likelihood of the training data, score(a,b)=count(ab)count(a) count(b) \text{score}(a,b)=\frac{\text{count}(ab)}{\text{count}(a)\,\text{count}(b)} , which favours pairs that occur together more often than their parts alone would predict; the mechanics are the same.

Worked example 7Tokenising the problem words

Input wordWordPiece output (illustrative)Why it works
playingplay + ##ingMorphology preserved: shares play with played and player
coronaviruscorona + ##virusNew in 2020, yet both pieces were already in the vocabulary, so no OOV
ecerfeferfe + ##cer + ##fe + ##fer + ##fEven garbage and typos decompose into known pieces (worst case: single characters)
ResultAll three problem words from slide 31 receive valid token sequences, so each one gets embeddings. The exact splits depend on the trained vocabulary.
PracticeWhy does a WordPiece model never hit an out-of-vocabulary error, while a word-level model does?
Think about what the vocabulary starts with before any merges.
AnswerWordPiece training starts from the full character vocabulary and only adds merged pieces, so every single character stays in the vocabulary. Any string can therefore be spelled with pieces, in the worst case one character at a time. A word-level vocabulary is a fixed list of whole words, so any unseen word (a new term or a typo) has no entry and no embedding.

5. Step 1b: transformer input embeddings

Sources: slides 24–30 · Video 3 (Input embeddings) · Lecture transcript (three embeddings, "I asked him")

Once the text is tokenised, each token must become a vector. A token ID says what the token is and leaves out where it appears, yet word order changes meaning entirely. The lecturer's example: "I asked him" and "he asked me" use almost the same words in different positions, with completely different meanings. Transformers therefore combine three types of embeddings (slide 25):

EmbeddingPurposeRequired?
Token embeddingConverts tokens into vectors; represents word meaning through a learned lookup tableAlways
Position embeddingTeaches the model word order: each position (0, 1, 2, …) gets its own learned vectorAlways
Segment embeddingDistinguishes different parts of the input (for example query and document); each segment gets its own learned vectorOptional: only some models, such as BERT

5.1 Token embeddings

The text is tokenised, then each token is looked up in a learned embedding table: one learned vector per vocabulary entry, capturing what the token means on its own. The idea comes from earlier word-vector models such as word2vec. Slide 26's example (vectors truncated):

StageValue
Text"Information Retrieval"
Tokenise["information", "retrieval"]
Lookupinformation → [0.1, 0.3, −0.2, 0.9, …]
retrieval → [0.4, −0.1, 0.6, 0.2, …]

5.2 Position embeddings

Word order matters: "information retrieval" ≠ "retrieval information". Each position in the sequence gets its own learned embedding vector (slide 27): Pos1 → [0.9, 0.4, 0.2, 0.8, …], Pos2 → [0.2, −0.4, 0.4, 0.2, …].

5.3 Segment embeddings (optional)

Models such as BERT use segment embeddings when the input holds several sequences, for example a query and a document processed together (the BERT cross-encoder of Section 12). Special tokens separate the segments ([CLS] at the start, [SEP] between and at the end), and every token in a segment receives that segment's embedding (slide 28):

StageValue
TextInformation (query) + Retrieval (document)
Tokenised[CLS] information [SEP] retrieval [SEP]
Segmentsquery segment → [0.6, 0.3, 0.7, −0.9, …]
doc segment → [0.5, 0.1, 0.8, 0.2, …]
Watch outDecoder-only models (GPT, Llama) typically skip segment embeddings, and modern LLMs use smarter position schemes (for example rotary embeddings, "RoPE", for very long contexts). The underlying idea is the same: tell the model what each token is and where it sits.

5.4 Putting it together

Inputi  =  Etoken(xi)  +  Eposition(i)  +  Esegment(si) \text{Input}_i \;=\; E_{\text{token}}(x_i)\;+\;E_{\text{position}}(i)\;+\;E_{\text{segment}}(s_i)
Where
ii
the position of the token in the sequence (0, 1, 2, …)
xix_i
the token at position ii
sis_i
the segment that token belongs to (A or B)
Etoken(xi)E_{\text{token}}(x_i)
the learned vector for that token, from the token lookup table
Eposition(i)E_{\text{position}}(i)
the learned vector for position ii
Esegment(si)E_{\text{segment}}(s_i)
the learned vector for segment sis_i (omitted in models without segments)
Inputi\text{Input}_i
the final input vector for position ii; same dimension as each part (768 in BERT-base)
Read it aloudThe input vector at position i equals the token embedding of the token x sub i, plus the position embedding of position i, plus the segment embedding of its segment s sub i.
Formula in plain wordsEach token's input is the element-wise sum of three vectors of the same length (vector addition from Section 2.1): first entry plus first entry plus first entry, and so on. The token part says what the token is, the position part says where it is, and the segment part says which part of the input it belongs to. The result is one vector per token with the same dimension as each part, so a sentence of nn tokens becomes an n×dn \times d input matrix (one row per token). Change any one ingredient, such as moving the same word to a new position, and the input vector changes too.
[CLS] my dog is cute [SEP] he likes play ##ing [SEP] Token E[CLS] E my E dog E is E cute E[SEP] E he E likes E play E ##ing E[SEP] + Segment E_A (segment A) for every token E_B (segment B) + Position E pos 0 E pos 1 E pos 2 E pos 3 E pos 4 E pos 5 E pos 6 E pos 7 E pos 8 E pos 9 E pos 10 = sum ↓ final input embeddings fed to the transformer: one vector per token
Slide 30's running example, [CLS] my dog is cute [SEP] he likes play ##ing [SEP]. Tokens up to the first [SEP] get segment A; tokens after it get segment B. Positions count 0, 1, 2, … so the model knows where each token sits.

For a visual, column-by-column version of this addition, see the companion explainer How [CLS] becomes [2, 1, 1, 2], and sheet "2 Embeddings" of the Transformer in a Spreadsheet workbook.

Worked example 8One token's final input embedding

Take the token likes from the diagram: token likes, at position 7, in segment B. Illustrative 4-dimensional embeddings:

ComponentVector
Etoken(likes) E_{\text{token}}(\text{likes}) [0.2, −0.1, 0.5, 0.3]
Eposition(7) E_{\text{position}}(7) [0.0, 0.3, −0.2, 0.1]
Esegment(B) E_{\text{segment}}(B) [0.1, 0.1, 0.0, −0.2]

Sum element-wise:

Inputlikes=[0.2+0.0+0.1,    −0.1+0.3+0.1,0.5−0.2+0.0,    0.3+0.1−0.2]=[0.3,  0.3,  0.3,  0.2] \begin{aligned} \text{Input}_{\text{likes}} &= [0.2{+}0.0{+}0.1,\;\; -0.1{+}0.3{+}0.1, \\ &\qquad 0.5{-}0.2{+}0.0,\;\; 0.3{+}0.1{-}0.2] \\ &= [0.3,\; 0.3,\; 0.3,\; 0.2] \end{aligned}
ResultInputlikes=[0.3,0.3,0.3,0.2]\text{Input}_{\text{likes}} = [0.3, 0.3, 0.3, 0.2]. If the same word likes appeared at position 2 in segment A, its token embedding would be identical while its position and segment embeddings would differ, so the final input vector would differ. That is how the model tells where and in which part a word occurs. The exam-style version of this calculation (in-class Exercise 3) is in Section 15.
PracticeIn "the bank by the river bank", the token bank appears at positions 1 and 5 (one segment). Do the two occurrences get the same input vector?
AnswerThey share the token embedding Etoken(bank)E_{\text{token}}(\text{bank}) and the segment embedding, but their position embeddings differ (Eposition(1)E_{\text{position}}(1) against Eposition(5)E_{\text{position}}(5)), so the input vectors differ. The input embedding still carries little context, though: telling a financial bank from a river bank is the job of self-attention in the encoder (Section 7).

6. Special tokens

Sources: slides 34–39 · Video 5 (Special tokens) · Lecture transcript (CLS for retrieval, PAD for batching, "[FRANKENSTEIN]")

Key ideaSpecial tokens are reserved symbols that act like instructions for the transformer. They tell the model where sequences start and end, how to separate different parts of the input, and what type of task to perform. They are preserved in the model's tokeniser, so they are guaranteed to become exactly one token when detected in text.

6.1 Encoder special tokens: [CLS], [SEP], [MASK], [PAD]

TokenNameRole
[CLS]ClassificationAlways at the beginning of the input; its output vector is used for sentence-level tasks (classification, ranking). The lecturer's note: it is called the classification token, but if you fine-tune the model to project this token to any task, it may work. Retriever models use it heavily to represent a whole sentence as one vector.
[SEP]SeparatorDivides different sequences; helps the model distinguish the parts of the input (for example query and document)
[MASK]Masked tokenUsed during training: the model learns to predict the masked words (Sections 10 and 11)
[PAD]PaddingPads sequences to the same length so they can be batched: a batch must have one length for the GPU to process it in parallel. Ignored during attention computation.
padding task signal separator end of input [PAD] [CLS] laptop repair [SEP] Computer maintenance guide for damaged notebooks [SEP] segment A: the query segment B: the document [PAD] keeps batch lengths equal and is ignored by attention
Slide 37's search-task input: special tokens mark the task, the boundary between query and document, and the end of the input.

Worked example 9A search-task input, token by token

Input: [PAD] [CLS] laptop repair [SEP] Computer maintenance guide for damaged notebooks [SEP]

PieceWhat it does here
[PAD]Keeps the length uniform when this input is batched with longer inputs (ignored by attention)
[CLS]Signals a classification or ranking task; its final vector will carry the relevance decision
laptop repairSegment A: the query
[SEP]Separates the query from the document
Computer maintenance guide …Segment B: the document
[SEP]Marks the end of the input sequence
ResultThis query + document format is exactly how a BERT reranker (cross-encoder) is fed (Section 12): the model reads both texts together and scores their relevance from the [CLS] vector.

6.2 Decoder special tokens: [BOS], [EOS]

TokenNameRole
[BOS]Beginning of sequenceSignals the start of generation; the first token fed to the decoder during inference
[EOS]End of sequenceSignals the end of generation and tells the decoder when to stop; critical for controlling output length. How does ChatGPT know it has said enough? It generates its EOS token.

Worked example 10BOS and EOS in a generation task

PartTokens
Prompt<BOS> The cat sits on
Generatedthe mat <EOS> (the model stops here)
Result<BOS> signals the start of the prompt; when the model itself generates <EOS>, generation is finished. This simple mechanism lets decoder models produce output of an appropriate length and stop cleanly.
Watch outEncoder models (BERT) and decoder models (LLMs) use different special-token sets: [CLS]/[SEP]/[MASK]/[PAD] for encoders, [BOS]/[EOS] (and friends) for decoders. These are only conventions. As the lecturer put it, you can invent any special token, even [FRANKENSTEIN], and as long as you train the model with it, the model can learn what it means.
PracticeA batch holds a 6-token input and a 10-token input. What does [PAD] do here, and why must attention ignore it?
AnswerFour [PAD] tokens are added to the shorter input so both have length 10 and the GPU can process them together as one rectangular batch. The pads carry no content, so attention ignores them; otherwise real tokens would mix meaningless padding vectors into their representations.

7. Step 2: self-attention and the encoder

Sources: slides 10–15, 19 · Video 1 (Overview of the transformer) · Lecture transcript (self-attention walkthrough, symmetry question, "fat and tall", Exercise 1)

Key ideaIn a transformer, every word in a sentence looks at every other word and decides how much attention to pay to it (slide 10). Because words attend to words within the same sequence, this is the self-attention mechanism. It lets the transformer capture relationships, context and long-range dependencies, which earlier architectures struggled with. The lecturer called attention "the magic beyond the previous line of deep neural networks": it makes tokens interact with each other.

7.1 The intuition: the attention matrix

For the sentence "The cat sat on the mat", self-attention produces a matrix in which each cell shows how much one word (row) pays attention to another word (column). For example, cat pays high attention to sat because they are grammatically related (subject and verb).

The cat sat on the mat The cat sat on the mat word paying attention (row) word being attended to (column); darker = more attention
Illustrative self-attention pattern for "The cat sat on the mat" (shading only; a real row of weights sums to 1). Each row shows how one word spreads its attention over the sentence; cat attends strongly to sat.

Where do these numbers come from? The in-class lecture answered with a five-step breakdown ("Self-attention: what is it really doing?"). If you know little about neural networks, read the five steps slowly: every operation involved is one you met in Section 2. Sheet "3 Attention" of the Transformer in a Spreadsheet workbook runs all five steps with live formulas, and the Transformer by Hand worksheet lets you compute a block yourself.

7.2 The five steps of self-attention

STEP 1 one input vector per token xᵢ (Section 5) STEP 2 project xᵢ into three vectors qᵢ, kᵢ, vᵢ STEP 3 q seeks, k offers, v carries content STEP 4 score qᵢ·kⱼ/√dₖ, softmax each row → weights aᵢⱼ STEP 5 weighted sum of value vectors zᵢ = Σⱼ aᵢⱼ vⱼ one token → xᵢ → qᵢ, kᵢ, vᵢ → attention weights (row i) → one new output vector zᵢ
The five steps of self-attention from the lecture slide; the one-line summary underneath is worth memorising.

Step 1: one input vector per token

Each token ii has one input vector xix_i: the sum of its token, segment and position embeddings from Section 5. For the toy sentence the · cat · sat: x1=xthe, x2=xcat, x3=xsatx_1 = x_{\text{the}},\ x_2 = x_{\text{cat}},\ x_3 = x_{\text{sat}}.

Step 2: project each input vector into q, k and v

The model holds three learned weight matrices WQW^Q, WKW^K, WVW^V (learned during training, like everything else). Each token's input vector is multiplied by each matrix, the vector × matrix operation of Section 2.3:

qi=xiWQki=xiWKvi=xiWV q_i = x_i W^Q \qquad k_i = x_i W^K \qquad v_i = x_i W^V
Where
xix_i
the input vector of token ii (a 1×d1 \times d row vector)
WQ,WK,WVW^Q, W^K, W^V
learned projection matrices, each d×dkd \times d_k (WVW^V may use a different output size dvd_v)
qiq_i
the query vector of token ii: what it is looking for
kik_i
the key vector of token ii: what it offers to be matched against
viv_i
the value vector of token ii: the content it contributes
d, dkd,\ d_k
the input dimension and the query/key dimension
Read it aloudThe query of token i is its input vector times the query weight matrix; its key is the input vector times the key weight matrix; its value is the input vector times the value weight matrix.
Formula in plain wordsOne input vector becomes three new vectors, each a different learned "view" of the same token. Because the three matrices are different, the same token can advertise one thing (its key), look for another (its query) and pass on a third (its value). Stacking all tokens' qq vectors as rows gives the matrix QQ; likewise KK and VV (each n×dkn \times d_k for nn tokens). Convention: lowercase qi,ki,viq_i, k_i, v_i are per-token vectors; uppercase Q,K,VQ, K, V are the stacked matrices. Training adjusts the matrices; at run time they are fixed and applied to every token in the same way.
x_cat × WQ (learned) × WK (learned) × WV (learned) q_cat k_cat v_cat what I am looking for what I offer to others how I represent myself
Step 2 for one token: three learned projections turn one input vector into a query, a key and a value.

Step 3: what are Q, K and V? (the lecturer's intuitions)

VectorSlide definitionLecturer's phrasing
Q: querywhat this token is looking for"What am I to myself; what am I really looking for?"
K: keywhat this token offers, what can be matched"What am I to others?"
V: valuethe information or content carried by this token"In the end, how do I represent myself?"
Watch outThis is abstract, and the lecturer admitted as much ("that's also how the machine learning people are"). Why three roles? Because they are learned: giving the model separate freedom over "what I seek", "what I advertise" and "what I contribute" is simply what works. If you have heard of KV caching in LLM serving, these are exactly the K and V being cached.

Step 4: compute attention weights

To find how much token ii should attend to token jj, take the dot product of qiq_i with kjk_j: "how well does what I am looking for match what you offer?" Scale it, then apply softmax across each row so every token's weights sum to 1.

sij=qi⋅kjdk,aij=esij∑l=1nesil⟺A=softmax ⁣(QK⊤dk) s_{ij} = \frac{q_i \cdot k_j}{\sqrt{d_k}}, \qquad a_{ij} = \frac{e^{s_{ij}}}{\sum_{l=1}^{n} e^{s_{il}}} \qquad\Longleftrightarrow\qquad A = \text{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)
Where
qi⋅kjq_i \cdot k_j
dot product of token ii's query with token jj's key (the raw score)
dkd_k
the dimension of the query and key vectors
sijs_{ij}
the scaled score of token ii attending to token jj
nn
the number of tokens in the sequence
ll
index running over all tokens in the softmax denominator
aija_{ij}
the attention weight: the share of token ii's attention given to token jj
Q,KQ, K
all queries and all keys stacked as rows (n×dkn \times d_k)
AA
the n×nn \times n attention weight matrix; row ii holds ai1,…,aina_{i1}, \dots, a_{in}
Read it aloudThe score s i j is q i dot k j divided by the square root of d k. The weight a i j is e to the s i j, divided by the sum over every token l of e to the s i l. In matrix form, A is the row-wise softmax of Q times K transpose, divided by the square root of d k.
Formula in plain wordsThe dot product measures how well what token ii seeks matches what token jj offers; one matrix multiplication, QK⊤QK^{\top}, computes it for every pair at once, giving an n×nn\times n grid (the transpose is only there to line up the shapes). Dividing by dk\sqrt{d_k} is a normalisation: dot products grow with the vector dimension, and without scaling the softmax would become extremely peaked, giving almost all weight to one token. Softmax then turns each row into weights between 0 and 1 that sum to 1: a higher score takes a larger share, at the expense of the other tokens in that row. AA is exactly the shaded grid of Section 7.1. The values VV are unused so far: deciding "how much do I care about you" needs only queries and keys.

Step 5: build the new vector for each token

Each token's new, contextualised representation is the attention-weighted sum of all tokens' value vectors, including its own (attention to self is part of the sum).

zi  =  ∑j=1naij vj z_i \;=\; \sum_{j=1}^{n} a_{ij}\, v_j
Where
ziz_i
the new output vector of token ii
aija_{ij}
token ii's attention weight on token jj (from Step 4)
vjv_j
token jj's value vector
j, nj,\ n
index over all nn tokens, including j=ij = i
Read it aloudz i equals the sum, over every token j, of the attention weight a i j times the value vector v j.
Formula in plain wordsIn the lecturer's words: "when I express myself, I take how much I attend to each token, times how that token represents itself, and add it all up." Because the weights sum to 1, ziz_i is a weighted average of the value vectors: tokens with high attention weight pull ziz_i toward their values, tokens with weight near 0 barely contribute. The result is again a vector of the same size, so it can be fed into the next layer. This is how bank in "river bank" and bank in "closest bank" end up with different vectors: they attend to different neighbours.

Putting the five steps together in matrix form gives the famous formula from Attention Is All You Need:

Attention(Q,K,V)=softmax ⁣(QK⊤dk)V \text{Attention}(Q,K,V)=\text{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V
Where
Q,K,VQ, K, V
the stacked query, key and value matrices (nn rows each)
K⊤K^{\top}
the transpose of KK, so that QK⊤QK^{\top} is n×nn \times n
dk\sqrt{d_k}
the scaling factor from the key dimension
softmax(⋅)\text{softmax}(\cdot)
applied to each row separately
Attention(Q,K,V)\text{Attention}(Q,K,V)
the nn-row output matrix whose row ii is ziz_i
Read it aloudAttention of Q, K and V equals the softmax of Q times K transpose over the square root of d k, multiplied by V.
Formula in plain wordsThis single line is Steps 4 and 5 for all tokens at once. The part inside the softmax scores every query against every key; the softmax turns each row into weights; multiplying by VV mixes the value vectors with those weights, giving one new row ziz_i per token. Everything is matrix multiplication, which is why the whole sequence can be processed in parallel on a GPU. Changing WQW^Q or WKW^K changes who attends to whom; changing WVW^V changes what gets passed along.

7.3 Self-attention by hand

Worked example 11Self-attention for the · cat · sat, every number computed

Illustrative toy sentence of three tokens with 2-dimensional input vectors (dk=2d_k = 2): xthe=[1, 0], xcat=[0, 2], xsat=[1, 1] x_{\text{the}}=[1,\,0],\ x_{\text{cat}}=[0,\,2],\ x_{\text{sat}}=[1,\,1] . The projection matrices are picked by hand (a real model learns them):

WQ=[1011]WK=[1102]WV=[1101] W^Q=\begin{bmatrix}1 & 0\\ 1 & 1\end{bmatrix}\qquad W^K=\begin{bmatrix}1 & 1\\ 0 & 2\end{bmatrix}\qquad W^V=\begin{bmatrix}1 & 1\\ 0 & 1\end{bmatrix}
1
Project. Each qi=xiWQq_i = x_i W^Q is a (1×2)(2×2)(1\times 2)(2\times 2) product (row · column, as in Section 2.3). For cat: qcat=[0,2] WQ=[0⋅1+2⋅1,    0⋅0+2⋅1]=[2, 2] q_{\text{cat}} = [0,2]\,W^Q = [0{\cdot}1 + 2{\cdot}1,\;\; 0{\cdot}0 + 2{\cdot}1] = [2,\,2] . All nine projections:
Tokenqi=xiWQ q_i = x_iW^Q ki=xiWK k_i = x_iW^K vi=xiWV v_i = x_iW^V
the[1, 0][1, 1][1, 1]
cat[2, 2][0, 4][0, 2]
sat[2, 1][1, 3][1, 2]
2
Raw scores QK⊤QK^{\top}: the dot product of every query with every key, a 3×33\times 3 matrix.
qi⋅kj q_i \cdot k_j k(the) = [1,1]k(cat) = [0,4]k(sat) = [1,3]
q(the) = [1,0]101
q(cat) = [2,2]488
q(sat) = [2,1]345
For example, row cat, column sat: qcat⋅ksat=2×1+2×3=8 q_{\text{cat}}\cdot k_{\text{sat}} = 2{\times}1 + 2{\times}3 = 8 .
3
Scale by dk=2≈1.414 \sqrt{d_k} = \sqrt{2} \approx 1.414 : row cat becomes [2.83,  5.66,  5.66] [2.83,\; 5.66,\; 5.66] .
4
Softmax row cat:e2.83≈16.9e5.66≈286.4e5.66≈286.4sum≈589.7acat≈[ 0.03,  0.485,  0.485 ] \begin{aligned} e^{2.83} &\approx 16.9 \\ e^{5.66} &\approx 286.4 \\ e^{5.66} &\approx 286.4 \\ \text{sum} &\approx 589.7 \\ a_{\text{cat}} &\approx [\,0.03,\; 0.485,\; 0.485\,] \end{aligned} The same for the other rows gives the full attention matrix (each row sums to 1):
A≈[0.400.200.400.030.4850.4850.140.280.58]←the←cat←sat A \approx \begin{bmatrix}0.40 & 0.20 & 0.40\\ 0.03 & 0.485 & 0.485\\ 0.14 & 0.28 & 0.58\end{bmatrix}\begin{matrix}\leftarrow \text{the}\\ \leftarrow \text{cat}\\ \leftarrow \text{sat}\end{matrix}
keys (attended to) the cat sat q: the q: cat q: sat 0.400.200.40 0.030.4850.485 0.140.280.58 row = query token; each row sums to 1
The computed attention weights: cat spends almost all its attention on cat and sat, and almost none on the.
5
Weighted sum of values for cat:
zcat=0.03 [1,1]+0.485 [0,2]+0.485 [1,2]≈[ 0.52,  1.97 ] z_{\text{cat}} = 0.03\,[1,1] + 0.485\,[0,2] + 0.485\,[1,2] \approx [\,0.52,\; 1.97\,]
Resultzcat≈[0.52,1.97]z_{\text{cat}} \approx [0.52, 1.97]: cat's new vector blends mostly information from cat itself and sat, so its representation has absorbed its context. Repeating for every token gives zthe≈[0.80,1.60] z_{\text{the}} \approx [0.80, 1.60] and zsat≈[0.72,1.86] z_{\text{sat}} \approx [0.72, 1.86] . Every token now carries information about the words around it.
Common trapA student asked in the lecture: "Is the attention matrix symmetric?" No. In the raw scores above, the → cat scores 0 while cat → the scores 4. Attention captures pairwise and directional relationships; in the lecturer's words, "I care about that guy; that guy doesn't care about me." Each token computes its own distribution over everyone else. Formally, qi⋅kj≠qj⋅ki q_i \cdot k_j \neq q_j \cdot k_i in general, because WQ≠WK W^Q \neq W^K .

7.4 What makes a real transformer "fat" and "tall"

The five steps describe one attention head in one layer. Real transformers scale this up in two directions, in the lecturer's memorable phrasing:

DirectionWhat it meansEffect
"Fat": multi-head attentionWithin a layer, the model keeps several independent copies of WQ,WK,WVW^Q, W^K, W^V; each copy is an attention head with its own learned weights. Each head computes its own attention pattern in parallel (one might track subject–verb links, another adjacent words), and their outputs are combined.Makes each layer wider
"Tall": stacked layersEach token goes in as a vector and comes out as a vector of the same dimension, so the whole block can be applied again: output of layer 1 → input of layer 2 → … BERT-base stacks 12 layers, each refining the representations further.Makes the model deeper
x₁ x₂ x₃ input embeddings one encoder block, repeated ×N (BERT-base: N = 12) multi-head self-attention each head: own Q, K, V weights ("fat") add & layer norm (residual) multilayer perceptron (MLP) project up, then back down add & layer norm z₁ z₂ z₃ same count, same size → stackable ("tall") fed back in as the next layer's input
Inside one encoder block, as hand-drawn on the lecture slides: multi-head self-attention → add & layer norm → MLP → add & layer norm. Input shape equals output shape, so blocks stack.

Two supporting components follow the attention step in every block:

For how a stack of these layers refines one vector per token, see the companion explainer The encoder backbone, and why it cannot generate. Sheet "4 Multi-Head" of the spreadsheet workbook shows two heads side by side.

7.5 What the encoder produces, and why GPUs love it

After the final layer the encoder outputs one encoded vector per input token: the same count and dimension as the input, where each vector now represents "the semantics of the word with respect to the sentence". For a single vector for the whole sentence there are two standard choices: take the [CLS] token's output vector (train the model to summarise the sentence there), or take the mean of all token vectors (mean pooling). Both are used heavily in retrieval (Section 12 and Week 5).

encoder output: one vector per token z[CLS] z the z cat z sat z[SEP] [CLS] pooling: keep z[CLS] mean pooling: average all z
Two standard sentence vectors: the [CLS] output vector, or the mean of all token output vectors.
vsentence  =  1n∑i=1nzi(mean pooling)orvsentence=z[CLS] v_{\text{sentence}} \;=\; \frac{1}{n}\sum_{i=1}^{n} z_i \qquad\text{(mean pooling)}\qquad\text{or}\qquad v_{\text{sentence}} = z_{[\text{CLS}]}
Where
ziz_i
the encoder's final output vector for token ii
nn
the number of tokens pooled (padding tokens are left out)
z[CLS]z_{[\text{CLS}]}
the final output vector of the [CLS] token
vsentencev_{\text{sentence}}
one vector for the whole text, same dimension as each ziz_i
Read it aloudThe sentence vector is one over n times the sum of all n token output vectors (mean pooling), or else simply the output vector of the CLS token.
Formula in plain wordsMean pooling adds the token vectors position by position and divides by the count, so every token contributes equally and the result sits at the centre of the sentence's token vectors. [CLS] pooling trusts one designated token: because [CLS] attends to every token in every layer, training can teach it to act as a summary. Either way the sentence becomes a single vector, which can then be compared with another text's vector by a dot product or cosine similarity. Which choice works better depends on how the model was trained.

Worked example 12A sentence vector by mean pooling

Use the three output vectors from Worked example 11: zthe≈[0.80,1.60]z_{\text{the}} \approx [0.80, 1.60], zcat≈[0.52,1.97]z_{\text{cat}} \approx [0.52, 1.97], zsat≈[0.72,1.86]z_{\text{sat}} \approx [0.72, 1.86].

v=13([0.80,1.60]+[0.52,1.97]+[0.72,1.86])=13[2.04, 5.43]≈[0.68, 1.81] \begin{aligned} v &= \frac{1}{3}\big([0.80,1.60] + [0.52,1.97] + [0.72,1.86]\big) \\ &= \frac{1}{3}[2.04,\ 5.43] \\ &\approx [0.68,\ 1.81] \end{aligned}
ResultThe sentence "the cat sat" is represented by the single vector [0.68,1.81][0.68, 1.81], ready to be compared with a query vector.
Key ideaParallelism (in-class Exercise 1): the encoder processes all input tokens simultaneously. All those dot products are batched into large matrix multiplications, exactly what GPUs are built for (a CPU computes in sequence; a GPU computes thousands of small operations at once). This is why transformers and GPUs made each other successful.
PracticeIn Worked example 11, compute the raw score of sat attending to the and of the attending to sat. Are they equal?
Use qsat=[2,1]q_{\text{sat}} = [2,1], kthe=[1,1]k_{\text{the}} = [1,1], qthe=[1,0]q_{\text{the}} = [1,0], ksat=[1,3]k_{\text{sat}} = [1,3].
Answerqsat⋅kthe=2+1=3qthe⋅ksat=1+0=1 \begin{aligned} q_{\text{sat}}\cdot k_{\text{the}} &= 2 + 1 = 3 \\ q_{\text{the}}\cdot k_{\text{sat}} &= 1 + 0 = 1 \end{aligned} They differ, confirming that attention is directional: queries and keys come from different learned matrices.
PracticeWhy can BERT-base repeat the same block structure 12 times, and what would break if the MLP output a vector of a different size?
AnswerEach block takes nn vectors of dimension 768 and returns nn vectors of dimension 768, so any block's output is a valid input for the next one. If the MLP changed the size, the residual addition and the next layer's WQ,WK,WVW^Q, W^K, W^V (which expect 768 inputs) would no longer fit the shapes, and the stack would break.

8. Steps 3–4: the decoder and generation

Sources: slides 16–19, 23 · Video 1 (Overview of the transformer) · Video 7 (Autoregressive decoders) · Lecture transcript (decoder, linear-layer Q&A, sampling)

"Don't worry too much about the decoder; it's almost the same." The decoder is built from the same pieces as the encoder: tokenise the input (think of it as your prompt), build input embeddings, run stacked blocks of self-attention + MLP. Two things differ, and both exist because the decoder's job is generation.

8.1 Difference 1: masked (causal) attention

In the decoder each token may attend only to itself and earlier tokens. During generation the future tokens do not exist yet: when the model generates the first word of its answer, there is nothing after it to look at. The lecturer: "It's not that we don't want it; we don't need it. How could it look at future tokens? They're not yet generated." In practice the scores for future positions are set to −∞-\infty before the softmax, which blanks out the upper-right triangle of the attention matrix (compare the two grids in Section 9).

sijmask={qi⋅kjdkif j≤i−∞if j>iaij=esijmask∑l=1nesilmask s^{\text{mask}}_{ij} = \begin{cases} \dfrac{q_i\cdot k_j}{\sqrt{d_k}} & \text{if } j \le i \\[6pt] -\infty & \text{if } j > i \end{cases} \qquad a_{ij} = \frac{e^{s^{\text{mask}}_{ij}}}{\sum_{l=1}^{n} e^{s^{\text{mask}}_{il}}}
Where
ii
the position of the token doing the attending
jj
the position being attended to
j≤ij \le i
“itself or earlier”: position jj is position ii or comes before it
qi⋅kj/dkq_i\cdot k_j/\sqrt{d_k}
the ordinary scaled score from Section 7
−∞-\infty
a score that becomes e−∞=0e^{-\infty} = 0 after exponentiation
sijmasks^{\text{mask}}_{ij}
the masked score
aija_{ij}
the resulting attention weight (0 for every future position)
Read it aloudThe masked score s i j equals q i dot k j over the square root of d k when j is at or before i, and minus infinity when j is after i. The weight a i j is the softmax of the masked scores across row i.
Formula in plain wordsThe mask keeps the attention recipe unchanged for past and present tokens and forces every future token's weight to exactly 0, because e−∞=0e^{-\infty} = 0. The remaining weights in each row are renormalised by the softmax, so they still sum to 1. The first token can only attend to itself (weight 1); the last token sees the whole sequence, so its row matches the unmasked encoder row. Training with this mask means the model learns to predict each next token from the past alone, which is exactly the situation it faces during generation.

Worked example 13The causal mask on the · cat · sat

Reuse the scaled scores from Worked example 11 and mask every future position.

1
Row the: [0.71,−∞,−∞][0.71, -\infty, -\infty] → softmax [1,0,0][1, 0, 0]. The first token attends only to itself.
2
Row cat: [2.83,5.66,−∞][2.83, 5.66, -\infty]. Apply softmax to the two unmasked scores:e2.83≈16.9e5.66≈286.4sum≈303.3acat≈[0.056,0.944,0] \begin{aligned} e^{2.83} &\approx 16.9 \\ e^{5.66} &\approx 286.4 \\ \text{sum} &\approx 303.3 \\ a_{\text{cat}} &\approx [0.056, 0.944, 0] \end{aligned}
3
Row sat: nothing is masked (it is the last token), so it stays [0.14,0.28,0.58][0.14, 0.28, 0.58].
keys (attended to) the cat sat q: the q: cat q: sat 1 masked masked 0.056 0.944 masked 0.14 0.28 0.58 masked = future token, weight exactly 0
With the causal mask, each token spreads its attention over itself and earlier tokens only.
4
New vectors:zthe=1⋅vthe=[1,1]zcat=0.056 [1,1]+0.944 [0,2]≈[0.06,1.94]zsat≈[0.72,1.86](as before) \begin{aligned} z_{\text{the}} &= 1\cdot v_{\text{the}} = [1, 1] \\ z_{\text{cat}} &= 0.056\,[1,1] + 0.944\,[0,2] \approx [0.06, 1.94] \\ z_{\text{sat}} &\approx [0.72, 1.86] \quad \text{(as before)} \end{aligned}
ResultMasked attention matrix [1000.0560.94400.140.280.58]\begin{bmatrix}1 & 0 & 0\\ 0.056 & 0.944 & 0\\ 0.14 & 0.28 & 0.58\end{bmatrix}. Only the rows of earlier tokens change; cat can no longer draw on sat, so its vector [0.06,1.94][0.06, 1.94] now reflects only "the cat".

8.2 Difference 2: the output head, from vector to next-token probabilities

The encoder ends with vectors; the decoder must end with a choice of next token. The lecturer wrote the full chain out on paper for a student, and it is one matrix multiplication plus one softmax:

1
After the final block, take the last-layer vector of the last token, hh, with dimension nn (BERT: 768; Llama: 2,048).
2
Multiply by a learned linear projection layer WW of shape n×mn \times m, where mm is the vocabulary size (about 30,000): (1×n)(n×m)=(1×m)(1\times n)(n \times m) = (1\times m). The result is one raw score, a logit, for every token in the vocabulary.
3
Apply softmax to turn the logits into probabilities between 0 and 1 that sum to 1.
4
Pick the next token from this distribution.
p  =  softmax(h W),h⏟1×n  W⏟n×m=ℓ⏟1×m p \;=\; \text{softmax}\big(h\,W\big), \qquad \underbrace{h}_{1\times n}\;\underbrace{W}_{n\times m} = \underbrace{\ell}_{1\times m}
Where
hh
the last token's final-layer hidden vector (dimension nn)
WW
the learned output projection ("linear layer"), n×mn \times m
mm
the vocabulary size (about 30,000 for BERT-style vocabularies)
ℓ=hW\ell = hW
the logits: one raw score per vocabulary token
pp
the next-token probability distribution
pwp_w
the probability of vocabulary token ww
Read it aloudp equals the softmax of h times W, where a one by n vector h times an n by m matrix W gives a one by m vector of logits.
Formula in plain wordsEach column of WW acts like a learned "template" for one vocabulary token; the dot product of hh with that column scores how well the context fits that token next. Softmax turns the mm scores into a probability distribution. A token whose column points in the same direction as hh gets a high logit and a large probability; negative logits simply become small probabilities. This is the vector × matrix operation of Worked example 4, scaled up from 3 outputs to about 30,000.
h (1 × n) 0.9 −0.2 0.4 last token's last-layer vector × W (n × m) learned linear projection m = vocab (~30K) = logits (1 × m): one score per token 2.0the 1.0a 0.5cat −1.0pie 0.0… softmax probabilities (sum to 1) 0.56 0.21 0.13 0.03 0.08 the a cat pie … ↑ picked (greedy)
The decoder's output head: one vector × matrix multiplication maps the hidden vector onto the whole vocabulary, and softmax turns the scores into next-token probabilities.

Worked example 14The output head with numbers

Illustrative vocabulary of 5 tokens (the, a, cat, pie, …) with logits [2.0, 1.0, 0.5, −1.0, 0.0][2.0,\ 1.0,\ 0.5,\ -1.0,\ 0.0].

1
Exponentiate: e2.0=7.39e1.0=2.72e0.5=1.65e−1.0=0.37e0.0=1.00sum=13.13 \begin{aligned} e^{2.0} &= 7.39 \\ e^{1.0} &= 2.72 \\ e^{0.5} &= 1.65 \\ e^{-1.0} &= 0.37 \\ e^{0.0} &= 1.00 \\ \text{sum} &= 13.13 \end{aligned}
2
Divide by the sum: P=[ 0.56,  0.21,  0.13,  0.03,  0.08 ] P = [\,0.56,\; 0.21,\; 0.13,\; 0.03,\; 0.08\,] .
ResultThe highest-probability token is the (0.56); under greedy decoding it becomes the next token. Softmax handles the negative logit of pie without trouble: it just becomes a small probability (0.03).

8.3 The generation loop and sampling

One pass through the model produces one token. To generate text, the predicted token is appended to the input and the whole process runs again ("generate, put it back, generate, put it back") until the model produces the [EOS] token and stops. Section 13 formalises this as autoregressive generation.

Watch outFrom the lecture Q&A: always choosing the single most probable token (greedy decoding) can create problems such as repetitive, degenerate text. Real systems use sampling strategies, for example keeping only the top tokens above a probability threshold and picking one of them at random. That one random choice changes the context for every following token, which is why the same question to ChatGPT gives a different answer every time.
PracticeA model has hidden size 2,048 and a vocabulary of 32,000 tokens. What shape is the output projection, how many numbers does it produce per step, and why do we take only the last token's vector?
AnswerWW is 2048×320002048 \times 32000, and hWhW gives 32,000 logits (one per vocabulary token), which softmax turns into 32,000 probabilities. Only the last token's vector is used because, under the causal mask, it is the only position that has attended to the whole prompt, and the next token follows the last one.

9. Encoder vs decoder

Sources: slides 20–23 · Video 2 (Encoder vs decoder) · Lecture transcript (using the parts separately, Exercise 1)

In the original transformer the encoder and decoder work as one integrated system: the encoder converts text into vector representations; the decoder takes those vectors plus the ⟨start⟩ token and predicts the next token. The crucial insight of slides 20–21 is that they can be used separately. The lecturer framed it as two questions: "If my task is not generation, do I need the decoder? If my task is only generation, do I need the encoder?" Answering "no" to each led to some of the most important breakthroughs in AI:

Encoder: "understand and represent"Decoder: "generate and predict"
PurposeTakes input text; creates a rich, dense vector representation; understands context and relationshipsTakes input (optionally with encoded representations) and generates output step by step, predicting next-token probabilities
AttentionBidirectional: every token can look at all tokens before and after it; the encoder sees the entire sentence at onceMasked (unidirectional): a token sees only previous words
ExampleInput How to fix laptop? → output dense vector [0.23, −0.45, 0.67, …] capturing the sentence's meaningInput UQ is (+ encoded vectors, optional) → output the → University → of → Queensland → …
ApplicationsSearch and classification using embeddings: tasks that need the full meaning of textTranslation, text generation, and all modern LLM usage
Encoder: bidirectional Decoder: masked (causal) howtofixlaptop? UQistheUni…of howtofixlaptop? UQistheUni…of every token sees every token (before and after) grey = masked future tokens
Attention patterns. Left: an encoder token attends to the whole sequence (bidirectional). Right: a decoder token attends only to itself and earlier tokens (masked, or causal, attention).
Key ideaEncoders suit search and classification because they build one deep understanding of the whole text at once: to capture a sentence's meaning, every token needs to see the tokens both before and after it (in "the bank makes loans to clients", the later token loans tells you which bank is meant). Decoders suit generation because they predict what comes next, one token at a time. Same core technology, adapted to different purposes.
PracticeYou need to embed ten million product descriptions for semantic search, and separately to write product summaries. Which transformer part suits each task, and which attention pattern does each use?
AnswerEmbedding for search is an understanding task: use an encoder (bidirectional attention, every token sees the whole description, all tokens processed in parallel). Writing summaries is generation: use a decoder (masked attention, one token at a time until [EOS]).

10. How transformers are trained

Sources: slides 41, 44, 53 · Video 6 (BERT) · Video 7 (Autoregressive decoders) · Lecture transcript (training, data and lawsuits)

A randomly initialised model knows nothing: the embedding tables, the WQ,WK,WVW^Q, W^K, W^V matrices in every head of every layer, the MLP weights and the output projection all start as meaningless random numbers. Training adjusts all those weights until the model "understands" language. The practical problem: this needs colossal amounts of training signal without human labelling. The solution used by both BERT and GPT is self-supervised learning: take the whole web as training data and let the text itself provide the labels.

Encoder (BERT): masked LM The bank [MASK] loans to clients predict: makes uses context on both sides Decoder (GPT): next-token prediction UQ→ is UQ is→ the UQ is the→ University UQ is the University→ of uses only the tokens before
Both objectives turn raw web text into labelled examples for free: hide a word and predict it (encoder), or predict each next word from the words before (decoder).
ObjectiveHow it worksLecture illustration
Masked language modelling (encoder, BERT-style)Take web text, randomly replace a fraction of its tokens with [MASK], and train the model to predict the masked tokens from all the surrounding context (both directions, which is why bidirectional attention matters). Compare prediction and real token, measure the error (loss), nudge the weights to reduce it, and repeat over the whole web. Details and a loss calculation in Section 11."The bank [MASK] loans to clients" → predict makes
Next-token prediction (decoder, GPT-style)Even simpler. Feed the first token: can you predict the second? Feed the first two: predict the third. Feed the first three: predict the fourth. Every position of every sentence on the web becomes a training example.Train on your chat history and the model learns what you would reply next ("if you've written bullshit a thousand times, it knows the next token is bullshit"). The model is never taught grammar; it learns how people write by predicting them, billions of times.

Worked example 15One sentence, two training signals

Take the 6-token sentence UQ is the University of Queensland.

1
Next-token prediction: every prefix predicts the following token: UQ → is, UQ is → the, UQ is the → University, … University → of, … of → Queensland. That is 5 labelled examples from one sentence.
2
Masked LM at 15%: 0.15×6=0.90.15 \times 6 = 0.9, so about one token is masked, say UQ is the [MASK] of Queensland, and the model predicts University from both sides.
ResultNeither signal needed a human annotator: the labels are the text itself. Next-token prediction extracts a training target at every position; masked LM extracts targets only at masked positions but lets each prediction use context from both directions.
Watch outThe data-hunger side effect (from the lecture): model quality scales with training data, so companies scrape everything they can find, which is why several LLM providers have been sued by publishers over copyrighted training data. High-quality data, beyond sheer quantity, is one of the things that actually makes these models succeed.
Exam angleBoth schemes are unsupervised in the labelling sense: no human wrote relevance judgements or translation pairs. Yet the resulting weights encode meaning, synonymy and context, exactly the semantic understanding BM25 lacks (Section 1). Weeks 5–8 are about transferring that understanding to ranking with comparatively tiny amounts of labelled data (fine-tuning).
PracticeWhy would next-token prediction be a poor pre-training objective for an encoder with bidirectional attention?
What can a bidirectional token see when it tries to predict the token after it?
AnswerWith bidirectional attention every position can already see the next token, so "predicting" it becomes trivial copying and the model learns nothing useful. Masking hides the target from all positions, which forces genuine use of context. Decoders avoid the problem the other way: the causal mask hides the future, so next-token prediction stays a real prediction task.

11. BERT

Sources: slides 40–48, 50 · Video 6 (BERT) · Lecture transcript (MLM masking rate, scaling)

Key ideaBERT = Bidirectional Encoder Representations from Transformers. Released by Google in October 2018 (Devlin, Chang, Lee and Toutanova, NAACL 2019). It is an encoder-only transformer (no decoder) with 12 layers (BERT-base) and bidirectional attention: it sees the entire sentence at once. BERT revolutionised 11 NLP tasks, reaching state-of-the-art effectiveness on many of them, including information retrieval.

11.1 The two-stage training approach

STAGE 1 · Pre-training self-supervised, on huge amounts of UNLABELLED text (Wikipedia, web) learns general language understanding STAGE 2 · Fine-tuning supervised, on SMALL amounts of LABELLED task examples specialises: classification, ranking, QA …
Learn general language understanding from unlabelled text first, then adapt to a specific task with a few labelled examples; groundbreaking at the time.

11.2 Ingredients and input processing

BERT's success comes from combining three pre-training ingredients (slide 42): (1) an encoder-only transformer architecture with lots of parameters, (2) massive amounts of text data (Wikipedia, web pages), and (3) enormous computational power (cloud TPU pods were essential at the time).

Input starts with the familiar embedding layers, token + segment + position embeddings added together (Section 5), then flows through multiple transformer encoder layers with self-attention (Section 7). The output is a sequence of rich vector representations, one per input token, where every token has attended to every other token.

11.3 Pre-training task 1: masked language modelling (MLM)

Randomly mask about 15% of the input tokens and train the model to predict them from all the surrounding (bidirectional) context. The BERT paper and slide 50 use 15%; in class the lecturer quoted a loosely remembered 30%. The exact fraction is a design choice; the principle is what matters. The loss measures how well BERT predicts the masked tokens:

LMLM  =  −∑i ∈ Mlog⁡P(xi ∣ x∖M) \mathcal{L}_{\text{MLM}} \;=\; -\sum_{i\,\in\,M}\log P\big(x_i \,\big|\, x_{\setminus M}\big)
Where
MM
the set of masked positions in the sentence
xix_i
the true (original) token at masked position ii
x∖Mx_{\setminus M}
the sentence with the masked positions hidden: all the visible context
P(xi∣x∖M)P(x_i \mid x_{\setminus M})
the probability BERT assigns to the correct token at position ii, given that context
log⁡\log
natural logarithm
LMLM\mathcal{L}_{\text{MLM}}
the masked language modelling loss for this sentence (lower is better)
Read it aloudThe MLM loss equals minus the sum, over every masked position i, of the log of the probability of the true token x i given the unmasked rest of the sentence.
Formula in plain wordsFor each hidden word, look at the probability the model gave to the correct word and take minus its log. A confident correct guess (probability near 1) costs almost nothing; a poor guess (probability near 0) costs a lot, because −log⁡p-\log p grows quickly as pp shrinks. Summing over the masked positions gives the loss for the sentence, and training adjusts the weights to push it down across the whole web. This forces the model to develop a deep understanding of language and context.

Worked example 16MLM loss on the lecture's sentence

Original (slide 47): "The Mongol invasion of Europe in the 13th century was the conquest of much of Europe by the Mongol Empire." BERT masks century and Europe: "… in the 13th [MASK] was the conquest of much of [MASK] by the Mongol Empire." Using bidirectional context, it predicts:

Masked positionPredictionConfidence PPLoss contribution −log⁡P-\log P
… 13th [MASK] was …century0.94−ln⁡0.94≈0.062 -\ln 0.94 \approx 0.062
… much of [MASK] by …Europe0.97−ln⁡0.97≈0.030 -\ln 0.97 \approx 0.030
LMLM≈0.062+0.030=0.092 \mathcal{L}_{\text{MLM}} \approx 0.062 + 0.030 = 0.092
ResultA total loss of about 0.092: very low, because the model is confident and correct. Early in training the model might give the right word only P=0.10P = 0.10, costing −ln⁡0.10≈2.30-\ln 0.10 \approx 2.30 per mask, a large loss that pushes the weights to improve. The high confidences (94%, 97%) show how well the trained model captures historical and geographical relationships.

11.4 Pre-training task 2: next sentence prediction (NSP)

BERT also trains on sentence pairs, learning to predict whether the second sentence naturally follows the first (a binary classification on the [CLS] output). This teaches relationships between sentences, which matter for question answering and document ranking.

Sentence ASentence BLabel
My dog is cute.He likes playing.IsNext
My dog is cute.The stock market fell 2% today.NotNext

11.5 Model size matters

ModelParametersConfiguration
BERT-base110 million12 layers
BERT-large345 millionmore layers, larger hidden dimension, more attention heads

BERT-large consistently outperforms BERT-base, including on ranking and on small-scale tasks, and more training steps give higher accuracy (slide 46). This was an early hint of the scaling laws seen throughout today's large language models. The lecturer noted that billion-parameter models now count as small and frontier models run to trillions, though the industry has since had to balance raw scale against efficiency ("a user says hello, and answering hi costs me $20; I can't do that").

11.6 Fine-tuning to different tasks

One input: classify [CLS] one text [SEP] BERT class label from [CLS] Two inputs: QA, ranking [CLS] q [SEP] d [SEP] BERT relevance score from [CLS] Per-token labelling [CLS] w₁ w₂ w₃ [SEP] BERT tag tag tag
Slide 48's fine-tuning patterns: sentence-level tasks read the [CLS] output, two-input tasks put the texts in different segments, and sequence labelling reads every token's output.

11.7 BERT's limitations

LimitationDetail (slide 50)
Input length constraintInput is capped at 512 tokens, including special tokens. Long documents cannot be processed directly; they need chunking or truncation, which loses information.
Cannot generate text autoregressivelyMLM pre-training randomly masks about 15% of tokens and predicts them with bidirectional context. BERT is never trained on next-token prediction, so it cannot produce coherent sequences token by token like GPT. (Section 15 covers a modern twist: diffusion language models generate with encoder backbones.)

The second limitation is exactly where decoder-only models such as GPT come in (Section 13).

PracticeDuring training BERT gives the correct masked word probability 0.5 at one position and 0.05 at another. Compute each loss contribution and say which position drives learning more.
Answer−ln⁡0.5≈0.69−ln⁡0.05≈3.00 \begin{aligned} -\ln 0.5 &\approx 0.69 \\ -\ln 0.05 &\approx 3.00 \end{aligned} The second position contributes over four times the loss, so its error drives the larger weight updates: the loss punishes confident failure much more than mild uncertainty.

12. BERT in information retrieval

Sources: slide 49 · Video 6 (BERT) · Lecture transcript (ranking challenge: [CLS] or mean vectors, ColBERT-style interaction)

For IR, BERT is used in two main ways (slide 49). Both significantly improve search quality: dense retrievers are the topic of Week 5 and BERT-based rerankers of Week 8.

Dense retriever (Week 5) Query Document BERT BERT V(query) V(doc) similarity: cos(V(q), V(d)) encoded separately: fast, indexable Reranker, cross-encoder (Week 8) Query + [SEP] + Document BERT relevance score from [CLS] read together: more accurate, but one BERT run per (q, d) pair
Two ways to use BERT for search. Left, the dense retriever (bi-encoder): encode query and documents separately and compare vectors. Right, the reranker (cross-encoder): concatenate query and document with [SEP] and let BERT's classification head predict relevance directly.
Dense retriever (bi-encoder)Reranker (cross-encoder)
InputQuery and document encoded separately[CLS] query [SEP] document [SEP], read together (segments A and B, Section 5)
ScoreCosine similarity or dot product of V(q)V(q) and V(d)V(d)Classification score from the [CLS] head
CostDocument vectors computed once at indexing time; fast searchOne BERT run per (query, document) pair; used to rerank a short candidate list
AccuracyGoodHigher: every query token attends to every document token
CourseWeek 5Week 8 (this is the monoBERT style of reranker)

From the in-class discussion: for the dense-retriever route, the sentence vector V(⋅)V(\cdot) usually comes from either the [CLS] output vector or the mean of all token output vectors (Section 7.5). The relevance score is then a cosine similarity or dot product: "either way, you can have a score, as long as you train it." A student proposed the more advanced version: keep one vector per token for both query and document, compare every query token with every document token (a full similarity matrix, like attention), and aggregate the per-token similarities into a final score. That is the intuition behind ColBERT-style late interaction: more accurate than a single vector, at the cost of storing many vectors per document (later weeks).

cos⁡(q,d)=q⋅d∥q∥ ∥d∥=∑iqidi∑iqi2 ∑idi2 \cos(q,d)=\frac{q\cdot d}{\lVert q\rVert\,\lVert d\rVert} = \frac{\sum_{i} q_i d_i}{\sqrt{\sum_i q_i^2}\,\sqrt{\sum_i d_i^2}}
Where
q,dq, d
the query vector V(q)V(q) and document vector V(d)V(d) from BERT
q⋅dq\cdot d
their dot product (Section 2.1)
qi,diq_i, d_i
entry ii of each vector
∥q∥,∥d∥\lVert q\rVert, \lVert d\rVert
vector lengths (Euclidean norms)
cos⁡(q,d)\cos(q,d)
cosine similarity, between −1 and 1
Read it aloudThe cosine of q and d equals q dot d divided by the length of q times the length of d; that is, the sum of q i times d i, over the square root of the sum of q i squared times the square root of the sum of d i squared.
Formula in plain wordsThis is the same cosine used by the vector space model in Week 3, now applied to learned dense vectors in place of term-weight vectors. The numerator rewards vectors whose large entries line up; dividing by both lengths removes the effect of vector size, so only the direction counts. A value near 1 means the texts point the same way in meaning space, near 0 means unrelated. Because BERT places synonyms close together, two texts can score high while sharing no words at all. When vectors are normalised to length 1, cosine and dot product give the same ranking.

Worked example 17Dense retrieval fixes the fix-laptop query

Encode the query and two documents into illustrative 3-dimensional BERT vectors:

TextVector
qq = V("how to fix laptop")[0.90, 0.30, 0.10]
d1d_1 = V("computer repair guide for damaged notebooks")[0.80, 0.40, 0.00]
d2d_2 = V("river bank fishing spots")[0.10, 0.20, 0.95]
1
Document 1:q⋅d1=0.9(0.8)+0.3(0.4)+0.1(0)=0.84∥q∥=0.91≈0.954∥d1∥=0.80≈0.894cos⁡(q,d1)=0.840.954×0.894≈0.985 \begin{aligned} q\cdot d_1 &= 0.9(0.8)+0.3(0.4)+0.1(0) = 0.84 \\ \lVert q\rVert &= \sqrt{0.91}\approx 0.954 \\ \lVert d_1\rVert &= \sqrt{0.80}\approx 0.894 \\ \cos(q,d_1) &= \frac{0.84}{0.954\times 0.894}\approx 0.985 \end{aligned}
2
Document 2:q⋅d2=0.09+0.06+0.095=0.245∥d2∥=0.9525≈0.976cos⁡(q,d2)=0.2450.954×0.976≈0.263 \begin{aligned} q\cdot d_2 &= 0.09+0.06+0.095 = 0.245 \\ \lVert d_2\rVert &= \sqrt{0.9525}\approx 0.976 \\ \cos(q,d_2) &= \frac{0.245}{0.954\times 0.976}\approx 0.263 \end{aligned}
ResultRanking d1≻d2d_1 \succ d_2 by a wide margin (0.985 against 0.263), although d1d_1 shares zero terms with the query. BERT's embeddings place fix/repair and laptop/notebook close together, achieving the semantic matching BM25 could not (Worked example 1). Week 5 (dense retrieval) builds on exactly this.
PracticeA collection has 8 million passages and you must answer queries in under 100 ms. Where does each BERT usage fit in the pipeline, and why?
AnswerUse the dense retriever (or BM25) for first-stage retrieval: passage vectors are computed once at indexing time, so a query needs one BERT pass plus fast vector search. Use the cross-encoder reranker only on the top candidates (say 100): it needs one BERT pass per (query, passage) pair, far too slow for 8 million passages, but its joint reading gives the more accurate final order.

13. Autoregressive decoders and GPT

Sources: slides 51–54 · Video 7 (Autoregressive decoders)

Key ideaIn an autoregressive text-generation model, at each time step tt the model takes the sequence of previous tokens y<ty_{<t} as input and outputs a new token y^t\hat{y}_t (slide 51). Generation happens step by step: generate one token, append it, and use the expanded sequence to generate the next. This loop is how today's LLMs produce coherent text. Mechanically, each step is the linear layer + softmax of Section 8.
y^t=arg⁡max⁡y∈V  P(y ∣ y<t) \hat{y}_t=\arg\max_{y \in \mathcal{V}}\;P\big(y \,\big|\, y_{<t}\big)
Where
tt
the current time step (position being generated)
y<ty_{<t}
all tokens before step tt: the prompt plus everything generated so far
V\mathcal{V}
the vocabulary
yy
one candidate token from the vocabulary; the arg max tries every such token
P(y∣y<t)P(y \mid y_{<t})
the model's probability that yy comes next (the softmax output of Section 8)
arg⁡max⁡\arg\max
"the yy that gives the largest value"
y^t\hat{y}_t
the token chosen at step tt (hat = predicted)
Read it aloudy hat t is the vocabulary token y that maximises the probability of y given all the tokens before t.
Formula in plain wordsThis is greedy decoding: at each step, pick the single most probable next token given everything so far. The chosen token then becomes part of y<t+1y_{<t+1} for the next step, so an early choice shapes everything after it. Replacing the argmax with random sampling from the top candidates (Section 8.3) makes outputs more varied at the cost of determinism.
P(y1,…,yT)=∏t=1TP(yt ∣ y<t) P(y_1,\dots,y_T)=\prod_{t=1}^{T}P\big(y_t \,\big|\, y_{<t}\big)
Where
y1,…,yTy_1,\dots,y_T
a whole sequence of TT tokens
TT
the sequence length
P(yt∣y<t)P(y_t \mid y_{<t})
the probability of token tt given the tokens before it
∏t=1T\prod_{t=1}^{T}
multiply the TT step probabilities together
Read it aloudThe probability of the whole sequence y 1 to y T equals the product, over t from 1 to T, of the probability of y t given all the tokens before it.
Formula in plain wordsThis is the chain rule of probability: the chance of a whole sentence is the chance of its first token, times the chance of the second given the first, and so on. It is why next-token prediction is enough to model entire texts. Every factor is at most 1, so longer sequences get smaller probabilities, and one unlikely token drags the whole product down.
y₁ … yₜ₋₁ (previous tokens) context so far Generative LLM decoder-only transformer ŷₜ (next token) ŷₜ is appended to the context and the loop repeats (until [EOS])
The autoregressive loop: each generated token becomes part of the input for generating the next one.

Worked example 18Autoregressive generation of "UQ is …"

Prompt: UQ is. At each step the decoder outputs a distribution over the vocabulary and greedily picks the top token (illustrative probabilities):

StepInput sequenceTop candidates P(yt∣y<t) P(y_t\mid y_{<t}) Picked
1UQ isthe 0.62 · a 0.15 · ranked 0.08 · …the
2UQ is theUniversity 0.87 · best 0.05 · …University
3UQ is the Universityof 0.95 · in 0.02 · …of
4UQ is the University ofQueensland 0.98 · Qld 0.01 · …Queensland

The probability of the generated continuation is the product of the step probabilities (chain rule):

P(the, University, of, Queensland∣UQ is)=0.62×0.87×0.95×0.98≈0.502 P(\text{the, University, of, Queensland}\mid \text{UQ is}) = 0.62\times 0.87\times 0.95\times 0.98 \approx 0.502
ResultThe continuation "the University of Queensland" has probability about 0.50. Note the contrast with BERT: at step 2 the model sees only UQ is the and never any future token (masked attention). BERT would see the whole sentence at once but cannot generate it.

13.1 GPT-3 and ChatGPT

GPT-3 was released in June 2020 by OpenAI (slide 52). It is a decoder-only transformer (no encoder) with 175 billion parameters, roughly 1,000× larger than BERT-base, using unidirectional attention (only previous tokens are visible) and, in its original release, a 2,048-token context window. The slide's example: the prompt "Recite the first law of robotics" → GPT-3 generates, token by token, "A robot may not injure a human …". In 2022 ChatGPT was released with a very similar architecture; it sparked the current AI revolution and made generative models (LLMs) extremely popular worldwide.

13.2 Training: next-token prediction leads to in-context learning

GPT models are trained with simple next-token prediction (Section 10) on massive amounts of text. Remarkably, this simple objective produces in-context learning: the model learns to perform tasks just by seeing examples in the prompt (arithmetic, spelling correction, translation) with no task-specific training (Brown et al., 2020, "Language models are few-shot learners", slide 53).

Worked example 19In-context learning with a few-shot prompt

RoleText
Promptsea otter → loutre de mer
peppermint → menthe poivrée
cheese →
Model continuesfromage
ResultThe model outputs fromage. It was never fine-tuned for French translation; it recognises the pattern in the prompt and continues it, which is just next-token prediction applied to a well-chosen context.

13.3 Instruction fine-tuning

To make these models more useful they undergo instruction fine-tuning (slide 54): training on instruction–response pairs, for example "Summarize this text: [text]" → [summary]. This teaches the model to follow human instructions and perform tasks on request. It is what turns a raw language model into something like ChatGPT that can hold conversations and help with tasks.

PracticeAt step 1 in Worked example 18 the model had instead sampled "a" (probability 0.15). Can the final text still be "the University of Queensland"? What does this say about sampling?
AnswerNo. Once "a" is appended, every later step conditions on UQ is a, so the continuation changes (for example "a university in Brisbane"). Each token is fixed once generated, and one random choice early on changes all following probabilities; this is why sampled answers differ between runs.

14. LLMs in information retrieval

Sources: slides 55–56 · Video 7 (Autoregressive decoders) · Lecture transcript (ranking challenge)

Large language models are changing IR in three main directions (slide 55):

Prompting LLMs for retrieval and ranking LLM as rewriter, retriever, reranker or reader e.g. RankGPT, PromptReps Retrieval-augmented generation (RAG) retrieve documents, LLM answers with them (IR for LLMs) e.g. Self-RAG, BlendFilter Model-based IR (generative retrieval) the LLM is both the indexer and the retriever e.g. Differentiable Search Index (DSI)
Three directions from slide 55: LLMs inside the search pipeline, retrieval feeding LLMs, and LLMs replacing the index itself.
DirectionIdeaExamples
Prompting LLMs for retrieval and rankingUse the LLM directly as a rewriter, retriever, reranker or reader in the search pipeline (LLM-enhanced IR)RankGPT, PromptReps
Retrieval-augmented generation (RAG)Retrieve relevant documents, then use them as context so the LLM generates better, more reliable responses (IR for LLMs)Self-RAG, BlendFilter
Model-based IRThe LLM itself acts as both indexer and retriever: generative (document) retrievalDSI (Differentiable Search Index), GR

The in-class challenge unpacked several concrete ways to do LLM-based ranking (pointwise, pairwise and listwise prompting, query and document expansion, embeddings from decoders); all are worked through in Section 15.

14.1 The evolutionary trend

2017 2018 2020 2022 now Transformer BERT GPT-3 ChatGPT GPT-4, Claude, … "Attention isall you need" explosion ofencoder models rise ofdecoder models AI revolutionsparked new modelsconstantly
From the 2017 transformer paper, through encoder models (BERT, 2018), to decoder LLMs (GPT-3 2020, ChatGPT 2022) and today's rapidly changing model landscape (slide 56, Yang et al., 2023).
PracticeA company chatbot answers questions about internal policy documents and must cite them. Which of the three directions is this, and what does the IR component contribute?
AnswerRetrieval-augmented generation. A retriever (BM25, a dense retriever, or both) finds the relevant policy passages; the LLM then generates the answer using those passages as context. Retrieval supplies up-to-date, checkable evidence the LLM never saw in training, which makes the answers more reliable and citable.

15. In-class exercises, worked

Sources: Lecture transcript (in-class Exercises 1–3 and the challenge questions) · slides 20–23, 29, 49, 55

The lecture closed with three exercises plus a challenge. Attempt each before opening the answer. The lecturer's reason for setting them: "thinking about how you would use these components is exactly how the techniques later in this course were invented."

15.1 Exercise 1: transformer components

PracticeWhich statement about the transformer architecture is correct? A. The encoder uses masked attention to prevent looking at future tokens. B. The decoder uses bidirectional attention to see all tokens in the sequence. C. The encoder processes input tokens in parallel using bidirectional attention. D. Both encoder and decoder use the same type of attention mechanism.
Two of the wrong options swap the encoder and decoder.
AnswerC.
OptionVerdictWhy
AwrongBackwards: the encoder looks at future tokens (bidirectional); the decoder is the masked one. The encoder needs every token to see all tokens around it; for example the token loans can disambiguate an earlier bank.
BwrongSwapped again: bidirectional attention belongs to the encoder. The decoder only generates, and future tokens do not exist yet while generating.
CcorrectThe encoder sees the whole sequence at once (bidirectional) and processes all tokens in parallel: attention is batched matrix multiplication, exactly what GPUs accelerate. (CPUs compute in sequence; GPUs push everything through at once. This parallelism is why transformers made GPUs so successful, and the reverse.)
DwrongThey use different attention: encoder = bidirectional (all positions, earlier and later); decoder = masked or causal (earlier positions only).

15.2 Exercise 2: BERT vs GPT

PracticeWhat is the main architectural difference between BERT and GPT? A. BERT uses only the decoder part, while GPT uses only the encoder part. B. BERT uses only the encoder part, while GPT uses only the decoder part. C. BERT has more layers than GPT. D. BERT uses different tokenization techniques than GPT.
AnswerB. BERT is encoder-only; GPT is decoder-only. A is exactly backwards.

Why C is wrong (and interesting): in reality BERT has fewer layers than modern GPT models, but that is a size choice, separate from the architecture: you could build an encoder with more layers than a GPT. Generation models tend to be much bigger because generation is a harder task than representation. BERT-base's 110M parameters were "sort of already enough" for classification and ranking, whereas small generative models (GPT-2 era, T5 era) worked poorly; generative quality arrived only with very large parameter counts (the scaling law), hence the name large language models.

Why D is wrong (and interesting): both use the same technique, trained subword tokenisation (WordPiece-style, Section 4). The follow-up on the slide asks: "do BERT and GPT-2 have a different number of overall tokens?" Yes: they were trained on different collections with different target vocabulary sizes and goals (BERT about 30K pieces; some newer models hundreds of thousands), so the same text tokenises differently in each model. Same technique, different vocabulary.

15.3 Exercise 3: constructing input embeddings

PracticeThe sentence "The cat sits" is tokenised as [CLS] the cat sits [SEP], with embedding dimension 4. Token embeddings: [CLS] = [1,0,0,1], the = [0,1,1,0], cat = [1,1,0,0], sits = [0,0,1,1], [SEP] = [1,0,1,0]. Segment embeddings: A = [0,1,0,1], B = [0,0,0,2] (all special tokens belong to segment A). Position embeddings: pos₀ = [1,0,1,0], pos₁ = [0,1,0,1], pos₂ = [1,1,0,0], pos₃ = [0,0,1,1], pos₄ = [1,0,0,1]. Construct the complete input embedding for every token.
Decide each token's segment first, then use the formula of Section 5.4.
AnswerStep 1: assign the pieces. There is only one sentence (no second segment after a middle [SEP]), so every token, including [CLS] and [SEP], belongs to segment A and gets [0,1,0,1]. Segment B's embedding is never used. Positions count 0–4 left to right.

Step 2: add the three vectors for each token (element-wise addition, Section 2.1; one vector per token):

TokenToken E+ Segment E (A)+ Position E= Final input embedding
[CLS] (pos 0)[1,0,0,1][0,1,0,1][1,0,1,0][2,1,1,2]
the (pos 1)[0,1,1,0][0,1,0,1][0,1,0,1][0,3,1,2]
cat (pos 2)[1,1,0,0][0,1,0,1][1,1,0,0][2,3,0,1]
sits (pos 3)[0,0,1,1][0,1,0,1][0,0,1,1][0,1,2,3]
[SEP] (pos 4)[1,0,1,0][0,1,0,1][1,0,0,1][2,1,1,2]

Sample computation for [CLS]: [1+0+1,  0+1+0,  0+0+1,  1+1+0]=[2,1,1,2] [1{+}0{+}1,\; 0{+}1{+}0,\; 0{+}0{+}1,\; 1{+}1{+}0] = [2,1,1,2] .

Common traps (both came up in class): (1) putting tokens after [SEP] into segment B, which is wrong here because [SEP] merely ends the single sentence; (2) stacking the three embeddings into a 3×4 matrix, which is wrong because they are added into one 4-dimensional vector per token. The five final vectors, stacked as rows, form the 5×4 input matrix fed to the encoder. Real models use 768+ dimensions, and this week's practical computes them the same way. The explainer How [CLS] becomes [2, 1, 1, 2] animates the [CLS] row.

15.4 Challenge: encoders and decoders for ranking, encoders for generation

PracticeQ1: How can we use encoders or decoders for ranking? Name at least two ways.
Think about every place a vector or a probability appears in Sections 7 and 8.
AnswerThe full menu discussed in class.

With encoders

ApproachHow it worksConnects to
Single-vector dense retrieval (bi-encoder)Encode the query into one vector ([CLS] or mean of token embeddings); encode each document into one vector; score = cosine similarity or dot product. Documents are encoded offline at indexing time, so search is fast.Week 5
Multi-vector, late interaction (ColBERT-style)Keep one vector per token for query and document; compute every query-token × document-token similarity (a full similarity matrix); aggregate into a final score. More precise, more storage.Later weeks
Cross-encoder rerankingConcatenate query [SEP] document, feed it through BERT, read a relevance score off the [CLS] head. Most accurate, but one model run per (q, d) pair.Week 8

With decoders (LLMs)

ApproachHow it works
Prompted ranking (RankGPT-style)Just ask the LLM. Pointwise: "here is a query and a document; give me a relevance score." Pairwise: "document A or document B: which is more relevant?" Listwise: "here are 10 documents; give me an order." (The lecturer: "you may be using this every single day; why can't people think of the easy way?")
Query expansion (query2doc-style)Have the LLM generate a pseudo-document or answer for the query (like asking Gemini and using its answer), then use that richer text with a lexical ranker. The generated text contains car, cars, automobile, Tesla, BMW …, bridging the vocabulary mismatch of Section 1.
Document expansion (doc2query-style)The same trick flipped, and better for latency: at indexing time, expand each document with the queries or terms it could answer. It costs nothing at query time because it is precomputed while building the index (like BM25 indexing in the Week 3 practical).
Decoder hidden states as embeddings"Every time you see a vector, you can do computations." Cut the model before the output linear layer and use the last hidden vector as a dense representation for similarity search: decoders can be embedders too.
Output probabilities as representations (PromptReps-style)Use the next-token probability distribution over the whole vocabulary as a (sparse, term-statistics-like) representation of the document. A document about this course would put high probability on information, retrieval, web, search, ranking, University of Queensland …; rank by matching those distributions.
PracticeQ2 (challenge): Can we use an encoder backbone for generation?
What was the encoder trained to predict?
AnswerYes, and it is one of the newest ideas in the field: diffusion language models. The reasoning chain from the lecture:
  1. What was the encoder trained to do? Masked language modelling: predicting [MASK] tokens, which is itself a form of token prediction.
  2. So take the prompt, append a batch of [MASK] tokens (say 256), run the encoder once, and read off predictions for all masked positions at the same time, in place of one token at a time as in an autoregressive decoder.
  3. The catch: each masked token is predicted independently, so the output may be incoherent.
  4. The fix (borrowed from image diffusion models): iterative refinement. Like denoising an image step by step, re-run the prediction a few times (3–5 passes), each pass making the text more coherent.
  5. The payoff: 256 tokens in about 5 passes against 256 sequential passes; GPUs handle parallel tokens happily, so generation gets much faster.
The lecturer noted that his own recent paper uses diffusion language models for retrieval. The meta-lesson: these techniques are recombinations of this week's building blocks, well within reach. "Keep your imagination." For the backbone picture, see The encoder backbone, and why it cannot generate.
PracticeA student asked after class: document expansion or query expansion with an LLM, which is cheaper for users, and why?
AnswerDocument expansion. It runs the expensive LLM once per document at indexing time, before any user arrives, so query latency is unchanged. Query expansion runs the LLM while the user waits. Both enrich the lexical representation so that exact-match rankers such as BM25 can bridge vocabulary mismatch.

16. Summary and study checklist

16.1 BERT vs GPT at a glance

BERT (encoder-only)GPT-3 and LLMs (decoder-only)
ReleasedOctober 2018 (Google)June 2020 (OpenAI); ChatGPT 2022
ArchitectureTransformer encoder, 12 layers (base)Transformer decoder
Parameters110M (base) / 345M (large)175B (about 1,000× BERT-base)
AttentionBidirectional: sees the whole sentenceUnidirectional (masked): sees only the past
Pre-trainingMasked language modelling (about 15% masked) + next sentence predictionNext-token prediction
OutputOne contextual vector per token ([CLS] or mean pooling for the sentence)Probability distribution over the vocabulary (linear layer + softmax on the last hidden vector)
StrengthUnderstanding and representationGeneration and prediction; in-context learning
Best forSearch, classification, rankingText generation, translation, conversation
Key limits512-token input cap; cannot generate autoregressivelyNo access to future context; expensive at scale
IR usageDense retrieval (Week 5), reranking (Week 8)RankGPT-style prompting, query and document expansion, RAG, model-based IR

16.2 Key takeaways

  1. Term-based models (BM25) fail on vocabulary mismatch and polysemy: they match strings, and meaning escapes them. Modern retrieval needs semantic matching and context awareness.
  2. All the maths a transformer needs is in Section 2: vector addition (summing embeddings), dot products (similarity scores), vector × matrix multiplication (learned projections; (1×n)(n×m)=(1×m)(1{\times}n)(n{\times}m) = (1{\times}m), row · column per cell) and softmax (scores → probabilities).
  3. The transformer (2017) achieves semantic understanding through self-attention: project each token's vector into q (what I look for), k (what I offer) and v (my content); score qi⋅kjq_i \cdot k_j, scale by dk\sqrt{d_k}, softmax, then output zi=∑jaijvjz_i = \sum_j a_{ij} v_j. The attention matrix is directional (asymmetric).
  4. Real transformers are "fat" (several attention heads per layer) and "tall" (stacked layers, possible because input and output shapes match), with MLP and layer-norm steps after attention.
  5. The pipeline: WordPiece tokenisation (about 30K learned subword pieces; solves vocabulary size, OOV and morphology) → input embeddings (token + position + segment, summed element-wise) → encoder and/or decoder. Special tokens ([CLS], [SEP], [MASK], [PAD]; [BOS], [EOS]) act as instructions.
  6. Encoders (BERT) understand: bidirectional attention, parallel processing, one contextual vector per token; pre-trained with MLM + NSP; fine-tuned for tasks; used as dense retrievers and cross-encoder rerankers in IR.
  7. Decoders (GPT) generate: masked attention, then last hidden vector × vocabulary projection → softmax → next-token probabilities, looped until [EOS]; sampling explains why answers vary. Trained by next-token prediction; show in-context learning; used in IR through prompted ranking, query and document expansion, RAG and model-based IR.
  8. Both are trained self-supervised on web-scale text: the text labels itself (masked words, next words). Scale matters (scaling laws), and efficiency now matters too.
  9. Coming up: Week 5, BERT-based dense retrieval; Week 8, BERT-based rerankers. From here on, essentially every model in the course is a transformer.

16.3 Study checklist

17. Video index and resources

VideoTopicLengthUsed in sections
Video 1Overview of the transformer (BM25 limits, attention, the four steps)13:351, 3, 7, 8
Video 2Encoder vs decoder3:159
Video 3Input embeddings5:255
Video 4Tokenisation (WordPiece)2:584
Video 5Special tokens4:006
Video 6BERT7:1810, 11, 12
Video 7Autoregressive decoders4:458, 10, 13, 14

18. References