The Transformer Architecture
Term-based models such as BM25 match strings and miss meaning. This week builds the transformer from the ground up so that later weeks can use it for retrieval: the vector and matrix maths it needs, WordPiece tokenisation, token, position and segment embeddings, special tokens, self-attention with queries, keys and values (worked by hand), the decoder and its next-token output head, the split into encoder-only and decoder-only models, self-supervised training, BERT (masked language modelling, next sentence prediction, fine-tuning, dense retrieval and cross-encoder reranking) and autoregressive decoders such as GPT, and the main ways LLMs enter information retrieval. Dr Shuai Wang, UQ ielab.
1. Why BM25 is not enough
Sources: slides 2–6 · Video 1 (Overview of the transformer) · Lecture transcript (ranking history, term mismatch)
Weeks 1–3 built the classical retrieval stack: search engine architecture and text processing (Week 1), evaluation (Week 2), and term-based lexical models such as TF, TF-IDF, VSM, BIM and BM25 (Week 3). BM25 has been the gold standard for decades, and it has one fundamental blind spot: it can only do exact word matching. It has no model of what words or sentences mean.
1.1 Two classic failure modes
| Failure mode | Example | What BM25 does |
|---|---|---|
| Vocabulary mismatch (false negative) | Query car; document says automobile. The lecturer's second example: a query with biomedical against a document about medicine or biology, clearly related concepts written as different strings. | Zero overlapping terms, so a score of (near) zero. The relevant document is missed. |
| Polysemy (false positive) | Query where is the closest bank? (a financial institution); document about a river bank (a geographic feature). The in-class version: "I want to fix my laptop" against "I want to fix this guy": the same word means completely different things in different sentences. | bank appears in both, so the document may rank highly. An irrelevant document is retrieved. |
Worked example 1A relevant document BM25 cannot retrieve
Query: how to fix laptop. Document title: "Computer repair guide for damaged notebooks" (slide 5). A human sees at once that they match. Check the term overlap after stop-word removal:
| Query term | Matching document term? | Overlap |
|---|---|---|
fix | repair: a synonym written as a different string | ✗ |
laptop | computer, notebooks: related concepts, different strings | ✗ |
The BM25 score from Week 3 is a sum over the query terms that also occur in the document. Here that set is empty:
1.2 What a modern retrieval system needs
| Requirement | Meaning | Example |
|---|---|---|
| Semantic matching | Understand meaning as well as surface strings | car should match automobile; laptop and computer are related concepts |
| Context awareness | The same word can mean different things in different contexts; the system must understand query intent beyond surface words | Apple in a technology context is a company; apple in a recipe is a fruit |
1.3 Where we are in the history of ranking
The lecture opened with a "ranker from the history point of view" timeline. Week 4 sits at the "Industrial Revolution" (2017–2018): the arrival of the transformer. Everything after this point in the course builds on it.
| Era | Models | Where in this course |
|---|---|---|
| "Jurassic" (1950s–1990s) | Term-based lexical models (Boolean, TF-IDF, BM25) | Week 3 |
| "Human evolution" (2000s–2010s) | Feature-based reranking models (LTR, online LTR; fusion, RF) | Weeks 7, 11, 12 |
| "Stone Age transitioning" (2010s) | Word2vec, RNN, LSTM | Outside the course |
| "Industrial Revolution!" (2017–2018): we are here | Transformer. Encoder: BERT · Decoder: GPT | Week 4 |
| "New inventions" (2018–2022) | Encoder-based rerankers | Week 8 |
| (same period) | Dense retrievers; learnt sparse models | Weeks 5, 6 |
| "AI age" (2022 onwards) | ChatGPT, Mistral, Claude, Deepseek, Qwen | Parts of Weeks 5, 9, 10 |
| "The future is accelerating" | LLM rerankers and retrievers, LLM-assisted ranking, RAG, search agents | Parts of Weeks 5, 9, 10 |
PracticeA user searches jaguar top speed and BM25 returns a page about the Jaguar car brand's dealerships, while a wildlife page saying "the big cat can sprint at 80 km/h" is missed. Name both failure modes involved.
2. Maths refresher: vectors, matrices and softmax
Sources: Lecture transcript (linear-layer Q&A, Exercise 3 vector addition, self-attention walkthrough) · background for slides 10–19 and 25–29
Everything a transformer does is built from a handful of simple operations on vectors and matrices. Neural models read numbers only; as the lecturer put it, "we need to represent things in numbers, we need to do number calculations. That's how those neural models are formed." This section rebuilds those operations from scratch so the rest of the notes can use them freely. The companion workbook Transformer in a Spreadsheet (sheet "1 Maths") recomputes every example here with live formulas.
2.1 Vectors, vector addition and the dot product
A vector is an ordered list of numbers, for example . The number of entries is its dimension (here 4). In transformers every token is represented as a vector: BERT uses 768-dimensional vectors, Llama 2,048-dimensional ones, and so on.
- two vectors of the same dimension
- their dimension (number of entries)
- a position index, from 1 to
- entry of the sum vector
- the dot product of and : a single number
- entry of each vector
- the shared dimension
- add the products over all positions
Worked example 2Add three vectors, then take a dot product
[CLS] in Exercise 3, Section 15). The dot product is a single number.2.2 Matrices and the transpose
A matrix is a rectangular grid of numbers with rows and columns, written ("n by m", always rows first). A vector is a matrix with one row (a row vector, ) or one column (a column vector, ). For example is and is .
- an matrix
- its transpose, an matrix
- the entry of in row , column
- row and column indices of the transposed matrix
2.3 Matrix multiplication, step by step
- the left matrix,
- the right matrix,
- the product,
- the entry of in row , column
- the inner dimension: columns of , which must equal rows of
- the running index over the inner dimension
Worked example 3Every cell of A × B
Multiply the matrix by the matrix from Section 2.2. The inner dimensions match (3 = 3), so the answer is .
| Cell | Row of A · column of B | Value |
|---|---|---|
| row 1, col 1 | 1×7 + 2×9 + 3×11 = 7 + 18 + 33 | 58 |
| row 1, col 2 | 1×8 + 2×10 + 3×12 = 8 + 20 + 36 | 64 |
| row 2, col 1 | 4×7 + 5×9 + 6×11 = 28 + 45 + 66 | 139 |
| row 2, col 2 | 4×8 + 5×10 + 6×12 = 32 + 50 + 72 | 154 |
The special case transformers use constantly: vector × matrix
A row vector times a matrix gives a row vector . This is exactly what a neural network linear layer (also called a projection) is: it converts a vector of one size into a vector of another size using a matrix of learned weights. The lecturer derived this on paper during the Q&A: an -dimensional token vector times a learned projection matrix gives an -dimensional output vector. Any time a transformer "transforms" one vector into another, it does this.
Worked example 4A tiny linear layer
Project the 2-dimensional vector with the weight matrix . Each output entry is the vector dotted with one column.
2.4 Softmax: turning scores into probabilities
Transformers repeatedly need to turn a vector of raw scores (any real numbers, called logits) into probabilities: numbers between 0 and 1 that sum to 1. The softmax function does this.
- a vector of raw scores (logits),
- the score of item
- the exponential of that score (); always positive
- the sum of the exponentials of all scores
- the probability given to item
Worked example 5Softmax of three scores
Scores .
| Operation | Shapes | Where it appears this week |
|---|---|---|
| Vector + vector (element-wise) | same length in, same length out | Summing token, segment and position embeddings (Section 5) |
| Vector · vector (dot product) | two vectors in, one number out | Attention scores (Section 7); cosine similarity (Section 12) |
| Vector × matrix | Q, K, V projections (Section 7); the output linear layer (Section 8) | |
| Matrix × matrix | All attention scores at once, in parallel; this is why GPUs suit transformers | |
| Softmax | scores in, probabilities out (same length) | Attention weights (Section 7); next-token probabilities (Section 8) |
PracticeA token vector has dimension 768 and a learned matrix has shape 768 × 64. What is the shape of the product, and what happens if you try to multiply the vector by a 64 × 768 matrix instead?
PracticeCompute softmax of and of . What does this show?
3. The transformer: overview and the four steps
Sources: slides 7–19 · Video 1 (Overview of the transformer) · Lecture transcript (four steps, block contents)
In 2017 researchers at Google published "Attention Is All You Need" (Vaswani et al.), introducing the transformer architecture. The full architecture diagram looks intimidating. The lecturer's honest assessment: "it may look quite complicated, and in fact it was actually complicated … and this is already a simplified version". The key insight is simple to state.
3.1 Built for translation, reused for search
The original transformer was designed for machine translation: feed an English sentence such as "The cat sat on the mat" into an encoder–decoder pipeline, and the model outputs the same sentence in another language (the slide uses Chinese, 猫坐在垫子上).
3.2 The four processing steps
Slides 13–19 build the architecture up in four steps. This is the map for Sections 4 to 8.
⟨start⟩ token and the encoded representations. In the video's words, the decoder says "I'm going to generate; encoder, give me the context."⟨end⟩ token (Section 8). This step-by-step generation is how the decoder writes coherent text.The translation of "The cat sat on the mat" therefore runs as a loop. At each iteration the decoder outputs a probability distribution over the vocabulary and picks the highest-probability token:
| Iteration | Decoder input | Predicted token |
|---|---|---|
| 1 | ⟨start⟩ + encoded vectors | 猫 |
| 2 | ⟨start⟩ 猫 + encoded vectors | 坐在 |
| 3 | ⟨start⟩ 猫 坐在 + encoded vectors | 垫子 |
| 4 | ⟨start⟩ 猫 坐在 垫子 + encoded vectors | 上 |
| 5 | ⟨start⟩ 猫 坐在 垫子 上 + encoded vectors | ⟨end⟩: generation stops |
3.3 What is inside each block
The hand-drawn architecture shown in the lecture reveals what each encoder or decoder block contains, bottom to top: multi-head self-attention → layer norm → multilayer perceptron (MLP) → layer norm, with arrows that skip around each part (residual connections). Section 7.4 draws it. Two facts to hold on to:
- The dashed loop means "repeat". A block's output is a set of vectors with exactly the same shape as its input, so it can be fed straight back in. "We can repeat this 12 times": stacking blocks is what gives BERT its 12 layers.
- The decoder ends differently. Its final block feeds a linear layer + softmax that outputs probabilities over the vocabulary in place of vectors; that is what makes generation possible (Section 8).
PracticeIn the translation loop, which of the four steps run once per sentence and which run once per generated token?
4. Step 1a: tokenisation and WordPiece
Sources: slides 30–33 · Video 4 (Tokenisation) · Lecture transcript (WordPiece, token IDs)
Before any embedding happens, text must be broken into tokens: the units the model actually reads. Each token corresponds to an integer ID in a fixed vocabulary (the lecturer's example: the token my might be ID 3500), and these IDs are what get mapped to vectors. Week 1 tokenised text for an inverted index; here the tokeniser serves a neural model with a fixed-size lookup table, which changes the design goals.
The obvious approach splits on spaces: "my dog is cute" → [my, dog, is, cute]. Slide 31 lists three serious problems with simple word splitting:
- Huge vocabulary. English has about 500,000 unique words, which would need a massive embedding lookup table; once names, typos and new words are included, the vocabulary space is "almost infinite".
- Out-of-vocabulary (OOV) words. New words such as coronavirus (new in 2020), typos, or garbage strings like
ecerfeferf: the model has never seen them, so it has no embedding for them. - Lost morphological relationships. run, running and runner become completely unrelated tokens even though they are clearly related words.
The opposite extreme uses single characters as tokens (just a–z, 26 tokens). It can represent anything, but characters alone do not capture how people combine letters into meaningful units, and sequences become very long. The sweet spot lies in between.
##.| Tokeniser | Tokens for "[CLS] My dog is cute [SEP] he likes playing [SEP]" |
|---|---|
| Word splitting | [CLS], my, dog, is, cute, [SEP], he, likes, playing, [SEP] |
| WordPiece | [CLS], my, dog, is, cute, [SEP], he, likes, play, ##ing, [SEP] |
ing appears extremely often in English. Keeping every -ing word as its own token would blow up the vocabulary, so the tokeniser keeps play and ##ing as separate reusable pieces.4.1 How WordPiece training works
[a, b, c, …, z].th, ing, tion, …Worked example 6Subword merges on a toy corpus
Illustrative training data with four word types and their counts: low ×5, lower ×2, newest ×6, widest ×3.
l o w, l o w e r, n e w e s t, w i d e s t.| Pair | Where it occurs | Count |
|---|---|---|
e s | newest (6) + widest (3) | 9 (most frequent) |
s t | newest (6) + widest (3) | 9 (tied) |
l o | low (5) + lower (2) | 7 |
o w | low (5) + lower (2) | 7 |
e s → es. The words become n e w es t and w i d es t.es t occurs 9 times → merge into est.l o (7) → lo; then lo w (7) → low; and so on, growing the vocabulary one useful piece at a time: es, est, lo, low, …low + ##est: both pieces are in the vocabulary, the link to low and widest is preserved, and there is no OOV failure. This illustrates frequency-based merging. WordPiece itself scores a candidate merge by how much it improves the likelihood of the training data, , which favours pairs that occur together more often than their parts alone would predict; the mechanics are the same.Worked example 7Tokenising the problem words
| Input word | WordPiece output (illustrative) | Why it works |
|---|---|---|
playing | play + ##ing | Morphology preserved: shares play with played and player |
coronavirus | corona + ##virus | New in 2020, yet both pieces were already in the vocabulary, so no OOV |
ecerfeferf | e + ##cer + ##fe + ##fer + ##f | Even garbage and typos decompose into known pieces (worst case: single characters) |
PracticeWhy does a WordPiece model never hit an out-of-vocabulary error, while a word-level model does?
5. Step 1b: transformer input embeddings
Sources: slides 24–30 · Video 3 (Input embeddings) · Lecture transcript (three embeddings, "I asked him")
Once the text is tokenised, each token must become a vector. A token ID says what the token is and leaves out where it appears, yet word order changes meaning entirely. The lecturer's example: "I asked him" and "he asked me" use almost the same words in different positions, with completely different meanings. Transformers therefore combine three types of embeddings (slide 25):
| Embedding | Purpose | Required? |
|---|---|---|
| Token embedding | Converts tokens into vectors; represents word meaning through a learned lookup table | Always |
| Position embedding | Teaches the model word order: each position (0, 1, 2, …) gets its own learned vector | Always |
| Segment embedding | Distinguishes different parts of the input (for example query and document); each segment gets its own learned vector | Optional: only some models, such as BERT |
5.1 Token embeddings
The text is tokenised, then each token is looked up in a learned embedding table: one learned vector per vocabulary entry, capturing what the token means on its own. The idea comes from earlier word-vector models such as word2vec. Slide 26's example (vectors truncated):
| Stage | Value |
|---|---|
| Text | "Information Retrieval" |
| Tokenise | ["information", "retrieval"] |
| Lookup | information → [0.1, 0.3, −0.2, 0.9, …]retrieval → [0.4, −0.1, 0.6, 0.2, …] |
5.2 Position embeddings
Word order matters: "information retrieval" ≠ "retrieval information". Each position in the sequence gets its own learned embedding vector (slide 27): Pos1 → [0.9, 0.4, 0.2, 0.8, …], Pos2 → [0.2, −0.4, 0.4, 0.2, …].
5.3 Segment embeddings (optional)
Models such as BERT use segment embeddings when the input holds several sequences, for example a query and a document processed together (the BERT cross-encoder of Section 12). Special tokens separate the segments ([CLS] at the start, [SEP] between and at the end), and every token in a segment receives that segment's embedding (slide 28):
| Stage | Value |
|---|---|
| Text | Information (query) + Retrieval (document) |
| Tokenised | [CLS] information [SEP] retrieval [SEP] |
| Segments | query segment → [0.6, 0.3, 0.7, −0.9, …]doc segment → [0.5, 0.1, 0.8, 0.2, …] |
5.4 Putting it together
- the position of the token in the sequence (0, 1, 2, …)
- the token at position
- the segment that token belongs to (A or B)
- the learned vector for that token, from the token lookup table
- the learned vector for position
- the learned vector for segment (omitted in models without segments)
- the final input vector for position ; same dimension as each part (768 in BERT-base)
[CLS] my dog is cute [SEP] he likes play ##ing [SEP]. Tokens up to the first [SEP] get segment A; tokens after it get segment B. Positions count 0, 1, 2, … so the model knows where each token sits.For a visual, column-by-column version of this addition, see the companion explainer How [CLS] becomes [2, 1, 1, 2], and sheet "2 Embeddings" of the Transformer in a Spreadsheet workbook.
Worked example 8One token's final input embedding
Take the token likes from the diagram: token likes, at position 7, in segment B. Illustrative 4-dimensional embeddings:
| Component | Vector |
|---|---|
[0.2, −0.1, 0.5, 0.3] | |
[0.0, 0.3, −0.2, 0.1] | |
[0.1, 0.1, 0.0, −0.2] |
Sum element-wise:
PracticeIn "the bank by the river bank", the token bank appears at positions 1 and 5 (one segment). Do the two occurrences get the same input vector?
6. Special tokens
Sources: slides 34–39 · Video 5 (Special tokens) · Lecture transcript (CLS for retrieval, PAD for batching, "[FRANKENSTEIN]")
6.1 Encoder special tokens: [CLS], [SEP], [MASK], [PAD]
| Token | Name | Role |
|---|---|---|
[CLS] | Classification | Always at the beginning of the input; its output vector is used for sentence-level tasks (classification, ranking). The lecturer's note: it is called the classification token, but if you fine-tune the model to project this token to any task, it may work. Retriever models use it heavily to represent a whole sentence as one vector. |
[SEP] | Separator | Divides different sequences; helps the model distinguish the parts of the input (for example query and document) |
[MASK] | Masked token | Used during training: the model learns to predict the masked words (Sections 10 and 11) |
[PAD] | Padding | Pads sequences to the same length so they can be batched: a batch must have one length for the GPU to process it in parallel. Ignored during attention computation. |
Worked example 9A search-task input, token by token
Input: [PAD] [CLS] laptop repair [SEP] Computer maintenance guide for damaged notebooks [SEP]
| Piece | What it does here |
|---|---|
[PAD] | Keeps the length uniform when this input is batched with longer inputs (ignored by attention) |
[CLS] | Signals a classification or ranking task; its final vector will carry the relevance decision |
laptop repair | Segment A: the query |
[SEP] | Separates the query from the document |
Computer maintenance guide … | Segment B: the document |
[SEP] | Marks the end of the input sequence |
[CLS] vector.6.2 Decoder special tokens: [BOS], [EOS]
| Token | Name | Role |
|---|---|---|
[BOS] | Beginning of sequence | Signals the start of generation; the first token fed to the decoder during inference |
[EOS] | End of sequence | Signals the end of generation and tells the decoder when to stop; critical for controlling output length. How does ChatGPT know it has said enough? It generates its EOS token. |
Worked example 10BOS and EOS in a generation task
| Part | Tokens |
|---|---|
| Prompt | <BOS> The cat sits on |
| Generated | the mat <EOS> (the model stops here) |
<BOS> signals the start of the prompt; when the model itself generates <EOS>, generation is finished. This simple mechanism lets decoder models produce output of an appropriate length and stop cleanly.[FRANKENSTEIN], and as long as you train the model with it, the model can learn what it means.PracticeA batch holds a 6-token input and a 10-token input. What does [PAD] do here, and why must attention ignore it?
7. Step 2: self-attention and the encoder
Sources: slides 10–15, 19 · Video 1 (Overview of the transformer) · Lecture transcript (self-attention walkthrough, symmetry question, "fat and tall", Exercise 1)
7.1 The intuition: the attention matrix
For the sentence "The cat sat on the mat", self-attention produces a matrix in which each cell shows how much one word (row) pays attention to another word (column). For example, cat pays high attention to sat because they are grammatically related (subject and verb).
Where do these numbers come from? The in-class lecture answered with a five-step breakdown ("Self-attention: what is it really doing?"). If you know little about neural networks, read the five steps slowly: every operation involved is one you met in Section 2. Sheet "3 Attention" of the Transformer in a Spreadsheet workbook runs all five steps with live formulas, and the Transformer by Hand worksheet lets you compute a block yourself.
7.2 The five steps of self-attention
Step 1: one input vector per token
Each token has one input vector : the sum of its token, segment and position embeddings from Section 5. For the toy sentence the · cat · sat: .
Step 2: project each input vector into q, k and v
The model holds three learned weight matrices , , (learned during training, like everything else). Each token's input vector is multiplied by each matrix, the vector × matrix operation of Section 2.3:
- the input vector of token (a row vector)
- learned projection matrices, each ( may use a different output size )
- the query vector of token : what it is looking for
- the key vector of token : what it offers to be matched against
- the value vector of token : the content it contributes
- the input dimension and the query/key dimension
Step 3: what are Q, K and V? (the lecturer's intuitions)
| Vector | Slide definition | Lecturer's phrasing |
|---|---|---|
| Q: query | what this token is looking for | "What am I to myself; what am I really looking for?" |
| K: key | what this token offers, what can be matched | "What am I to others?" |
| V: value | the information or content carried by this token | "In the end, how do I represent myself?" |
Step 4: compute attention weights
To find how much token should attend to token , take the dot product of with : "how well does what I am looking for match what you offer?" Scale it, then apply softmax across each row so every token's weights sum to 1.
- dot product of token 's query with token 's key (the raw score)
- the dimension of the query and key vectors
- the scaled score of token attending to token
- the number of tokens in the sequence
- index running over all tokens in the softmax denominator
- the attention weight: the share of token 's attention given to token
- all queries and all keys stacked as rows ()
- the attention weight matrix; row holds
Step 5: build the new vector for each token
Each token's new, contextualised representation is the attention-weighted sum of all tokens' value vectors, including its own (attention to self is part of the sum).
- the new output vector of token
- token 's attention weight on token (from Step 4)
- token 's value vector
- index over all tokens, including
Putting the five steps together in matrix form gives the famous formula from Attention Is All You Need:
- the stacked query, key and value matrices ( rows each)
- the transpose of , so that is
- the scaling factor from the key dimension
- applied to each row separately
- the -row output matrix whose row is
7.3 Self-attention by hand
Worked example 11Self-attention for the · cat · sat, every number computed
Illustrative toy sentence of three tokens with 2-dimensional input vectors (): . The projection matrices are picked by hand (a real model learns them):
| Token | |||
|---|---|---|---|
the | [1, 0] | [1, 1] | [1, 1] |
cat | [2, 2] | [0, 4] | [0, 2] |
sat | [2, 1] | [1, 3] | [1, 2] |
| k(the) = [1,1] | k(cat) = [0,4] | k(sat) = [1,3] | |
|---|---|---|---|
| q(the) = [1,0] | 1 | 0 | 1 |
| q(cat) = [2,2] | 4 | 8 | 8 |
| q(sat) = [2,1] | 3 | 4 | 5 |
7.4 What makes a real transformer "fat" and "tall"
The five steps describe one attention head in one layer. Real transformers scale this up in two directions, in the lecturer's memorable phrasing:
| Direction | What it means | Effect |
|---|---|---|
| "Fat": multi-head attention | Within a layer, the model keeps several independent copies of ; each copy is an attention head with its own learned weights. Each head computes its own attention pattern in parallel (one might track subject–verb links, another adjacent words), and their outputs are combined. | Makes each layer wider |
| "Tall": stacked layers | Each token goes in as a vector and comes out as a vector of the same dimension, so the whole block can be applied again: output of layer 1 → input of layer 2 → … BERT-base stacks 12 layers, each refining the representations further. | Makes the model deeper |
Two supporting components follow the attention step in every block:
- Multilayer perceptron (MLP, feed-forward layer): a plain neural network applied to each token vector. It projects the vector up to a higher dimension and back down again (two vector × matrix multiplications with a non-linearity between). Much of the model's "thinking capacity" lives here.
- Layer norm and residual connections: housekeeping that keeps the numbers stable during training and lets each layer add to what previous layers computed, so earlier information is preserved.
For how a stack of these layers refines one vector per token, see the companion explainer The encoder backbone, and why it cannot generate. Sheet "4 Multi-Head" of the spreadsheet workbook shows two heads side by side.
7.5 What the encoder produces, and why GPUs love it
After the final layer the encoder outputs one encoded vector per input token: the same count and dimension as the input, where each vector now represents "the semantics of the word with respect to the sentence". For a single vector for the whole sentence there are two standard choices: take the [CLS] token's output vector (train the model to summarise the sentence there), or take the mean of all token vectors (mean pooling). Both are used heavily in retrieval (Section 12 and Week 5).
- the encoder's final output vector for token
- the number of tokens pooled (padding tokens are left out)
- the final output vector of the [CLS] token
- one vector for the whole text, same dimension as each
Worked example 12A sentence vector by mean pooling
Use the three output vectors from Worked example 11: , , .
PracticeIn Worked example 11, compute the raw score of sat attending to the and of the attending to sat. Are they equal?
PracticeWhy can BERT-base repeat the same block structure 12 times, and what would break if the MLP output a vector of a different size?
8. Steps 3–4: the decoder and generation
Sources: slides 16–19, 23 · Video 1 (Overview of the transformer) · Video 7 (Autoregressive decoders) · Lecture transcript (decoder, linear-layer Q&A, sampling)
"Don't worry too much about the decoder; it's almost the same." The decoder is built from the same pieces as the encoder: tokenise the input (think of it as your prompt), build input embeddings, run stacked blocks of self-attention + MLP. Two things differ, and both exist because the decoder's job is generation.
8.1 Difference 1: masked (causal) attention
In the decoder each token may attend only to itself and earlier tokens. During generation the future tokens do not exist yet: when the model generates the first word of its answer, there is nothing after it to look at. The lecturer: "It's not that we don't want it; we don't need it. How could it look at future tokens? They're not yet generated." In practice the scores for future positions are set to before the softmax, which blanks out the upper-right triangle of the attention matrix (compare the two grids in Section 9).
- the position of the token doing the attending
- the position being attended to
- “itself or earlier”: position is position or comes before it
- the ordinary scaled score from Section 7
- a score that becomes after exponentiation
- the masked score
- the resulting attention weight (0 for every future position)
Worked example 13The causal mask on the · cat · sat
Reuse the scaled scores from Worked example 11 and mask every future position.
8.2 Difference 2: the output head, from vector to next-token probabilities
The encoder ends with vectors; the decoder must end with a choice of next token. The lecturer wrote the full chain out on paper for a student, and it is one matrix multiplication plus one softmax:
- the last token's final-layer hidden vector (dimension )
- the learned output projection ("linear layer"),
- the vocabulary size (about 30,000 for BERT-style vocabularies)
- the logits: one raw score per vocabulary token
- the next-token probability distribution
- the probability of vocabulary token
Worked example 14The output head with numbers
Illustrative vocabulary of 5 tokens (the, a, cat, pie, …) with logits .
the (0.56); under greedy decoding it becomes the next token. Softmax handles the negative logit of pie without trouble: it just becomes a small probability (0.03).8.3 The generation loop and sampling
One pass through the model produces one token. To generate text, the predicted token is appended to the input and the whole process runs again ("generate, put it back, generate, put it back") until the model produces the [EOS] token and stops. Section 13 formalises this as autoregressive generation.
PracticeA model has hidden size 2,048 and a vocabulary of 32,000 tokens. What shape is the output projection, how many numbers does it produce per step, and why do we take only the last token's vector?
9. Encoder vs decoder
Sources: slides 20–23 · Video 2 (Encoder vs decoder) · Lecture transcript (using the parts separately, Exercise 1)
In the original transformer the encoder and decoder work as one integrated system: the encoder converts text into vector representations; the decoder takes those vectors plus the ⟨start⟩ token and predicts the next token. The crucial insight of slides 20–21 is that they can be used separately. The lecturer framed it as two questions: "If my task is not generation, do I need the decoder? If my task is only generation, do I need the encoder?" Answering "no" to each led to some of the most important breakthroughs in AI:
- BERT uses only the encoder: an encoder-only model, hugely influential in search and language understanding (Section 11).
- Modern large language models (GPT, ChatGPT, Claude, …) use only the decoder (Section 13).
| Encoder: "understand and represent" | Decoder: "generate and predict" | |
|---|---|---|
| Purpose | Takes input text; creates a rich, dense vector representation; understands context and relationships | Takes input (optionally with encoded representations) and generates output step by step, predicting next-token probabilities |
| Attention | Bidirectional: every token can look at all tokens before and after it; the encoder sees the entire sentence at once | Masked (unidirectional): a token sees only previous words |
| Example | Input How to fix laptop? → output dense vector [0.23, −0.45, 0.67, …] capturing the sentence's meaning | Input UQ is (+ encoded vectors, optional) → output the → University → of → Queensland → … |
| Applications | Search and classification using embeddings: tasks that need the full meaning of text | Translation, text generation, and all modern LLM usage |
PracticeYou need to embed ten million product descriptions for semantic search, and separately to write product summaries. Which transformer part suits each task, and which attention pattern does each use?
10. How transformers are trained
Sources: slides 41, 44, 53 · Video 6 (BERT) · Video 7 (Autoregressive decoders) · Lecture transcript (training, data and lawsuits)
A randomly initialised model knows nothing: the embedding tables, the matrices in every head of every layer, the MLP weights and the output projection all start as meaningless random numbers. Training adjusts all those weights until the model "understands" language. The practical problem: this needs colossal amounts of training signal without human labelling. The solution used by both BERT and GPT is self-supervised learning: take the whole web as training data and let the text itself provide the labels.
| Objective | How it works | Lecture illustration |
|---|---|---|
| Masked language modelling (encoder, BERT-style) | Take web text, randomly replace a fraction of its tokens with [MASK], and train the model to predict the masked tokens from all the surrounding context (both directions, which is why bidirectional attention matters). Compare prediction and real token, measure the error (loss), nudge the weights to reduce it, and repeat over the whole web. Details and a loss calculation in Section 11. | "The bank [MASK] loans to clients" → predict makes |
| Next-token prediction (decoder, GPT-style) | Even simpler. Feed the first token: can you predict the second? Feed the first two: predict the third. Feed the first three: predict the fourth. Every position of every sentence on the web becomes a training example. | Train on your chat history and the model learns what you would reply next ("if you've written bullshit a thousand times, it knows the next token is bullshit"). The model is never taught grammar; it learns how people write by predicting them, billions of times. |
Worked example 15One sentence, two training signals
Take the 6-token sentence UQ is the University of Queensland.
UQ → is, UQ is → the, UQ is the → University, … University → of, … of → Queensland. That is 5 labelled examples from one sentence.UQ is the [MASK] of Queensland, and the model predicts University from both sides.PracticeWhy would next-token prediction be a poor pre-training objective for an encoder with bidirectional attention?
11. BERT
Sources: slides 40–48, 50 · Video 6 (BERT) · Lecture transcript (MLM masking rate, scaling)
11.1 The two-stage training approach
11.2 Ingredients and input processing
BERT's success comes from combining three pre-training ingredients (slide 42): (1) an encoder-only transformer architecture with lots of parameters, (2) massive amounts of text data (Wikipedia, web pages), and (3) enormous computational power (cloud TPU pods were essential at the time).
Input starts with the familiar embedding layers, token + segment + position embeddings added together (Section 5), then flows through multiple transformer encoder layers with self-attention (Section 7). The output is a sequence of rich vector representations, one per input token, where every token has attended to every other token.
11.3 Pre-training task 1: masked language modelling (MLM)
Randomly mask about 15% of the input tokens and train the model to predict them from all the surrounding (bidirectional) context. The BERT paper and slide 50 use 15%; in class the lecturer quoted a loosely remembered 30%. The exact fraction is a design choice; the principle is what matters. The loss measures how well BERT predicts the masked tokens:
- the set of masked positions in the sentence
- the true (original) token at masked position
- the sentence with the masked positions hidden: all the visible context
- the probability BERT assigns to the correct token at position , given that context
- natural logarithm
- the masked language modelling loss for this sentence (lower is better)
Worked example 16MLM loss on the lecture's sentence
Original (slide 47): "The Mongol invasion of Europe in the 13th century was the conquest of much of Europe by the Mongol Empire." BERT masks century and Europe: "… in the 13th [MASK] was the conquest of much of [MASK] by the Mongol Empire." Using bidirectional context, it predicts:
| Masked position | Prediction | Confidence | Loss contribution |
|---|---|---|---|
… 13th [MASK] was … | century | 0.94 | |
… much of [MASK] by … | Europe | 0.97 |
11.4 Pre-training task 2: next sentence prediction (NSP)
BERT also trains on sentence pairs, learning to predict whether the second sentence naturally follows the first (a binary classification on the [CLS] output). This teaches relationships between sentences, which matter for question answering and document ranking.
| Sentence A | Sentence B | Label |
|---|---|---|
| My dog is cute. | He likes playing. | IsNext |
| My dog is cute. | The stock market fell 2% today. | NotNext |
11.5 Model size matters
| Model | Parameters | Configuration |
|---|---|---|
| BERT-base | 110 million | 12 layers |
| BERT-large | 345 million | more layers, larger hidden dimension, more attention heads |
BERT-large consistently outperforms BERT-base, including on ranking and on small-scale tasks, and more training steps give higher accuracy (slide 46). This was an early hint of the scaling laws seen throughout today's large language models. The lecturer noted that billion-parameter models now count as small and frontier models run to trillions, though the industry has since had to balance raw scale against efficiency ("a user says hello, and answering hi costs me $20; I can't do that").
11.6 Fine-tuning to different tasks
- Single-input classification: feed one text; the
[CLS]output representation serves as the sentence summary for the classifier. - Two-input tasks (question answering, ranking): process both inputs together in different segments (query
[SEP]document). - Sequence labelling: each token gets its own prediction from its own output vector.
11.7 BERT's limitations
| Limitation | Detail (slide 50) |
|---|---|
| Input length constraint | Input is capped at 512 tokens, including special tokens. Long documents cannot be processed directly; they need chunking or truncation, which loses information. |
| Cannot generate text autoregressively | MLM pre-training randomly masks about 15% of tokens and predicts them with bidirectional context. BERT is never trained on next-token prediction, so it cannot produce coherent sequences token by token like GPT. (Section 15 covers a modern twist: diffusion language models generate with encoder backbones.) |
The second limitation is exactly where decoder-only models such as GPT come in (Section 13).
PracticeDuring training BERT gives the correct masked word probability 0.5 at one position and 0.05 at another. Compute each loss contribution and say which position drives learning more.
12. BERT in information retrieval
Sources: slide 49 · Video 6 (BERT) · Lecture transcript (ranking challenge: [CLS] or mean vectors, ColBERT-style interaction)
For IR, BERT is used in two main ways (slide 49). Both significantly improve search quality: dense retrievers are the topic of Week 5 and BERT-based rerankers of Week 8.
| Dense retriever (bi-encoder) | Reranker (cross-encoder) | |
|---|---|---|
| Input | Query and document encoded separately | [CLS] query [SEP] document [SEP], read together (segments A and B, Section 5) |
| Score | Cosine similarity or dot product of and | Classification score from the [CLS] head |
| Cost | Document vectors computed once at indexing time; fast search | One BERT run per (query, document) pair; used to rerank a short candidate list |
| Accuracy | Good | Higher: every query token attends to every document token |
| Course | Week 5 | Week 8 (this is the monoBERT style of reranker) |
From the in-class discussion: for the dense-retriever route, the sentence vector usually comes from either the [CLS] output vector or the mean of all token output vectors (Section 7.5). The relevance score is then a cosine similarity or dot product: "either way, you can have a score, as long as you train it." A student proposed the more advanced version: keep one vector per token for both query and document, compare every query token with every document token (a full similarity matrix, like attention), and aggregate the per-token similarities into a final score. That is the intuition behind ColBERT-style late interaction: more accurate than a single vector, at the cost of storing many vectors per document (later weeks).
- the query vector and document vector from BERT
- their dot product (Section 2.1)
- entry of each vector
- vector lengths (Euclidean norms)
- cosine similarity, between −1 and 1
Worked example 17Dense retrieval fixes the fix-laptop query
Encode the query and two documents into illustrative 3-dimensional BERT vectors:
| Text | Vector |
|---|---|
| = V("how to fix laptop") | [0.90, 0.30, 0.10] |
| = V("computer repair guide for damaged notebooks") | [0.80, 0.40, 0.00] |
| = V("river bank fishing spots") | [0.10, 0.20, 0.95] |
PracticeA collection has 8 million passages and you must answer queries in under 100 ms. Where does each BERT usage fit in the pipeline, and why?
13. Autoregressive decoders and GPT
Sources: slides 51–54 · Video 7 (Autoregressive decoders)
- the current time step (position being generated)
- all tokens before step : the prompt plus everything generated so far
- the vocabulary
- one candidate token from the vocabulary; the arg max tries every such token
- the model's probability that comes next (the softmax output of Section 8)
- "the that gives the largest value"
- the token chosen at step (hat = predicted)
- a whole sequence of tokens
- the sequence length
- the probability of token given the tokens before it
- multiply the step probabilities together
Worked example 18Autoregressive generation of "UQ is …"
Prompt: UQ is. At each step the decoder outputs a distribution over the vocabulary and greedily picks the top token (illustrative probabilities):
| Step | Input sequence | Top candidates | Picked |
|---|---|---|---|
| 1 | UQ is | the 0.62 · a 0.15 · ranked 0.08 · … | the |
| 2 | UQ is the | University 0.87 · best 0.05 · … | University |
| 3 | UQ is the University | of 0.95 · in 0.02 · … | of |
| 4 | UQ is the University of | Queensland 0.98 · Qld 0.01 · … | Queensland |
The probability of the generated continuation is the product of the step probabilities (chain rule):
UQ is the and never any future token (masked attention). BERT would see the whole sentence at once but cannot generate it.13.1 GPT-3 and ChatGPT
GPT-3 was released in June 2020 by OpenAI (slide 52). It is a decoder-only transformer (no encoder) with 175 billion parameters, roughly 1,000× larger than BERT-base, using unidirectional attention (only previous tokens are visible) and, in its original release, a 2,048-token context window. The slide's example: the prompt "Recite the first law of robotics" → GPT-3 generates, token by token, "A robot may not injure a human …". In 2022 ChatGPT was released with a very similar architecture; it sparked the current AI revolution and made generative models (LLMs) extremely popular worldwide.
13.2 Training: next-token prediction leads to in-context learning
GPT models are trained with simple next-token prediction (Section 10) on massive amounts of text. Remarkably, this simple objective produces in-context learning: the model learns to perform tasks just by seeing examples in the prompt (arithmetic, spelling correction, translation) with no task-specific training (Brown et al., 2020, "Language models are few-shot learners", slide 53).
Worked example 19In-context learning with a few-shot prompt
| Role | Text |
|---|---|
| Prompt | sea otter → loutre de merpeppermint → menthe poivréecheese → |
| Model continues | fromage |
fromage. It was never fine-tuned for French translation; it recognises the pattern in the prompt and continues it, which is just next-token prediction applied to a well-chosen context.13.3 Instruction fine-tuning
To make these models more useful they undergo instruction fine-tuning (slide 54): training on instruction–response pairs, for example "Summarize this text: [text]" → [summary]. This teaches the model to follow human instructions and perform tasks on request. It is what turns a raw language model into something like ChatGPT that can hold conversations and help with tasks.
PracticeAt step 1 in Worked example 18 the model had instead sampled "a" (probability 0.15). Can the final text still be "the University of Queensland"? What does this say about sampling?
UQ is a, so the continuation changes (for example "a university in Brisbane"). Each token is fixed once generated, and one random choice early on changes all following probabilities; this is why sampled answers differ between runs.14. LLMs in information retrieval
Sources: slides 55–56 · Video 7 (Autoregressive decoders) · Lecture transcript (ranking challenge)
Large language models are changing IR in three main directions (slide 55):
| Direction | Idea | Examples |
|---|---|---|
| Prompting LLMs for retrieval and ranking | Use the LLM directly as a rewriter, retriever, reranker or reader in the search pipeline (LLM-enhanced IR) | RankGPT, PromptReps |
| Retrieval-augmented generation (RAG) | Retrieve relevant documents, then use them as context so the LLM generates better, more reliable responses (IR for LLMs) | Self-RAG, BlendFilter |
| Model-based IR | The LLM itself acts as both indexer and retriever: generative (document) retrieval | DSI (Differentiable Search Index), GR |
The in-class challenge unpacked several concrete ways to do LLM-based ranking (pointwise, pairwise and listwise prompting, query and document expansion, embeddings from decoders); all are worked through in Section 15.
14.1 The evolutionary trend
PracticeA company chatbot answers questions about internal policy documents and must cite them. Which of the three directions is this, and what does the IR component contribute?
15. In-class exercises, worked
Sources: Lecture transcript (in-class Exercises 1–3 and the challenge questions) · slides 20–23, 29, 49, 55
The lecture closed with three exercises plus a challenge. Attempt each before opening the answer. The lecturer's reason for setting them: "thinking about how you would use these components is exactly how the techniques later in this course were invented."
15.1 Exercise 1: transformer components
PracticeWhich statement about the transformer architecture is correct? A. The encoder uses masked attention to prevent looking at future tokens. B. The decoder uses bidirectional attention to see all tokens in the sequence. C. The encoder processes input tokens in parallel using bidirectional attention. D. Both encoder and decoder use the same type of attention mechanism.
| Option | Verdict | Why |
|---|---|---|
| A | wrong | Backwards: the encoder looks at future tokens (bidirectional); the decoder is the masked one. The encoder needs every token to see all tokens around it; for example the token loans can disambiguate an earlier bank. |
| B | wrong | Swapped again: bidirectional attention belongs to the encoder. The decoder only generates, and future tokens do not exist yet while generating. |
| C | correct | The encoder sees the whole sequence at once (bidirectional) and processes all tokens in parallel: attention is batched matrix multiplication, exactly what GPUs accelerate. (CPUs compute in sequence; GPUs push everything through at once. This parallelism is why transformers made GPUs so successful, and the reverse.) |
| D | wrong | They use different attention: encoder = bidirectional (all positions, earlier and later); decoder = masked or causal (earlier positions only). |
15.2 Exercise 2: BERT vs GPT
PracticeWhat is the main architectural difference between BERT and GPT? A. BERT uses only the decoder part, while GPT uses only the encoder part. B. BERT uses only the encoder part, while GPT uses only the decoder part. C. BERT has more layers than GPT. D. BERT uses different tokenization techniques than GPT.
Why C is wrong (and interesting): in reality BERT has fewer layers than modern GPT models, but that is a size choice, separate from the architecture: you could build an encoder with more layers than a GPT. Generation models tend to be much bigger because generation is a harder task than representation. BERT-base's 110M parameters were "sort of already enough" for classification and ranking, whereas small generative models (GPT-2 era, T5 era) worked poorly; generative quality arrived only with very large parameter counts (the scaling law), hence the name large language models.
Why D is wrong (and interesting): both use the same technique, trained subword tokenisation (WordPiece-style, Section 4). The follow-up on the slide asks: "do BERT and GPT-2 have a different number of overall tokens?" Yes: they were trained on different collections with different target vocabulary sizes and goals (BERT about 30K pieces; some newer models hundreds of thousands), so the same text tokenises differently in each model. Same technique, different vocabulary.
15.3 Exercise 3: constructing input embeddings
PracticeThe sentence "The cat sits" is tokenised as [CLS] the cat sits [SEP], with embedding dimension 4. Token embeddings: [CLS] = [1,0,0,1], the = [0,1,1,0], cat = [1,1,0,0], sits = [0,0,1,1], [SEP] = [1,0,1,0]. Segment embeddings: A = [0,1,0,1], B = [0,0,0,2] (all special tokens belong to segment A). Position embeddings: pos₀ = [1,0,1,0], pos₁ = [0,1,0,1], pos₂ = [1,1,0,0], pos₃ = [0,0,1,1], pos₄ = [1,0,0,1]. Construct the complete input embedding for every token.
Step 2: add the three vectors for each token (element-wise addition, Section 2.1; one vector per token):
| Token | Token E | + Segment E (A) | + Position E | = Final input embedding |
|---|---|---|---|---|
[CLS] (pos 0) | [1,0,0,1] | [0,1,0,1] | [1,0,1,0] | [2,1,1,2] |
the (pos 1) | [0,1,1,0] | [0,1,0,1] | [0,1,0,1] | [0,3,1,2] |
cat (pos 2) | [1,1,0,0] | [0,1,0,1] | [1,1,0,0] | [2,3,0,1] |
sits (pos 3) | [0,0,1,1] | [0,1,0,1] | [0,0,1,1] | [0,1,2,3] |
[SEP] (pos 4) | [1,0,1,0] | [0,1,0,1] | [1,0,0,1] | [2,1,1,2] |
Sample computation for [CLS]: .
Common traps (both came up in class): (1) putting tokens after [SEP] into segment B, which is wrong here because [SEP] merely ends the single sentence; (2) stacking the three embeddings into a 3×4 matrix, which is wrong because they are added into one 4-dimensional vector per token. The five final vectors, stacked as rows, form the 5×4 input matrix fed to the encoder. Real models use 768+ dimensions, and this week's practical computes them the same way. The explainer How [CLS] becomes [2, 1, 1, 2] animates the [CLS] row.
15.4 Challenge: encoders and decoders for ranking, encoders for generation
PracticeQ1: How can we use encoders or decoders for ranking? Name at least two ways.
With encoders
| Approach | How it works | Connects to |
|---|---|---|
| Single-vector dense retrieval (bi-encoder) | Encode the query into one vector ([CLS] or mean of token embeddings); encode each document into one vector; score = cosine similarity or dot product. Documents are encoded offline at indexing time, so search is fast. | Week 5 |
| Multi-vector, late interaction (ColBERT-style) | Keep one vector per token for query and document; compute every query-token × document-token similarity (a full similarity matrix); aggregate into a final score. More precise, more storage. | Later weeks |
| Cross-encoder reranking | Concatenate query [SEP] document, feed it through BERT, read a relevance score off the [CLS] head. Most accurate, but one model run per (q, d) pair. | Week 8 |
With decoders (LLMs)
| Approach | How it works |
|---|---|
| Prompted ranking (RankGPT-style) | Just ask the LLM. Pointwise: "here is a query and a document; give me a relevance score." Pairwise: "document A or document B: which is more relevant?" Listwise: "here are 10 documents; give me an order." (The lecturer: "you may be using this every single day; why can't people think of the easy way?") |
| Query expansion (query2doc-style) | Have the LLM generate a pseudo-document or answer for the query (like asking Gemini and using its answer), then use that richer text with a lexical ranker. The generated text contains car, cars, automobile, Tesla, BMW …, bridging the vocabulary mismatch of Section 1. |
| Document expansion (doc2query-style) | The same trick flipped, and better for latency: at indexing time, expand each document with the queries or terms it could answer. It costs nothing at query time because it is precomputed while building the index (like BM25 indexing in the Week 3 practical). |
| Decoder hidden states as embeddings | "Every time you see a vector, you can do computations." Cut the model before the output linear layer and use the last hidden vector as a dense representation for similarity search: decoders can be embedders too. |
| Output probabilities as representations (PromptReps-style) | Use the next-token probability distribution over the whole vocabulary as a (sparse, term-statistics-like) representation of the document. A document about this course would put high probability on information, retrieval, web, search, ranking, University of Queensland …; rank by matching those distributions. |
PracticeQ2 (challenge): Can we use an encoder backbone for generation?
- What was the encoder trained to do? Masked language modelling: predicting [MASK] tokens, which is itself a form of token prediction.
- So take the prompt, append a batch of [MASK] tokens (say 256), run the encoder once, and read off predictions for all masked positions at the same time, in place of one token at a time as in an autoregressive decoder.
- The catch: each masked token is predicted independently, so the output may be incoherent.
- The fix (borrowed from image diffusion models): iterative refinement. Like denoising an image step by step, re-run the prediction a few times (3–5 passes), each pass making the text more coherent.
- The payoff: 256 tokens in about 5 passes against 256 sequential passes; GPUs handle parallel tokens happily, so generation gets much faster.
PracticeA student asked after class: document expansion or query expansion with an LLM, which is cheaper for users, and why?
16. Summary and study checklist
16.1 BERT vs GPT at a glance
| BERT (encoder-only) | GPT-3 and LLMs (decoder-only) | |
|---|---|---|
| Released | October 2018 (Google) | June 2020 (OpenAI); ChatGPT 2022 |
| Architecture | Transformer encoder, 12 layers (base) | Transformer decoder |
| Parameters | 110M (base) / 345M (large) | 175B (about 1,000× BERT-base) |
| Attention | Bidirectional: sees the whole sentence | Unidirectional (masked): sees only the past |
| Pre-training | Masked language modelling (about 15% masked) + next sentence prediction | Next-token prediction |
| Output | One contextual vector per token ([CLS] or mean pooling for the sentence) | Probability distribution over the vocabulary (linear layer + softmax on the last hidden vector) |
| Strength | Understanding and representation | Generation and prediction; in-context learning |
| Best for | Search, classification, ranking | Text generation, translation, conversation |
| Key limits | 512-token input cap; cannot generate autoregressively | No access to future context; expensive at scale |
| IR usage | Dense retrieval (Week 5), reranking (Week 8) | RankGPT-style prompting, query and document expansion, RAG, model-based IR |
16.2 Key takeaways
- Term-based models (BM25) fail on vocabulary mismatch and polysemy: they match strings, and meaning escapes them. Modern retrieval needs semantic matching and context awareness.
- All the maths a transformer needs is in Section 2: vector addition (summing embeddings), dot products (similarity scores), vector × matrix multiplication (learned projections; , row · column per cell) and softmax (scores → probabilities).
- The transformer (2017) achieves semantic understanding through self-attention: project each token's vector into q (what I look for), k (what I offer) and v (my content); score , scale by , softmax, then output . The attention matrix is directional (asymmetric).
- Real transformers are "fat" (several attention heads per layer) and "tall" (stacked layers, possible because input and output shapes match), with MLP and layer-norm steps after attention.
- The pipeline: WordPiece tokenisation (about 30K learned subword pieces; solves vocabulary size, OOV and morphology) → input embeddings (token + position + segment, summed element-wise) → encoder and/or decoder. Special tokens ([CLS], [SEP], [MASK], [PAD]; [BOS], [EOS]) act as instructions.
- Encoders (BERT) understand: bidirectional attention, parallel processing, one contextual vector per token; pre-trained with MLM + NSP; fine-tuned for tasks; used as dense retrievers and cross-encoder rerankers in IR.
- Decoders (GPT) generate: masked attention, then last hidden vector × vocabulary projection → softmax → next-token probabilities, looped until [EOS]; sampling explains why answers vary. Trained by next-token prediction; show in-context learning; used in IR through prompted ranking, query and document expansion, RAG and model-based IR.
- Both are trained self-supervised on web-scale text: the text labels itself (masked words, next words). Scale matters (scaling laws), and efficiency now matters too.
- Coming up: Week 5, BERT-based dense retrieval; Week 8, BERT-based rerankers. From here on, essentially every model in the course is a transformer.
16.3 Study checklist
- Explain vocabulary mismatch and polysemy with one example each, and show why BM25 scores 0 for "how to fix laptop" against "computer repair guide for damaged notebooks".
- Multiply small matrices by hand, state the shape rule, and compute a softmax of three scores.
- List the four transformer steps and say which run once and which loop during generation.
- Explain why WordPiece beats word splitting (vocabulary size, OOV, morphology) and describe the merge-training loop.
- Build a token's input embedding from token, segment and position vectors (Exercise 3), assigning segments correctly.
- State the role of [CLS], [SEP], [MASK], [PAD], [BOS] and [EOS].
- Compute q, k, v, scaled scores, attention weights and an output vector for a 3-token toy example; explain and why attention is asymmetric.
- Apply the causal mask to an attention matrix and explain why decoders need it.
- Trace the decoder output head from hidden vector to next-token probability, including the shapes.
- Compare encoder and decoder (attention, purpose, examples) and BERT and GPT (Exercise 2).
- Compute an MLM loss from predicted probabilities and describe NSP.
- Contrast a BERT dense retriever with a cross-encoder reranker, and compute a cosine similarity.
- Name at least four ways to use encoders or decoders for ranking, and explain diffusion language models in one paragraph.
17. Video index and resources
| Video | Topic | Length | Used in sections |
|---|---|---|---|
| Video 1 | Overview of the transformer (BM25 limits, attention, the four steps) | 13:35 | 1, 3, 7, 8 |
| Video 2 | Encoder vs decoder | 3:15 | 9 |
| Video 3 | Input embeddings | 5:25 | 5 |
| Video 4 | Tokenisation (WordPiece) | 2:58 | 4 |
| Video 5 | Special tokens | 4:00 | 6 |
| Video 6 | BERT | 7:18 | 10, 11, 12 |
| Video 7 | Autoregressive decoders | 4:45 | 8, 10, 13, 14 |
- Slide deck (56 slides): w4 slide-deck.pdf. Transcripts of the videos are in
Pre-class youtube Transcripts/; the in-person lecture transcript isINFS7410_S2_2026_STL-Lecture-transcript.txt. - Companion explainers: How [CLS] becomes [2, 1, 1, 2] (Exercise 3, element-wise addition) and The encoder backbone, and why it cannot generate (stacked layers, Q2 challenge).
- Spreadsheets: Transformer in a Spreadsheet (live formulas for the maths refresher, embeddings, five-step attention, multi-head attention, the decoder mask and output head) and Transformer by Hand (compute one transformer block yourself with practice, check and reveal modes).
18. References
- Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS 2017.
- Devlin, J., Chang, M.-W., Lee, K. and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019.
- Brown, T. et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020, 33, 1877–1901.
- Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Yin, B. and Hu, X. (2023). Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond. arXiv:2304.13712.