Week 4: The encoder backbone, and why it cannot generateExplainer4
INFS7410 · Information Retrieval and Web Search · Week 4
The encoder backbone, and why it cannot generate
How a stack of bidirectional self-attention layers refines one vector per token, what a single layer does to a single token (“bank” in “the river bank flooded”), and why generating text left to right needs a decoder: a causal mask, a next-token objective and a sequential loop.
1. The backbone: a stack of layers refining one vector per token
Sources: slides 14–15, 22, 40, 43 · Video 2 (encoder vs decoder) · Video 6 (BERT)
Input embeddings enter at the bottom: the token + position + segment matrix built in Exercise 3 (see the element-wise addition explainer). Each encoder layer applies bidirectional self-attention and then a feed-forward network (FFN), and passes a matrix of the same shape upward. All tokens are processed in parallel at every layer. BERT-base stacks 12 such layers (slide 40).
The matrix keeps its shape (one vector per token) from bottom to top; only the content of the vectors changes, becoming more contextualised at every layer. The [CLS] output feeds ranking or classification; “bank” ends up encoding the river bank specifically.
This is the context awareness that term matching lacks. BM25 from Week 3 treats “bank” as the same string in every document (slide 4: “where is the closest bank?” versus a river bank). The encoder instead gives “bank” a different output vector in each sentence, because every layer mixes in information from the surrounding tokens.
Key ideaShape in = shape out. Stacking works because each layer maps an n×d matrix to another n×d matrix: n tokens, d numbers per token (768 in BERT-base).
PracticeBERT-base reads [CLS] the river bank flooded [SEP] with hidden size 768. What is the shape of the matrix after layer 1, and after layer 12?
Answer6 × 768 in both cases: one 768-dimensional vector per token. The shape never changes through the stack; each layer rewrites the numbers so the vectors carry more context. The [CLS] row of the final layer is the one used for ranking or classification.
2. What one layer does to one token
Sources: slides 10–13 · Video 1 (overview transformer) · Video 6 (BERT) · Week 4 lecture notes (self-attention worked example)
Inside each layer, “bank” compares its query vector against every token's key vector (in both directions), turns the scores into weights with softmax, blends the value vectors by those weights, and the FFN transforms the result. Every other token does the same, simultaneously. This is the self-attention calculation worked by hand in the Week 4 lecture notes.
“Bank” attends to every token, including itself; the heaviest weights go to the words that settle its meaning.
indices over all tokens of the input, including i itself and tokens to its left and right
n
number of tokens in the input
qi
query vector of token i (“what I am looking for”), a learned projection of its current vector
kj
key vector of token j (“what I offer”), another learned projection
vj
value vector of token j (“my content”), a third learned projection
dk
length of the query and key vectors; dividing by dk keeps the scores in a moderate range
aij
attention weight from token i to token j: non-negative, and the weights of one token sum to 1
zi
attention output for token i, which then goes through the FFN
Read it aloud
The attention weight a-i-j equals e to the power of (q-i dot k-j divided by the square root of d-k), divided by the sum over all tokens l of e to the power of (q-i dot k-l divided by the square root of d-k). The output z-i is the sum over all tokens j of a-i-j times the value vector v-j.
Formula in plain words
Token i scores every token j with a dot product between its query and their key: a large score means “this token is useful for understanding me”. Softmax (the Week 4 maths refresher) turns the scores into weights that are positive and sum to 1, so a higher score takes a larger share at the expense of the others. The output is a weighted average of the value vectors, so tokens with large weights shape the new vector most. Because j runs over the whole input, words on both sides count: this is the bidirectional part. The scaling by dk stops long vectors from producing huge scores that would push softmax towards a single winner.
The self-attention worked example, inside the backbone: score, softmax, blend of values, FFN, refined vector.
Worked example 1One attention step for “bank”
Illustrative 2-dimensional vectors (dk=2, dk≈1.414), using three tokens: qbank=[1,1]; keys river [2,0], bank [0,1], flooded [1,1]; values river [1,0], bank [0,1], flooded [1,1].
ResultThe new “bank” vector is about [0.80, 0.60]: mostly built from “river” and “flooded”, with a smaller share of its own value. The FFN then transforms this vector before it moves up to the next layer.
PracticeIn Worked example 1, change the key of “bank” to [1,1]. What are the new weights and the new output?
All three scores become equal.
AnswerEach score is 1.4142=1.414, so softmax gives equal weights of 31. The output is the plain average of the values: 31([1,0]+[0,1]+[1,1])=[0.667,0.667].
3. Why this machine cannot generate
Sources: slides 20–23, 44, 50–51 · Video 2 (encoder vs decoder) · Video 6 (BERT) · Video 7 (autoregressive decoders) · Week 4 in-class lecture (encoder backbone for generation)
Generation is a sequential loop over a causal mask: predict, append, repeat (slides 23 and 51). The encoder is a parallel machine with no mask and a masked-language-modelling (MLM) objective: it can fill a blank brilliantly, but it has no “next token” to predict and no stable loop to run. Slide 50 lists this as BERT limitation 2: it cannot generate text autoregressively.
A decoder runs a loop: predict the next token from the past, append it, and repeat until [EOS].The encoder is a parallel machine with no mask and an MLM objective: it can fill a blank, but it has no “next token” to predict and no stable loop to run.
the current generation step (the position being produced)
y<t
all tokens before position t: the prompt plus everything generated so far
y^t
the token the decoder outputs at step t
m
a masked position in the encoder's input (about 15% of positions during BERT pre-training)
y∖m
the whole input except position m: tokens on both the left and the right
y^m
the encoder's prediction for the masked position
w,V
a candidate token and the vocabulary (about 30K WordPiece tokens)
argmax
“the w that gives the largest probability” (the greedy choice; sampling picks from the distribution instead)
Read it aloud
For the decoder: y-hat-t is the word w in the vocabulary V that maximises the probability of w given all tokens before position t. For the encoder: y-hat-m is the word w in V that maximises the probability of w given every token of the input except position m.
Formula in plain words
The two models answer different questions. The decoder learns “what comes next, given only the past”, so its output at step t can be appended and fed back in: the conditioning set y<t simply grows by one token each step and the earlier predictions stay valid. The encoder learns “what fills this blank, given both sides”; its conditioning set already contains the future. Append a new token and every earlier vector changes (bidirectional attention sees it), so there is no stable loop, and each blank is predicted on its own, so several blanks filled at once need not agree with each other.
Watch out“Cannot generate” means cannot generate autoregressively, left to right. In class the lecturer described a very recent workaround, diffusion language models: append many [MASK] tokens (for example 256), predict them all in parallel, then run a few refinement passes (3–5) to make the text coherent. That is a different generation procedure built on the encoder's fill-in-the-blank skill; see the in-class exercises in the Week 4 lecture notes.
TakeawaysOne-line summary: the encoder backbone = input embeddings → 12 × (bidirectional self-attention + FFN) → one contextualised vector per token, all in parallel. Generation needs the opposite regime (sequential prediction over a causal mask with a next-token training objective), which is exactly what the decoder half of the transformer provides (BERT limitation 2, slide 50).
PracticeWhy can you not generate a sentence with BERT by repeatedly appending one [MASK] to the input and taking the prediction?
Think about what happens to the earlier tokens' vectors, and what BERT was trained to do.
AnswerThree reasons. (1) BERT has no causal mask, so appending a token changes every earlier vector and the “past” never stays fixed. (2) It was trained with MLM to fill blanks using context from both sides, so it never learned P(next∣past). (3) Each blank is predicted independently, so the words need not fit together into coherent left-to-right text. Decoders avoid all three with a causal mask and next-token training.
4. Summary and study checklist
The encoder backbone: input embeddings → 12 layers (BERT-base) of bidirectional self-attention + FFN → one contextualised vector per token.
Each layer keeps the n×d shape; only the content of the vectors changes. The [CLS] output is used for ranking and classification.
Self-attention: scores qi⋅kj/dk, softmax to weights, weighted sum of values, then the FFN.
All tokens are processed in parallel and see the whole input (no mask).
Generation needs a causal mask, a next-token objective and a sequential loop: the decoder's design.
Can you explain, with the two argmax formulas, why an MLM-trained encoder cannot run the predict-append-repeat loop?