INFS7410 · Information Retrieval and Web Search · Week 4

The encoder backbone, and why it cannot generate

How a stack of bidirectional self-attention layers refines one vector per token, what a single layer does to a single token (“bank” in “the river bank flooded”), and why generating text left to right needs a decoder: a causal mask, a next-token objective and a sequential loop.

1. The backbone: a stack of layers refining one vector per token

Sources: slides 14–15, 22, 40, 43 · Video 2 (encoder vs decoder) · Video 6 (BERT)

Input embeddings enter at the bottom: the token + position + segment matrix built in Exercise 3 (see the element-wise addition explainer). Each encoder layer applies bidirectional self-attention and then a feed-forward network (FFN), and passes a matrix of the same shape upward. All tokens are processed in parallel at every layer. BERT-base stacks 12 such layers (slide 40).

[CLS] output → ranking / classification “bank” output → now encodes the river-bank sense output vectors, one per token Encoder layer 12 ⋮ 12 layers in BERT-base Encoder layer 2 · same two sub-layers Encoder layer 1 Bidirectional self-attention every token attends to every token (no mask) Feed-forward network: each vector on its own input embeddings = token + position + segment [CLS] the river bank flooded [SEP] ↑ all tokens processed in parallel; the whole input is visible
The matrix keeps its shape (one vector per token) from bottom to top; only the content of the vectors changes, becoming more contextualised at every layer. The [CLS] output feeds ranking or classification; “bank” ends up encoding the river bank specifically.

This is the context awareness that term matching lacks. BM25 from Week 3 treats “bank” as the same string in every document (slide 4: “where is the closest bank?” versus a river bank). The encoder instead gives “bank” a different output vector in each sentence, because every layer mixes in information from the surrounding tokens.

Key ideaShape in = shape out. Stacking works because each layer maps an n×dn \times d matrix to another n×dn \times d matrix: nn tokens, dd numbers per token (768 in BERT-base).
PracticeBERT-base reads [CLS] the river bank flooded [SEP] with hidden size 768. What is the shape of the matrix after layer 1, and after layer 12?
Answer6 × 768 in both cases: one 768-dimensional vector per token. The shape never changes through the stack; each layer rewrites the numbers so the vectors carry more context. The [CLS] row of the final layer is the one used for ranking or classification.

2. What one layer does to one token

Sources: slides 10–13 · Video 1 (overview transformer) · Video 6 (BERT) · Week 4 lecture notes (self-attention worked example)

Inside each layer, “bank” compares its query vector against every token's key vector (in both directions), turns the scores into weights with softmax, blends the value vectors by those weights, and the FFN transforms the result. Every other token does the same, simultaneously. This is the self-attention calculation worked by hand in the Week 4 lecture notes.

strongest attention: “river” and “flooded” line thickness = attention weight from “bank” [CLS] the river bank flooded [SEP] bidirectional: context on the left and on the right both count
“Bank” attends to every token, including itself; the heaviest weights go to the words that settle its meaning.
aij=exp⁡(qi⋅kj/dk)∑l=1nexp⁡(qi⋅kl/dk)zi=∑j=1naij vj \begin{aligned} a_{ij} &= \frac{\exp(\mathbf{q}_i \cdot \mathbf{k}_j / \sqrt{d_k})}{\sum_{l=1}^{n} \exp(\mathbf{q}_i \cdot \mathbf{k}_l / \sqrt{d_k})} \\ \mathbf{z}_i &= \sum_{j=1}^{n} a_{ij}\, \mathbf{v}_j \end{aligned}
Where
ii
the token being updated (for example “bank”)
j, lj,\ l
indices over all tokens of the input, including ii itself and tokens to its left and right
nn
number of tokens in the input
qi\mathbf{q}_i
query vector of token ii (“what I am looking for”), a learned projection of its current vector
kj\mathbf{k}_j
key vector of token jj (“what I offer”), another learned projection
vj\mathbf{v}_j
value vector of token jj (“my content”), a third learned projection
dkd_k
length of the query and key vectors; dividing by dk\sqrt{d_k} keeps the scores in a moderate range
aija_{ij}
attention weight from token ii to token jj: non-negative, and the weights of one token sum to 1
zi\mathbf{z}_i
attention output for token ii, which then goes through the FFN
Read it aloud The attention weight a-i-j equals e to the power of (q-i dot k-j divided by the square root of d-k), divided by the sum over all tokens l of e to the power of (q-i dot k-l divided by the square root of d-k). The output z-i is the sum over all tokens j of a-i-j times the value vector v-j.
Formula in plain words Token ii scores every token jj with a dot product between its query and their key: a large score means “this token is useful for understanding me”. Softmax (the Week 4 maths refresher) turns the scores into weights that are positive and sum to 1, so a higher score takes a larger share at the expense of the others. The output is a weighted average of the value vectors, so tokens with large weights shape the new vector most. Because jj runs over the whole input, words on both sides count: this is the bidirectional part. The scaling by dk\sqrt{d_k} stops long vectors from producing huge scores that would push softmax towards a single winner.
Scores: q(bank) · k(j) / √dₖ one per token j, left and right Softmax → attention weights non-negative, sum to 1 Weighted blend of all value vectors z(bank) = Σ aⱼ vⱼ Feed-forward network (FFN) transforms the blended vector New “bank” vector passed up to the next layer
The self-attention worked example, inside the backbone: score, softmax, blend of values, FFN, refined vector.

Worked example 1One attention step for “bank”

Illustrative 2-dimensional vectors (dk=2d_k = 2, dk≈1.414\sqrt{d_k} \approx 1.414), using three tokens: qbank=[1,1]\mathbf{q}_{\text{bank}} = [1,1]; keys river [2,0][2,0], bank [0,1][0,1], flooded [1,1][1,1]; values river [1,0][1,0], bank [0,1][0,1], flooded [1,1][1,1].

1
Scores.river:21.414=1.414bank:11.414=0.707flooded:21.414=1.414 \begin{aligned} \text{river:}\quad \frac{2}{1.414} &= 1.414 \\ \text{bank:}\quad \frac{1}{1.414} &= 0.707 \\ \text{flooded:}\quad \frac{2}{1.414} &= 1.414 \end{aligned}
2
Exponentiate.e1.414=4.113e0.707=2.028e1.414=4.113total=10.255 \begin{aligned} e^{1.414} &= 4.113 \\ e^{0.707} &= 2.028 \\ e^{1.414} &= 4.113 \\ \text{total} &= 10.255 \end{aligned}
3
Weights. river 0.4010.401, bank 0.1980.198, flooded 0.4010.401 (they sum to 1).
4
Blend the values.
zbank=0.401 [1,0]+0.198 [0,1]+0.401 [1,1]=[0.802, 0.599] \begin{aligned} \mathbf{z}_{\text{bank}} &= 0.401\,[1,0] + 0.198\,[0,1] \\ &\quad + 0.401\,[1,1] \\ &= [0.802,\ 0.599] \end{aligned}
ResultThe new “bank” vector is about [0.80, 0.60]: mostly built from “river” and “flooded”, with a smaller share of its own value. The FFN then transforms this vector before it moves up to the next layer.
PracticeIn Worked example 1, change the key of “bank” to [1,1]. What are the new weights and the new output?
All three scores become equal.
AnswerEach score is 21.414=1.414\frac{2}{1.414} = 1.414, so softmax gives equal weights of 13\frac{1}{3}. The output is the plain average of the values: 13([1,0]+[0,1]+[1,1])=[0.667, 0.667]\frac{1}{3}([1,0] + [0,1] + [1,1]) = [0.667,\ 0.667].

3. Why this machine cannot generate

Sources: slides 20–23, 44, 50–51 · Video 2 (encoder vs decoder) · Video 6 (BERT) · Video 7 (autoregressive decoders) · Week 4 in-class lecture (encoder backbone for generation)

Generation is a sequential loop over a causal mask: predict, append, repeat (slides 23 and 51). The encoder is a parallel machine with no mask and a masked-language-modelling (MLM) objective: it can fill a blank brilliantly, but it has no “next token” to predict and no stable loop to run. Slide 50 lists this as BERT limitation 2: it cannot generate text autoregressively.

✓ Decoder: built for generation UQ is UQ is the UQ is the University UQ is the University of UQ is the University of Queensland predict → append → repeat · trained on next-token prediction: knows P(next | past) · causal mask: each step sees only the past · “next position” is a built-in concept
A decoder runs a loop: predict the next token from the past, append it, and repeat until [EOS].
✗ Encoder: no notion of “next” its attention (no mask) everything sees everything generation needs (causal) future hidden: BERT lacks this · trained on MLM: fill blanks from BOTH sides, never learned P(next | past) · append a token → ALL earlier vectors change (bidirectional attention sees the new token) · what it CAN do: [MASK] prediction the 13th [MASK] was → century (94%) one blank at a time ≠ coherent left-to-right text (rows = token doing the attending; columns = token attended to)
The encoder is a parallel machine with no mask and an MLM objective: it can fill a blank, but it has no “next token” to predict and no stable loop to run.
decoder (next token):y^t=arg⁡max⁡w∈V P(w∣y<t)encoder (MLM):y^m=arg⁡max⁡w∈V P(w∣y∖m) \begin{aligned} &\text{decoder (next token):} \\ &\hat{y}_t = \arg\max_{w \in V}\, P\left(w \mid y_{<t}\right) \\ &\text{encoder (MLM):} \\ &\hat{y}_m = \arg\max_{w \in V}\, P\left(w \mid \mathbf{y}_{\setminus m}\right) \end{aligned}
Where
tt
the current generation step (the position being produced)
y<ty_{<t}
all tokens before position tt: the prompt plus everything generated so far
y^t\hat{y}_t
the token the decoder outputs at step tt
mm
a masked position in the encoder's input (about 15% of positions during BERT pre-training)
y∖m\mathbf{y}_{\setminus m}
the whole input except position mm: tokens on both the left and the right
y^m\hat{y}_m
the encoder's prediction for the masked position
w, Vw,\ V
a candidate token and the vocabulary (about 30K WordPiece tokens)
arg⁡max⁡\arg\max
“the ww that gives the largest probability” (the greedy choice; sampling picks from the distribution instead)
Read it aloud For the decoder: y-hat-t is the word w in the vocabulary V that maximises the probability of w given all tokens before position t. For the encoder: y-hat-m is the word w in V that maximises the probability of w given every token of the input except position m.
Formula in plain words The two models answer different questions. The decoder learns “what comes next, given only the past”, so its output at step tt can be appended and fed back in: the conditioning set y<ty_{<t} simply grows by one token each step and the earlier predictions stay valid. The encoder learns “what fills this blank, given both sides”; its conditioning set already contains the future. Append a new token and every earlier vector changes (bidirectional attention sees it), so there is no stable loop, and each blank is predicted on its own, so several blanks filled at once need not agree with each other.
Watch out“Cannot generate” means cannot generate autoregressively, left to right. In class the lecturer described a very recent workaround, diffusion language models: append many [MASK] tokens (for example 256), predict them all in parallel, then run a few refinement passes (3–5) to make the text coherent. That is a different generation procedure built on the encoder's fill-in-the-blank skill; see the in-class exercises in the Week 4 lecture notes.
TakeawaysOne-line summary: the encoder backbone = input embeddings → 12 × (bidirectional self-attention + FFN) → one contextualised vector per token, all in parallel. Generation needs the opposite regime (sequential prediction over a causal mask with a next-token training objective), which is exactly what the decoder half of the transformer provides (BERT limitation 2, slide 50).
PracticeWhy can you not generate a sentence with BERT by repeatedly appending one [MASK] to the input and taking the prediction?
Think about what happens to the earlier tokens' vectors, and what BERT was trained to do.
AnswerThree reasons. (1) BERT has no causal mask, so appending a token changes every earlier vector and the “past” never stays fixed. (2) It was trained with MLM to fill blanks using context from both sides, so it never learned P(next∣past)P(\text{next} \mid \text{past}). (3) Each blank is predicted independently, so the words need not fit together into coherent left-to-right text. Decoders avoid all three with a causal mask and next-token training.

4. Summary and study checklist