Week 4: How [CLS] becomes [2,1,1,2] by element-wise additionExplainer4
INFS7410 · Information Retrieval and Web Search · Week 4
How the [CLS] row becomes [2, 1, 1, 2]
A visual walk-through of in-class Exercise 3 (INFS7410 Week 4): the input embedding of each token is the element-wise (column-by-column) sum Token E + Segment E + Position E. The page works the [CLS] row dimension by dimension, shows the one direction the addition runs, and then repeats the sum for every token to build the 5 × 4 input matrix that enters the encoder.
A transformer first turns text into numbers. After tokenisation, every token receives three learned vectors of the same length d: a token embedding (what the token is, slide 26), a segment embedding (which part of the input it belongs to, slide 28) and a position embedding (where it stands, slide 27). Slide 29 states the rule: the final input is the sum of the three. Like the document vectors of the vector space model in Week 3, each token becomes a vector; here the dimensions are learned features instead of vocabulary terms.
In-class Exercise 3 uses the sentence “The cat sits”, tokenised as [CLS] the cat sits [SEP], with embedding dimension d=4:
The task: construct the complete input embedding for every token.
xixi,k=ti+si+pi=ti,k+si,k+pi,k,k=1,…,d
Where
i
position of the token in the input, counted from 0 (so [CLS] is i=0 and [SEP] is i=4)
xi
final input embedding of token i; stacked as rows, these vectors form the input matrix
ti
token embedding: the row of the learned lookup table for the token at position i
si
segment embedding of the segment (A or B) that token i belongs to
pi
position embedding for position i
k
dimension index, one column of the vectors
xi,k,ti,k,si,k,pi,k
the k-th number in each of the four vectors
d
embedding dimension: 4 in the exercise, 768 in BERT-base
Read it aloud
The input embedding x-i equals the token embedding t-i plus the segment embedding s-i plus the position embedding p-i. Written one number at a time: the k-th number of x-i equals the k-th number of t-i, plus the k-th number of s-i, plus the k-th number of p-i, for every k from 1 to d.
Formula in plain words
The sum packs three kinds of information into one vector per token: what the token is, which segment it belongs to, and where it stands. Addition keeps the length d, so the result has exactly the shape of each piece and the encoder receives one d-dimensional vector per token. Each dimension is added on its own: a larger value in dimension k of any piece raises dimension k of the result and leaves the other dimensions unchanged. Moving the same word to another position changes only pi, so the same word gets a different input vector at a different place in the sentence; moving it to segment B changes only si.
Each token gets three lookups of the same length; one element-wise addition turns them into its input embedding (shown for “cat”).PracticeWhy does every token of “The cat sits”, including the final [SEP], get segment A?
Look at slide 28: when does a second segment start?
AnswerThe input holds a single sentence. Segment B starts only when a second text follows a middle [SEP], for example the document after the query in [CLS] query [SEP] document [SEP]. Here the final [SEP] simply ends the one sentence, so every token (special tokens included) gets segment A = [0,1,0,1], and segment B's vector is never used.
2. Stack the three vectors, then add straight down each column
Take the three vectors from the [CLS] row of the table and stack them. Each of the 4 dimensions is its own column, shown as a coloured lane in the diagram: dim 1 blue, dim 2 orange, dim 3 green and dim 4 grey. The additions stay inside one lane: they combine the same dimension of three different vectors. This is the element-wise vector addition from the maths refresher in the Week 4 lecture notes.
Each coloured lane is one dimension. The three numbers in a lane add to give that lane's slot in the result, which is exactly the notation [1+0+1, 0+1+0, 0+0+1, 1+1+0].
Worked example 1The [CLS] row, one dimension at a time
1
Assign the right pieces. [CLS] is the token, so t0=[1,0,0,1]. There is only one sentence, so it is in segment A: s0=[0,1,0,1]. It stands at position 0: p0=[1,0,1,0].
2
Dimension 1:1+0+1=2.
3
Dimension 2:0+1+0=1.
4
Dimension 3:0+0+1=1.
5
Dimension 4:1+1+0=2.
x0=[1+0+1,0+1+0,0+0+1,1+1+0]=[2,1,1,2]
ResultThe input embedding of [CLS] is [2, 1, 1, 2]: still 4 numbers, one per dimension.
PracticeCompute the input embedding of “the” (position 1).
The sum combines the same dimension of three different vectors. The four numbers inside one vector describe four different features of the token, so they stay separate: adding them would squash the vector into a single number.
The sum combines the same dimension of three different vectors. The four numbers inside one vector stay separate.
Common trapBoth of these came up in class. (1) Putting the tokens after [SEP] into segment B: here [SEP] merely ends the single sentence, so there is no second segment. (2) Stacking the three embeddings into a 3 × 4 matrix and handing that on: the three vectors are added into a single 4-dimensional vector per token, so each token contributes one row.
Key ideaElement-wise addition maps three vectors of length d to one vector of length d. The output keeps the shape; only the numbers change.
PracticeA classmate adds 1 + 0 + 0 + 1 inside the [CLS] token vector and writes 2. What went wrong?
AnswerThey added across one vector, which mixes four different features into a single number and destroys the vector. The input embedding must stay a 4-dimensional vector, so the addition runs down each dimension across the three vectors (token, segment, position), giving [2,1,1,2] for [CLS].
4. Repeat for every token: the final matrix
Sources: slides 29–30 · Video 3 (input embeddings) · Video 6 (BERT) · Week 4 in-class lecture, Exercise 3
Apply the same column-wise sum to every row of the table. Each token's three vectors collapse into one 4-dimensional row, and the five rows stacked together form the 5×4 input embedding matrix. That matrix, with one vector per token, is what enters the encoder's self-attention (see the encoder backbone explainer). In real models the dimension is 768 or more, and the practical computes it in exactly the same way.
Token (position)
Token E
+ Segment E (A)
+ Position E
= Final input embedding
[CLS] (pos 0)
[1,0,0,1]
[0,1,0,1]
[1,0,1,0]
[2,1,1,2]
the (pos 1)
[0,1,1,0]
[0,1,0,1]
[0,1,0,1]
[0,3,1,2]
cat (pos 2)
[1,1,0,0]
[0,1,0,1]
[1,1,0,0]
[2,3,0,1]
sits (pos 3)
[0,0,1,1]
[0,1,0,1]
[0,0,1,1]
[0,1,2,3]
[SEP] (pos 4)
[1,0,1,0]
[0,1,0,1]
[1,0,0,1]
[2,1,1,2]
Each table row collapses (column-wise) into one row of the matrix. The matrix, one 4-dimensional vector per token, is what enters the encoder's self-attention.
Worked example 2Why rows 1 and 5 are identical
1
[CLS] at position 0: token [1,0,0,1] plus position [1,0,1,0].
2
[SEP] at position 4: token [1,0,1,0] plus position [1,0,0,1]. The two pairs contain the same two vectors in swapped roles.
3
Both add the same segment A vector [0,1,0,1], and addition is commutative, so the order of the pieces is irrelevant.
ResultBoth rows equal [2,1,1,2]. This is an artefact of the tiny hand-made numbers; with learned 768-dimensional vectors two tokens essentially never collide like this.
PracticeSuppose “sits” belonged to segment B = [0,0,0,2]. What is its input embedding, and which rows of the matrix change?
Only one of the three pieces changes.
Answer[0,0,1,1]+[0,0,0,2]+[0,0,1,1]=[0,0,2,4]. Only row 4 (sits) changes; every other token keeps segment A and its row stays the same.