INFS7410 · Information Retrieval and Web Search · Week 4

How the [CLS] row becomes [2, 1, 1, 2]

A visual walk-through of in-class Exercise 3 (INFS7410 Week 4): the input embedding of each token is the element-wise (column-by-column) sum Token E + Segment E + Position E. The page works the [CLS] row dimension by dimension, shows the one direction the addition runs, and then repeats the sum for every token to build the 5 × 4 input matrix that enters the encoder.

1. The exercise and the input-embedding formula

Sources: slides 24–30 · Video 3 (input embeddings) · Week 4 in-class lecture, Exercise 3

A transformer first turns text into numbers. After tokenisation, every token receives three learned vectors of the same length dd: a token embedding (what the token is, slide 26), a segment embedding (which part of the input it belongs to, slide 28) and a position embedding (where it stands, slide 27). Slide 29 states the rule: the final input is the sum of the three. Like the document vectors of the vector space model in Week 3, each token becomes a vector; here the dimensions are learned features instead of vocabulary terms.

In-class Exercise 3 uses the sentence “The cat sits”, tokenised as [CLS] the cat sits [SEP], with embedding dimension d=4d = 4:

Embedding typeGiven vectors
Token E[CLS] = [1,0,0,1] · the = [0,1,1,0] · cat = [1,1,0,0] · sits = [0,0,1,1] · [SEP] = [1,0,1,0]
Segment Esegment A = [0,1,0,1] · segment B = [0,0,0,2] (all special tokens belong to segment A)
Position Epos₀ = [1,0,1,0] · pos₁ = [0,1,0,1] · pos₂ = [1,1,0,0] · pos₃ = [0,0,1,1] · pos₄ = [1,0,0,1]

The task: construct the complete input embedding for every token.

xi=ti+si+pixi,k=ti,k+si,k+pi,k,k=1,…,d \begin{aligned} \mathbf{x}_i &= \mathbf{t}_i + \mathbf{s}_i + \mathbf{p}_i \\ x_{i,k} &= t_{i,k} + s_{i,k} + p_{i,k}, \qquad k = 1, \dots, d \end{aligned}
Where
ii
position of the token in the input, counted from 0 (so [CLS] is i=0i = 0 and [SEP] is i=4i = 4)
xi\mathbf{x}_i
final input embedding of token ii; stacked as rows, these vectors form the input matrix
ti\mathbf{t}_i
token embedding: the row of the learned lookup table for the token at position ii
si\mathbf{s}_i
segment embedding of the segment (A or B) that token ii belongs to
pi\mathbf{p}_i
position embedding for position ii
kk
dimension index, one column of the vectors
xi,k, ti,k, si,k, pi,kx_{i,k},\ t_{i,k},\ s_{i,k},\ p_{i,k}
the kk-th number in each of the four vectors
dd
embedding dimension: 4 in the exercise, 768 in BERT-base
Read it aloud The input embedding x-i equals the token embedding t-i plus the segment embedding s-i plus the position embedding p-i. Written one number at a time: the k-th number of x-i equals the k-th number of t-i, plus the k-th number of s-i, plus the k-th number of p-i, for every k from 1 to d.
Formula in plain words The sum packs three kinds of information into one vector per token: what the token is, which segment it belongs to, and where it stands. Addition keeps the length dd, so the result has exactly the shape of each piece and the encoder receives one dd-dimensional vector per token. Each dimension is added on its own: a larger value in dimension kk of any piece raises dimension kk of the result and leaves the other dimensions unchanged. Moving the same word to another position changes only pi\mathbf{p}_i, so the same word gets a different input vector at a different place in the sentence; moving it to segment B changes only si\mathbf{s}_i.
Token embedding cat → [1,1,0,0] Segment embedding segment A → [0,1,0,1] Position embedding pos₂ → [1,1,0,0] + Input embedding x₂ (cat) [2, 3, 0, 1]
Each token gets three lookups of the same length; one element-wise addition turns them into its input embedding (shown for “cat”).
PracticeWhy does every token of “The cat sits”, including the final [SEP], get segment A?
Look at slide 28: when does a second segment start?
AnswerThe input holds a single sentence. Segment B starts only when a second text follows a middle [SEP], for example the document after the query in [CLS] query [SEP] document [SEP]. Here the final [SEP] simply ends the one sentence, so every token (special tokens included) gets segment A = [0,1,0,1], and segment B's vector is never used.

2. Stack the three vectors, then add straight down each column

Sources: slides 29–30 · Video 3 (input embeddings) · Week 4 in-class lecture, Exercise 3

Take the three vectors from the [CLS] row of the table and stack them. Each of the 4 dimensions is its own column, shown as a coloured lane in the diagram: dim 1 blue, dim 2 orange, dim 3 green and dim 4 grey. The additions stay inside one lane: they combine the same dimension of three different vectors. This is the element-wise vector addition from the maths refresher in the Week 4 lecture notes.

dim 1 dim 2 dim 3 dim 4 Token E [CLS] Segment E segment A Position E pos₀ Input [CLS] = [2,1,1,2] ++++ ++++ 1001 0101 1010 1+0+1 0+1+0 0+0+1 1+1+0 2112
Each coloured lane is one dimension. The three numbers in a lane add to give that lane's slot in the result, which is exactly the notation [1+0+1, 0+1+0, 0+0+1, 1+1+0].

Worked example 1The [CLS] row, one dimension at a time

1
Assign the right pieces. [CLS] is the token, so t0=[1,0,0,1]\mathbf{t}_0 = [1,0,0,1]. There is only one sentence, so it is in segment A: s0=[0,1,0,1]\mathbf{s}_0 = [0,1,0,1]. It stands at position 0: p0=[1,0,1,0]\mathbf{p}_0 = [1,0,1,0].
2
Dimension 1: 1+0+1=21 + 0 + 1 = 2.
3
Dimension 2: 0+1+0=10 + 1 + 0 = 1.
4
Dimension 3: 0+0+1=10 + 0 + 1 = 1.
5
Dimension 4: 1+1+0=21 + 1 + 0 = 2.
x0=[1+0+1,  0+1+0,0+0+1,  1+1+0]=[2,1,1,2] \begin{aligned} \mathbf{x}_0 &= [1{+}0{+}1,\; 0{+}1{+}0, \\ &\qquad 0{+}0{+}1,\; 1{+}1{+}0] \\ &= [2,1,1,2] \end{aligned}
ResultThe input embedding of [CLS] is [2, 1, 1, 2]: still 4 numbers, one per dimension.
PracticeCompute the input embedding of “the” (position 1).
Token “the” = [0,1,1,0], segment A = [0,1,0,1], pos₁ = [0,1,0,1].
AnswerAdd each column: [0+0+0,  1+1+1,  1+0+0,  0+1+1]=[0,3,1,2][0{+}0{+}0,\; 1{+}1{+}1,\; 1{+}0{+}0,\; 0{+}1{+}1] = [0,3,1,2].

3. Add down each column, within one dimension

Sources: slides 29–30 · Video 3 (input embeddings) · Week 4 in-class lecture, Exercise 3 (common traps)

The sum combines the same dimension of three different vectors. The four numbers inside one vector describe four different features of the token, so they stay separate: adding them would squash the vector into a single number.

✓ Correct: add down one column Token E Segment E Position E 100 010 101 2 dim 1 of the input: 1 + 0 + 1 = 2 ✗ Wrong: add across one row Token E Segment E 1001 0101 2 ✗ summing within a vector is the wrong operation
The sum combines the same dimension of three different vectors. The four numbers inside one vector stay separate.
Common trapBoth of these came up in class. (1) Putting the tokens after [SEP] into segment B: here [SEP] merely ends the single sentence, so there is no second segment. (2) Stacking the three embeddings into a 3 × 4 matrix and handing that on: the three vectors are added into a single 4-dimensional vector per token, so each token contributes one row.
Key ideaElement-wise addition maps three vectors of length dd to one vector of length dd. The output keeps the shape; only the numbers change.
PracticeA classmate adds 1 + 0 + 0 + 1 inside the [CLS] token vector and writes 2. What went wrong?
AnswerThey added across one vector, which mixes four different features into a single number and destroys the vector. The input embedding must stay a 4-dimensional vector, so the addition runs down each dimension across the three vectors (token, segment, position), giving [2,1,1,2] for [CLS].

4. Repeat for every token: the final matrix

Sources: slides 29–30 · Video 3 (input embeddings) · Video 6 (BERT) · Week 4 in-class lecture, Exercise 3

Apply the same column-wise sum to every row of the table. Each token's three vectors collapse into one 4-dimensional row, and the five rows stacked together form the 5×45 \times 4 input embedding matrix. That matrix, with one vector per token, is what enters the encoder's self-attention (see the encoder backbone explainer). In real models the dimension is 768 or more, and the practical computes it in exactly the same way.

Token (position)Token E+ Segment E (A)+ Position E= Final input embedding
[CLS] (pos 0)[1,0,0,1][0,1,0,1][1,0,1,0][2,1,1,2]
the (pos 1)[0,1,1,0][0,1,0,1][0,1,0,1][0,3,1,2]
cat (pos 2)[1,1,0,0][0,1,0,1][1,1,0,0][2,3,0,1]
sits (pos 3)[0,0,1,1][0,1,0,1][0,0,1,1][0,1,2,3]
[SEP] (pos 4)[1,0,1,0][0,1,0,1][1,0,0,1][2,1,1,2]
Table (one row per token) Input matrix (5 × 4) token + segment + position [CLS]thecatsits[SEP] pos 0pos 1pos 2pos 3pos 4 [1,0,0,1]+[0,1,0,1]+[1,0,1,0] [0,1,1,0]+[0,1,0,1]+[0,1,0,1] [1,1,0,0]+[0,1,0,1]+[1,1,0,0] [0,0,1,1]+[0,1,0,1]+[0,0,1,1] [1,0,1,0]+[0,1,0,1]+[1,0,0,1] 2 1 1 2 0 3 1 2 2 3 0 1 0 1 2 3 2 1 1 2 rows 1 and 5 match by coincidence: the token and position vectors swap exactly
Each table row collapses (column-wise) into one row of the matrix. The matrix, one 4-dimensional vector per token, is what enters the encoder's self-attention.

Worked example 2Why rows 1 and 5 are identical

1
[CLS] at position 0: token [1,0,0,1] plus position [1,0,1,0].
2
[SEP] at position 4: token [1,0,1,0] plus position [1,0,0,1]. The two pairs contain the same two vectors in swapped roles.
3
Both add the same segment A vector [0,1,0,1], and addition is commutative, so the order of the pieces is irrelevant.
ResultBoth rows equal [2,1,1,2]. This is an artefact of the tiny hand-made numbers; with learned 768-dimensional vectors two tokens essentially never collide like this.
PracticeSuppose “sits” belonged to segment B = [0,0,0,2]. What is its input embedding, and which rows of the matrix change?
Only one of the three pieces changes.
Answer[0,0,1,1]+[0,0,0,2]+[0,0,1,1]=[0,0,2,4][0,0,1,1] + [0,0,0,2] + [0,0,1,1] = [0,0,2,4]. Only row 4 (sits) changes; every other token keeps segment A and its row stays the same.

5. Summary and study checklist