INFS7410 · Information Retrieval and Web Search · Week 5

Embedding-based Retrieval Models (Dense Retrievers)

This document tells you how dense retrievers use transformer encoders and decoders to find relevant documents. It covers bi-encoders (DPR, RepBERT, ANCE, Contriever), late interaction (ColBERT) and LLM-based embedders (E5-Mistral, LLM2Vec, RepLLaMA). For each model, it gives the representation, the similarity function, the training loss and the negative sampling method. Written in Simplified Technical English. Lecturer: Dr Shuai Wang, The University of Queensland.

1. The big picture: from lexical to dense retrieval

Sources: slides 2–3, 8 · Video 1 (Overview: embedding-based retrievers) · In-class slides 4, 7

Where we are in the course. In Week 3 you learned term-based lexical models such as BM25. These models score a document by the terms it shares with the query. In Week 4 you learned the transformer architecture. The encoder (BERT) reads the full text with bidirectional attention. It makes one contextual vector for each token. The decoder (GPT) reads the text from left to right with causal attention. It predicts the next token. This week connects the two topics. Transformers turn queries and documents into vectors. The system then ranks documents by vector similarity.

Watch outWhy lexical models are not sufficient. BM25 can only match documents that share exact words with the query. This causes the vocabulary mismatch problem. The words "car" and "automobile" share no terms. A lexical model gives them zero similarity. But the two words have the same meaning. Dense retrievers correct this problem. They put the two words at points near each other in a learned vector space.
Lexical match (BM25) q: car d: automobile ≠ shared terms: none term-overlap score = 0 vocabulary mismatch Dense match (embeddings) η(car) η(automobile) small angle: high similarity
Lexical models need shared terms. Dense retrievers compare learned vectors. Thus, two synonyms with no shared terms can get a high score.

The models of this week, in groups by backbone architecture:

BackboneRepresentation styleModels
Transformer encoderSingle vector: one vector for each query or documentDPR, ANCE, RepBERT, Contriever
Transformer encoderMulti vector: one vector for each tokenColBERT
Transformer decoder (LLM)Single vector: the vector of the last tokenE5-Mistral, LLM2Vec, RepLLaMA
Key ideaYou can use the encoder or the decoder of the transformer for dense retrieval. The decoder was made for text generation. But its internal layers also contain good text representations. The first models used encoders, because encoders read the full text at one time. Later models showed that large decoders also work well.
PracticeA user searches for "automobile insurance". The best document only says "car cover". Why does BM25 miss it? What does a dense retriever do differently?
Think about what BM25 counts. Think about where a dense retriever puts each text.
AnswerBM25 scores term overlap. The query and the document share no terms. Thus, BM25 gives this document a score of zero for those terms (vocabulary mismatch). A dense retriever puts the two texts into a learned vector space. In this space, synonyms are near each other. Thus, the similarity of the two vectors is high, although the texts share no words.

2. The table of relevance models: costs

Sources: slides 4–6 · Video 1 (Overview: embedding-based retrievers) · In-class slide 6

Recall the table of relevance models from Week 3. Two costs describe each model family:

Model familyExamplesScorer costEncoder cost
Term-based lexical modelsBM25, TF-IDFO(<1)O(<1): inverted-index lookupsO(∣d∣)O(|d|): one pass to count terms
Single-vector embedding-based retrieversDPR, ANCE, RepBERT, Contriever, E5, LLM2Vec, RepLLaMAO(1)O(1): one dot product or cosineO(∣d∣2)O(|d|^2): attention over all token pairs
Multi-vector embedding-based retrieversColBERTO(m×n)O(m \times n): each query token against each document tokenO(∣d∣2)O(|d|^2): the same transformer encoding
Learned sparse retrieversnext week (Week 6)(next week)(next week)

How to read the cost terms:

Offline, once per document (indexing time) Documents whole collection Encoder η O(|d|²) per document Vector index stored document vectors Online, for every query (the user is waiting) Query q short text Encoder η one short pass Similarity φ O(1) per document Ranked list
The system pays the quadratic encoder cost offline, one time for each document. At query time, it encodes only one short query. Then it computes cheap similarity scores.
Key ideaSingle-vector retrievers are fast in practice for one reason. The system does the expensive O(∣d∣2)O(|d|^2) encoding of documents one time, offline. At query time, the system encodes only the short query. It then computes millions of cheap O(1)O(1) similarity operations against the stored document vectors.
PracticeWhy is the scorer cost of a bi-encoder O(1) for each document, and its encoder cost O(|d|²)? Why does the quadratic part have almost no effect on query latency?
AnswerScoring compares two vectors of fixed size with one dot product or cosine. The document length has no effect on this step. Encoding runs self-attention over all token pairs. This work is quadratic in length. The system computes the document encodings offline and stores them. At query time, the system encodes only the short query (one pass). The work for each document is the constant-cost similarity.
PracticeWhich slot in the cost table stays empty this week? When does the course fill it?
AnswerThe slot for learned sparse retrievers. Week 6 covers these models. They are more complex to build. But they are often more efficient than the embedding-based models of this week.

3. Dense representations: embedders and similarity

Sources: slide 7 · Video 1 (Overview: embedding-based retrievers) · In-class slide 8

Each dense retriever has two parts. The first part is an embedder η(⋅)\eta(\cdot). This is a neural network (BERT, an LLM, or other) that changes a text into a vector of fixed size. The same type of embedder applies to the query qq and the document dd. Thus, the two vectors are in the same space. The second part is a similarity function ϕ(⋅,⋅)\phi(\cdot,\cdot). This is a cheap function (dot product, cosine similarity, or other) that compares the two vectors.

score(q,d)=ϕ(η(q), η(d)) \mathrm{score}(q,d) = \phi\big(\eta(q),\ \eta(d)\big)
Where
q, dq,\ d
the query text and the document text
η(⋅)\eta(\cdot)
the embedder (eta): a neural network that changes a text into a vector of fixed size
η(q), η(d)\eta(q),\ \eta(d)
the query embedding and the document embedding (for example, 768 numbers each)
ϕ(⋅,⋅)\phi(\cdot,\cdot)
the similarity function (phi), for example inner product or cosine similarity
score(q,d)\mathrm{score}(q,d)
one number that ranks dd for qq
Read it aloud The score of document d for query q is phi of eta of q and eta of d. This is the similarity between the query embedding and the document embedding.
Formula in plain words Put the query through the embedder. Put the document through the embedder. Then measure how near the two vectors are. The embedder η\eta contains all the "understanding". Training teaches it. The function ϕ\phi is cheap on purpose. Thus, the system can apply it to millions of stored document vectors. The goal: ϕ(η(q),η(d))\phi(\eta(q),\eta(d)) is large when dd is relevant to qq. A better embedder moves relevant pairs nearer and irrelevant pairs farther apart.
query q Embedder η η(q) [0.8, −1.2, …, 2.4, −0.3] document d Embedder η η(d) [0.5, 0.0, …, 2.6, −0.7] φ score
The dense retrieval recipe from slide 7. The same embedder changes the query and the document into low-dimension vectors. Then φ compares the vectors.

The vectors have a low number of dimensions. BERT-based models use 768 dimensions. Decoder (LLM) based models use approximately 3,000 to 4,000 dimensions. A lexical bag-of-words vector has tens of thousands of dimensions. Dense retrieval measures relevance as distance in a learned semantic space. Thus, "car" and "automobile" can have almost equal embeddings, although they share no characters.

3.1 The two standard similarity functions

Dot product (also called inner product). The Week 4 maths refresher shows vector operations.

ϕdot(q,d)=q⋅d=∑i=1kqi di \phi_{\text{dot}}(\mathbf{q},\mathbf{d}) = \mathbf{q}\cdot\mathbf{d} = \sum_{i=1}^{k} q_i\, d_i
Where
q, d\mathbf{q},\ \mathbf{d}
the query embedding and the document embedding, η(q)\eta(q) and η(d)\eta(d)
kk
the number of dimensions of each vector (for example, 768)
qi, diq_i,\ d_i
entry ii of the query vector and of the document vector
∑i=1k\sum_{i=1}^{k}
add over all kk dimensions
Read it aloud Phi-dot of q and d equals q dot d. This is the sum, for i from 1 to k, of q-i multiplied by d-i.
Formula in plain words Multiply the two vectors entry by entry. Then add all the products. The result is one number. A larger number means more similar. A dimension where both entries are large with the same sign increases the score. A dimension with opposite signs decreases the score. The formula uses the raw entries. Thus, a longer vector gets a larger score, also when the direction is the same. The dot product measures direction and length together.

Cosine similarity. The vector space model in Week 3 uses the same cosine. Here, it applies to learned embeddings.

ϕcos⁡(q,d)=q⋅d∥q∥ ∥d∥ \phi_{\cos}(\mathbf{q},\mathbf{d}) = \frac{\mathbf{q}\cdot\mathbf{d}}{\lVert\mathbf{q}\rVert\,\lVert\mathbf{d}\rVert}
Where
q⋅d\mathbf{q}\cdot\mathbf{d}
the dot product from the formula above
∥q∥\lVert\mathbf{q}\rVert
the length (L2 norm) of q\mathbf{q}: ∑iqi2\sqrt{\sum_i q_i^2}
∥d∥\lVert\mathbf{d}\rVert
the length (L2 norm) of d\mathbf{d}
Read it aloud Phi-cos of q and d equals q dot d, divided by the norm of q multiplied by the norm of d.
Formula in plain words Compute the dot product. Then divide it by the product of the two vector lengths. The division removes the effect of length. The result is the cosine of the angle between the vectors. It is 1 when the vectors point in the same direction. It is 0 when they are at a right angle. It is −1 when they point in opposite directions. A longer vector gives the same cosine. A vector that turns away from the other vector gives a lower cosine. If you first scale both vectors to length 1 (L2 normalisation), the dot product and the cosine are equal.
q d₁ d₂ = 2·d₁ θ q · d₁ = 8, q · d₂ = 16 dot product doubles with the length cos(q, d₁) = cos(q, d₂) ≈ 0.889 same angle θ, same cosine
The dot product changes with direction and length. The cosine changes only with direction. The numbers come from Worked example 1.

Worked example 1Dot product against cosine similarity

These are illustrative, small 3-dimension embeddings for a query and two documents:

q=[2,1,2],d1=[1,2,2],d2=[2,4,4] \mathbf{q} = [2, 1, 2],\quad \mathbf{d}_1 = [1, 2, 2],\quad \mathbf{d}_2 = [2, 4, 4]
1
Compute the dot product with d1\mathbf{d}_1. Multiply entry by entry. Then add. q⋅d1=(2)(1)+(1)(2)+(2)(2)=2+2+4=8 \mathbf{q}\cdot\mathbf{d}_1 = (2)(1) + (1)(2) + (2)(2) = 2 + 2 + 4 = 8
2
Compute the norms. A norm is the square root of the sum of the squared entries. ∥q∥=22+12+22=9=3∥d1∥=12+22+22=9=3 \begin{aligned} \lVert\mathbf{q}\rVert &= \sqrt{2^2+1^2+2^2} = \sqrt{9} = 3 \\ \lVert\mathbf{d}_1\rVert &= \sqrt{1^2+2^2+2^2} = \sqrt{9} = 3 \end{aligned}
3
Compute the cosine similarity with d1\mathbf{d}_1: ϕcos⁡(q,d1)=83×3=89≈0.889 \phi_{\cos}(\mathbf{q},\mathbf{d}_1) = \frac{8}{3 \times 3} = \frac{8}{9} \approx 0.889
4
Do the same for d2=2d1\mathbf{d}_2 = 2\mathbf{d}_1. This vector has the same direction and two times the length. q⋅d2=(2)(2)+(1)(4)+(2)(4)=16∥d2∥=4+16+16=6ϕcos⁡(q,d2)=163×6=89≈0.889 \begin{aligned} \mathbf{q}\cdot\mathbf{d}_2 &= (2)(2) + (1)(4) + (2)(4) = 16 \\ \lVert\mathbf{d}_2\rVert &= \sqrt{4+16+16} = 6 \\ \phi_{\cos}(\mathbf{q},\mathbf{d}_2) &= \frac{16}{3 \times 6} = \frac{8}{9} \approx 0.889 \end{aligned}
ResultThe dot product goes from 8 to 16 when the document vector length doubles. The cosine similarity stays at 0.889. The dot product reacts to vector length. The cosine reacts only to direction. DPR, ANCE and RepLLaMA use the dot product. Contriever uses cosine similarity. Thus, these models can behave differently. ColBERT normalises its token vectors to length 1 before the dot products. This makes its dot products equal to cosines.
PracticeTwo document embeddings point in the same direction. One vector is three times as long. Which similarity function gives them the same rank? Which function prefers the longer vector?
AnswerCosine similarity gives them the same rank, because it divides by the two lengths and keeps only the angle. The dot product gives the longer vector three times the score, because the dot product changes with length.

4. Two backbones: encoder vs decoder

Sources: slides 8–11, 14–16 · Video 1 (Overview: embedding-based retrievers) · In-class slides 9, 14–15

4.1 Encoder-only (BERT-style): one vector per token, then pool

A transformer encoder (Week 4) reads the full input with bidirectional attention. It outputs one contextual vector for each token. Retrieval needs one representation of the full text. Thus, a model must change many token vectors into one text representation. There are three standard methods:

input text: [CLS] deep learning for search … Transformer encoder (BERT) v[CLS] v₁ v₂ v₃ v₄ … one vector per token A: [CLS] vector keep only v[CLS] DPR · ANCE B: mean vector average all token vectors RepBERT · Contriever C: all vectors keep every token vector ColBERT
An encoder makes one vector for each token. Single-vector models keep the [CLS] vector (DPR, ANCE) or the mean of all token vectors (RepBERT, Contriever). ColBERT keeps all the vectors.

This is the formula for mean pooling:

ηmean(t)=1n∑i=1nvi \eta_{\text{mean}}(t) = \frac{1}{n}\sum_{i=1}^{n} \mathbf{v}_i
Where
tt
the text to embed (a query or a document)
nn
the number of tokens in tt
vi\mathbf{v}_i
the encoder output vector for token ii (for BERT, 768 numbers)
ηmean(t)\eta_{\text{mean}}(t)
the single text vector. It has the same number of dimensions as each vi\mathbf{v}_i.
Read it aloud Eta-mean of t equals one divided by n, multiplied by the sum, for i from 1 to n, of v-i.
Formula in plain words Add the token vectors, one dimension at a time. Then divide by the number of tokens. Each token has an equal part in the result. Thus, the result describes the full text and uses all positions. The division by nn keeps long texts and short texts on the same scale. Without it, long texts get larger vectors and larger dot products. The disadvantage: the other tokens make one very relevant token less strong.

Worked example 2Mean pooling

These are illustrative token vectors for a text with 3 tokens (3 dimensions, to make the example easy to read):

v1=[0.2,0.8,−0.4],v2=[0.6,0.0,0.2],v3=[0.4,0.4,0.8] \mathbf{v}_1 = [0.2, 0.8, -0.4],\quad \mathbf{v}_2 = [0.6, 0.0, 0.2],\quad \mathbf{v}_3 = [0.4, 0.4, 0.8]
1
Add the vectors, one coordinate at a time: v1+v2+v3=[0.2+0.6+0.4, 0.8+0.0+0.4, −0.4+0.2+0.8]=[1.2, 1.2, 0.6] \mathbf{v}_1+\mathbf{v}_2+\mathbf{v}_3 = [0.2+0.6+0.4,\ 0.8+0.0+0.4,\ -0.4+0.2+0.8] = [1.2,\ 1.2,\ 0.6]
2
Divide by the number of tokens (3): mean=13[1.2, 1.2, 0.6]=[0.4, 0.4, 0.2] \text{mean} = \frac{1}{3}[1.2,\ 1.2,\ 0.6] = [0.4,\ 0.4,\ 0.2]
ResultThe single vector [0.4, 0.4, 0.2] represents the full text. RepBERT and Contriever store this type of vector for each document.

4.2 Decoder-only (GPT-style): discard generation, keep the last-token vector

A decoder predicts the next token from the input. It uses causal attention: each token sees only the tokens before it. The decoder was made for generation. But its internal layers also learn good representations of the text. This is the method for retrieval:

input: what is dense retrieval [EOS] Transformer decoder (LLM) h(what) h(is) … h(last token) next-token prediction head discarded for retrieval text embedding stored in an ANN index
Decoder-based embedding: remove the next-token prediction part. Use the hidden state of the last token (usually an added [EOS] token) as the embedding of the full text. E5-Mistral and RepLLaMA use this method.
Key ideaWhy the last token? With causal attention, only the last token attends to all tokens before it. Thus, only its hidden state contains information about the full input. This makes it a good single-vector summary. For this reason, E5-Mistral and RepLLaMA add an [EOS] token to the end of the text. They use the vector of the last layer at that position.
PracticeWhy is mean pooling over the token vectors of a causal decoder a weaker summary than the last-token vector?
What does the first token see with causal attention?
AnswerWith causal attention, each token vector contains only information about the tokens to its left. Thus, the first tokens contain almost no information about the rest of the text. The average gives these partial vectors equal weight. The last token attends to all tokens. Thus, its vector is the only complete summary. (LLM2Vec in Section 13 first makes the attention bidirectional. For this reason, mean pooling works there.)
PracticeCompute the mean-pooled vector of the token vectors [1, 0], [0, 1] and [2, 2].
AnswerAdd the vectors. Then divide by the number of tokens: sum=[3, 3]mean=sum3=[1, 1] \begin{aligned} \text{sum} &= [3,\ 3] \\ \text{mean} &= \frac{\text{sum}}{3} = [1,\ 1] \end{aligned}

5. Bi-encoders vs late interaction

Sources: slides 12–13, 17 · Video 1 (Overview: embedding-based retrievers) · Video 2 (DPR) · In-class slides 12–13

5.1 Single-vector retrievers: the bi-encoder architecture

Single-vector models use a bi-encoder (also called dual encoder). One encoder is for the query. One encoder is for the document. The two encoders often share weights. Each encoder makes one vector. The score is one similarity calculation.

query q (online, at search time) document d (offline, pre-computed) Query encoder η Document encoder η η(q): 1 vector η(d): 1 vector φ = dot or cosine score(q, d): one O(1) step
The bi-encoder encodes the query and the document separately into single vectors. The two vectors meet only at the last, cheap similarity step. DPR uses [CLS] embeddings and a dot product, as slide 12 shows.
Key ideaWhy bi-encoders work at web scale. The bi-encoder encodes the two texts separately. Thus, the system computes all document embeddings offline. It stores them in a vector index (ANN: approximate nearest neighbour). At query time, the system encodes only the query (one transformer pass). Then it computes millions of cheap dot products with efficient vector operations. The expensive part does not occur at query time.

5.2 Multi-vector retrievers: late interaction

ColBERT keeps all token vectors of the query and of the document. The vectors interact only at scoring time. This is the reason for the name late interaction. This method finds more detailed matches at token level. It is often more effective. But scoring is more expensive (O(m×n)O(m \times n) instead of O(1)O(1)). Storage also increases from 1 vector to hundreds of vectors for each document. Section 11 shows the MaxSim scoring step by step.

query token vectors document token vectors E(q₁) E(q₂) E(q₃) E(d₁) E(d₂) E(d₃) E(d₄) E(d₅) score = sum of the highlighted maxima
Late interaction compares each query token with each document token (grey lines). It keeps the best match of each query token (green). Then it adds these maximum values.
PropertyBi-encoder (single vector)Late interaction (multi vector)
Vectors stored for each document1approximately 100–500 (one for each token)
Scoring for each documentO(1)O(1): one dot product or cosineO(m×n)O(m \times n): MaxSim over all token pairs
Match signalsummary of the full textmatches at token level
Usual trade-offfaster, less storageoften more effective, more computation and storage
PracticeWhy can a bi-encoder compute document vectors offline? What prevents this if one encoder reads the query and the document together (a cross-encoder)?
AnswerIn a bi-encoder, the document vector depends only on the document. Thus, the system computes it one time and uses it again for each query. If one encoder reads the query and the document together, each representation depends on the query. Then the system cannot compute anything before the query arrives. Each query needs one full transformer pass for each candidate document.

6. The five dimensions that differentiate models

Sources: slides 17–18 · Video 1 (Overview: embedding-based retrievers)

Each model of this week gives a different answer to the same five design questions. These questions help you to organise the week:

Design questionPossible answers in this week
1. Model backbone: encoder or decoder?Encoder (DPR, ANCE, RepBERT, Contriever, ColBERT) or decoder LLM (E5-Mistral, LLM2Vec, RepLLaMA)
2. Representation: how does the model turn a text into one or more vectors?[CLS] embedding · mean of token embeddings · all token embeddings · last-token/[EOS] embedding
3. Similarity function: how does the model compare vectors?Dot product · cosine similarity · late interaction (MaxSim)
4. Loss function: how does the model learn ranking during fine-tuning?Negative log likelihood (InfoNCE) · multi-label margin loss · SimCSE-style contrastive loss
5. Positive/negative sampling: where does the training signal come from?Random negatives · BM25 hard negatives · in-batch negatives · self-mined (ANCE checkpoint) negatives · spans of the same document (Contriever) · synthetic triples from an LLM (E5-Mistral)

To train these models, each training query needs a positive document (known to be relevant). It also needs a set of negative documents for comparison. The model learns to give the positive a higher score than the negatives. When it can do this, its embeddings are useful for dense retrieval.

query q positive p⁺ negatives p₁⁻ … pₙ⁻ Encoder η shared weights s⁺ = sim(q, p⁺) sj⁻ = sim(q, pj⁻) Loss InfoNCE or margin gradient update: raise s⁺, lower every sj⁻
All models of this week use this fine-tuning loop. They differ in the encoder, the similarity, the loss and the source of the negatives.
PracticeANCE and DPR have the same backbone, representation, similarity and loss. On which of the five dimensions are they different?
AnswerDimension 5, positive/negative sampling. DPR uses random, BM25 hard and in-batch negatives. ANCE finds hard negatives with its own recent checkpoints.

7. DPR: Dense Passage Retrieval

Sources: slides 19–25 · Video 2 (DPR) · In-class slides 10, 17–20

Karpukhin et al., "Dense Passage Retrieval for Open-Domain Question Answering", EMNLP 2020. DPR is the typical bi-encoder. It showed that a trained dense retriever can be equal to BM25 or better. These are its design choices:

Design choiceDPR
BackboneBERT encoder (bi-encoder: one query encoder and one passage encoder)
Representation[CLS] token embedding
SimilarityDot product (simple, faster than cosine)
LossNegative log likelihood (InfoNCE)
NegativesRandom + BM25 hard + in-batch

7.1 The InfoNCE loss

The loss has one goal. It makes the probability high that the model selects the relevant passage from a set of candidates. In-class slide 17 says that it "encourages the positive passage to receive the highest score among candidates".

L(q,p+,p1−,…,pn−)=−log⁡esim(q,p+)esim(q,p+)+∑j=1nesim(q,pj−) \mathcal{L}(q, p^+, p_1^-, \dots, p_n^-) = -\log \frac{e^{\mathrm{sim}(q,p^+)}}{e^{\mathrm{sim}(q,p^+)} + \sum_{j=1}^{n} e^{\mathrm{sim}(q,p_j^-)}}
Where
qq
the query embedding (DPR: the [CLS] vector of the query)
p+p^+
the positive (relevant) passage embedding
pj−p_j^-
negative (irrelevant) passage embedding number jj
nn
the number of negatives for this query
sim(q,p)=q⋅p\mathrm{sim}(q,p) = q \cdot p
the dot-product similarity
exe^{x}
the exponential function. It makes all terms positive.
−log⁡-\log
the negative natural logarithm. It changes a probability into a loss.
L\mathcal{L}
the loss for one training query. Training makes its average as small as possible.
Read it aloud The loss for query q, positive p-plus and negatives p-one-minus to p-n-minus equals minus the log of a fraction. The top of the fraction is e to the similarity of q and p-plus. The bottom is e to the similarity of q and p-plus, plus the sum, for j from 1 to n, of e to the similarity of q and p-j-minus.
Formula in plain words The top of the fraction is the "score mass" of the correct passage. The bottom adds this mass to the mass of all negatives. Thus, the fraction is a softmax. It is the probability that the model selects the correct passage from all candidates. The exponential makes differences larger. A positive at 0.8 and a negative at 0.2 have a difference of 0.6. But e0.8≈2.23e^{0.8} \approx 2.23 and e0.2≈1.22e^{0.2} \approx 1.22 make the preference stronger. The −log⁡-\log changes the probability into a loss. −log⁡(1)=0-\log(1) = 0: full confidence gives no loss. −log⁡(0.5)≈0.69-\log(0.5) \approx 0.69. The loss increases without limit when the probability of the correct passage goes to 0. A higher sim(q,p+)\mathrm{sim}(q,p^+) or a lower negative similarity decreases the loss. Thus, training pulls the query and the relevant passage together. It pushes irrelevant passages away. More negatives, or harder negatives, make the bottom larger and the task harder.

Worked example 3Computing the DPR InfoNCE loss

These are illustrative dot-product similarities for one query with one positive and two negatives:

sim(q,p+)=1.2,sim(q,p1−)=0.4,sim(q,p2−)=−0.3 \mathrm{sim}(q,p^+) = 1.2,\quad \mathrm{sim}(q,p_1^-) = 0.4,\quad \mathrm{sim}(q,p_2^-) = -0.3
1
Calculate the exponential of each similarity. This makes all terms positive and makes the differences larger. e1.2≈3.3201e0.4≈1.4918e−0.3≈0.7408 \begin{aligned} e^{1.2} &\approx 3.3201 \\ e^{0.4} &\approx 1.4918 \\ e^{-0.3} &\approx 0.7408 \end{aligned}
2
Calculate the softmax probability of the positive: the positive mass divided by the total mass. P(p+∣q)=3.32013.3201+1.4918+0.7408=3.32015.5527≈0.598 P(p^+ \mid q) = \frac{3.3201}{3.3201 + 1.4918 + 0.7408} = \frac{3.3201}{5.5527} \approx 0.598
3
Calculate the negative log: L=−log⁡(0.598)≈0.514 \mathcal{L} = -\log(0.598) \approx 0.514
4
Check the behaviour. Training increases sim(q,p+)\mathrm{sim}(q,p^+) to 3.0. The negatives stay the same. Then: P≈e3e3+1.4918+0.7408≈0.90L≈0.105 \begin{aligned} P &\approx \frac{e^{3}}{e^{3} + 1.4918 + 0.7408} \approx 0.90 \\ \mathcal{L} &\approx 0.105 \end{aligned} The model is more confident, and the loss is much lower.
ResultL≈0.514\mathcal{L} \approx 0.514. The model is only approximately 60% "sure" of the correct passage. Thus, the gradient is still large. It pushes η(q)\eta(q) towards η(p+)\eta(p^+) and away from the negatives.

7.2 The three-part negative sampling strategy

The success of DPR depends much on the negatives that it uses for comparison. DPR uses three sources. Each source is more difficult than the one before it:

whole corpus BM25 top-1000 q p⁺ Three kinds of negatives Random (easy) sampled anywhere in the corpus, e.g. a cooking recipe for an ML query BM25 hard top-ranked by BM25 but not relevant: shares keywords, wrong meaning In-batch (medium) other queries' positives in the batch: already encoded, so free DPR in practice: combine all three.
DPR negative sampling (slides 22–25). Easy negatives are random documents from the corpus. Hard negatives come from the top BM25 results: lexically similar, but with the wrong meaning. In-batch negatives are the positives of the other queries. The system uses them again at no cost.
  1. Random negatives (easy). Select passages at random from the full corpus. For a query about machine learning, a random negative can be a cooking recipe. These negatives help with basic discrimination. But the model quickly finds them too easy. Then it stops learning small differences. Real search has much harder cases.
  2. BM25 hard negatives. Get the BM25 top-kk for the query (for example, the top 1000). Remove all passages with a relevant label. Select negatives from the remaining set. These passages share keywords with the query, but their meaning is not relevant. Example: the query "how does machine learning work" and the passage "machine learning applications in healthcare". These negatives teach the model the difference between lexical similarity and true semantic relevance.
  3. In-batch negatives (medium, free). A training batch has queries q1,q2,q3q_1, q_2, q_3 and positives p1+,p2+,p3+p_1^+, p_2^+, p_3^+. For q1q_1, use p2+p_2^+ and p3+p_3^+ as negatives. The system already encoded these passages for the other queries. Thus, they cost nothing more. They come from the same dataset and domain. Thus, they are harder than random negatives: "medium" difficulty. Slide 25 says that they give efficiency and a gain for the domain.
p₁⁺ p₂⁺ p₃⁺ p₄⁺ q₁ q₂ q₃ q₄ +−−− −+−− −−+− −−−+ each row qᵢ: 1 positive (+) B − 1 in-batch negatives (−) one softmax (InfoNCE) per row B queries × B passages: B² scores from only 2B encodings
In-batch negatives with B = 4. The diagonal holds the positive of each query. The other cells of each row are free negatives.

Worked example 4Counting negatives in a DPR training batch

A training batch has B=8B = 8 queries. Each query has 1 positive passage with a label and 1 BM25 hard negative. The batch shares all passages as in-batch negatives. How many negatives does each query get? How many passage encodings does the batch need?

1
Passages that the batch encodes: 8 positives + 8 BM25 hard negatives = 16 passage encodings (and 8 query encodings).
2
Negatives for one query, for example q1q_1: the positives of the other queries (8−1=78 - 1 = 7), and all 8 hard negatives in the batch (its own and those of the other queries) = 7+8=157 + 8 = 15 negatives.
3
Compare with a simple method: to give each of the 8 queries its own 15 negatives, the batch needs 8×15=1208 \times 15 = 120 negative encodings. With in-batch negatives, it needs only 16 passage encodings in total. Each query gets 15 negatives for the cost of 2 passages.
ResultEach query trains against 15 negatives. The batch encodes only 16 passages. This efficiency is the reason why almost all models of this week use in-batch negatives.
PracticeCompute the InfoNCE loss for sim(q, p⁺) = 2.0 and two negatives with similarities 1.0 and 0.0.
Use the method of Worked example 3. Calculate the exponentials, then the softmax probability of the positive, then −log.
AnswerCalculate the exponentials, then the probability of the positive, then the negative log: e2≈7.389e1≈2.718e0=1P=7.38911.107≈0.665L=−log⁡(0.665)≈0.408 \begin{aligned} e^{2} &\approx 7.389 \\ e^{1} &\approx 2.718 \\ e^{0} &= 1 \\ P &= \frac{7.389}{11.107} \approx 0.665 \\ \mathcal{L} &= -\log(0.665) \approx 0.408 \end{aligned}
PracticeWhy are in-batch negatives "medium" difficulty? Why are they harder than random negatives and easier than BM25 hard negatives?
AnswerThey are real positives of other training queries. They come from the same dataset and domain as the true positive. Thus, they look like possible answers, and they are harder than random text. But the dataset selected them for a different query. Thus, they seldom share the specific keywords of this query. For this reason, they are easier than the top lexical matches of BM25.

8. RepBERT: mean pooling and margin loss

Sources: slides 26–27 · Video 3 (RepBERT) · In-class slides 10, 17

Zhan et al., "RepBERT: Contextualized Text Embeddings for First-Stage Retrieval", arXiv 2020. RepBERT is different from DPR in two ways. It pools the token vectors differently. It also uses a different training loss.

Design choiceRepBERT
BackboneBERT encoder
RepresentationMean of all token embeddings (Section 4.1)
SimilarityInner (dot) product
LossMulti-label margin loss
NegativesIn-batch (for efficiency)

8.1 Multi-label margin loss

In-class slide 17 gives the aim. Relevant documents must get a higher score than irrelevant documents, by a minimum fixed margin (here 1).

L=1∣D+∣ ∣D−∣∑di+∈D+∑dj−∈D−max⁡ ⁣(0, 1−(Rel(q,di+)−Rel(q,dj−))) \mathcal{L} = \frac{1}{|D^+|\,|D^-|} \sum_{d_i^+ \in D^+} \sum_{d_j^- \in D^-} \max\!\Big(0,\ 1 - \big(\mathrm{Rel}(q,d_i^+) - \mathrm{Rel}(q,d_j^-)\big)\Big)
Where
qq
the query
D+, D−D^+,\ D^-
the positive (relevant) and negative (irrelevant) documents for qq in the batch
∣D+∣ ∣D−∣|D^+|\,|D^-|
the number of positive–negative pairs. The formula divides by it to get the average.
di+, dj−d_i^+,\ d_j^-
one positive document and one negative document
Rel(q,d)\mathrm{Rel}(q,d)
the relevance score: the dot product of the mean-pooled query vector and document vector
11
the margin: the minimum gap between a positive score and a negative score
max⁡(0,⋅)\max(0,\cdot)
sets the loss of each pair to zero when it is below zero (a hinge)
Read it aloud The loss equals one divided by the number of positives multiplied by the number of negatives. This is multiplied by the double sum, over each positive d-i-plus and each negative d-j-minus, of the maximum of zero and one minus the difference between Rel of q and d-i-plus and Rel of q and d-j-minus.
Formula in plain words Look at each pair of one positive and one negative document. Measure how much higher the positive score is. Subtract this gap from 1. If the result is below zero, use zero. If the positive score is already 1 full point higher than the negative score, the loss of the pair is zero. The pair is satisfied. If the gap is less than 1, or if the negative score is higher, the pair gives a loss. This loss is the amount that the pair is short of the margin. Thus, pairs in a very wrong order cost more. Then calculate the average over all positive–negative pairs. A larger margin needs a larger gap before a pair gives zero loss.
Watch outSlide 27 writes the normaliser as 1n\frac{1}{n}, with the index range 1≤i≤m, m<j≤n1 \le i \le m,\ m < j \le n. The first mm documents are positives. The remaining documents are negatives. The solution to in-class Exercise 4 divides by the number of positive–negative pairs. The formula above uses this form. Set each pair to zero if it is below zero before you calculate the average.
−1 0 1 2 0 1 2 loss score gap Δ = Rel(q, d⁺) − Rel(q, d⁻) margin = 1 Case B: Δ = −0.6, loss 1.6 Case A: Δ = 0.7, loss 0.3 Δ = 1.3, loss 0
The margin (hinge) loss of one positive–negative pair decreases when the lead of the positive increases. It becomes zero when the lead reaches the margin of 1.
Key ideaMargin loss and InfoNCE use different approaches. InfoNCE is probabilistic. It pushes the softmax probability of the positive towards 1. Thus, each score always gives a small gradient. Margin loss uses a threshold. It sets a minimum gap (margin = 1) between relevant and irrelevant scores. When a pair has this gap, the loss ignores the pair. This behaviour ("stop when the result is good enough") can make training more stable in some conditions.

Worked example 5Margin loss on the two lecture cases

The margin is 1. Compute the loss of one positive–negative pair in each case (values from slide 27).

1
Case A: the model is almost correct. Rel(q,d+)=0.8\mathrm{Rel}(q,d^+) = 0.8, Rel(q,d−)=0.1\mathrm{Rel}(q,d^-) = 0.1. The difference is 0.8−0.1=0.70.8 - 0.1 = 0.7. This is less than the margin of 1. Thus: LA=max⁡(0, 1−0.7)=max⁡(0, 0.3)=0.3 \mathcal{L}_A = \max(0,\ 1 - 0.7) = \max(0,\ 0.3) = 0.3 The loss is small. The pair is near the target gap. It needs only a small push.
2
Case B: the model has the wrong order. Rel(q,d+)=0.2\mathrm{Rel}(q,d^+) = 0.2, Rel(q,d−)=0.8\mathrm{Rel}(q,d^-) = 0.8. The negative has a higher score than the positive. The difference is 0.2−0.8=−0.60.2 - 0.8 = -0.6. Thus: LB=max⁡(0, 1−(−0.6))=max⁡(0, 1.6)=1.6 \mathcal{L}_B = \max(0,\ 1 - (-0.6)) = \max(0,\ 1.6) = 1.6 The loss is much larger. The ranking error gets a strong correction.
3
Check the satisfied case. Rel(q,d+)=1.5\mathrm{Rel}(q,d^+) = 1.5 and Rel(q,d−)=0.2\mathrm{Rel}(q,d^-) = 0.2. The difference is 1.3≥11.3 \ge 1. Thus, L=max⁡(0, 1−1.3)=max⁡(0, −0.3)=0\mathcal{L} = \max(0,\ 1 - 1.3) = \max(0,\ -0.3) = 0. The pair gives no loss. Training works on the pairs that are still below the margin.
ResultL=0.3\mathcal{L} = 0.3 for the almost correct case, 1.61.6 for the wrong-order case, and 00 when the pair reaches the margin. The loss increases when the order of a pair is worse. A ranking loss must behave in this way.
PracticeA RepBERT pair has Rel(q, d⁺) = 0.9 and Rel(q, d⁻) = 0.4. What is the margin loss with margin 1?
AnswerThe gap is 0.5. Thus, the loss is max⁡(0, 1−0.5)=0.5\max(0,\ 1 - 0.5) = 0.5.
PracticeA query has one positive (score 1.0) and two negatives (scores 0.5 and −0.5). A student adds the raw values 1 − gap for the two pairs. The student gets 0.5 + (−0.5) = 0. What is the error?
AnswerThe student did not set the negative value to zero. The calculation for each pair and for the average loss is: first pair:max⁡(0,1−0.5)=0.5second pair:max⁡(0,1−1.5)=0average:L=0.52=0.25 \begin{aligned} \text{first pair:}\quad \max(0, 1 - 0.5) &= 0.5 \\ \text{second pair:}\quad \max(0, 1 - 1.5) &= 0 \\ \text{average:}\quad \mathcal{L} &= \frac{0.5}{2} = 0.25 \end{aligned} A satisfied pair gives exactly 0. It cannot cancel the loss of a different pair.

9. ANCE: self-mined hard negatives

Sources: slides 28–30 · Video 4 (ANCE) · In-class slide 11 · In-class slide 25 (Exercise 5)

Xiong et al., "Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval", ICLR 2021. ANCE keeps the representation, similarity and loss of DPR. Its only new idea is the source of the negatives.

Design choiceANCE
BackboneBERT encoder
Representation[CLS] token embedding
SimilarityInner (dot) product
LossNegative log likelihood (InfoNCE, Section 7.1)
NegativesSelf-mined hard negatives from regular checkpoints (Negative Contrastive Estimation)

9.1 The problem with BM25 negatives

The hard negatives of DPR are hard for BM25. But the model in training is a dense retriever. A document that is difficult for BM25 can be easy in the dense embedding space. Slide 29 shows the embeddings on a 2-D map:

dense embedding space (2-D sketch) q query vector relevant documents: near q BM25 negatives: mostly far easy for the dense model random negatives: far, mixed ANCE negatives: ranked high by the dense model itself: close to q, yet irrelevant Hard for this model ≠ hard for BM25
In the dense space, most BM25 negatives and random negatives are far from the query. The dense retriever separates them easily and learns little. The useful negatives (blue) are the irrelevant documents that the dense model itself gives a high rank.

9.2 Mining negatives from checkpoints

How can you get "negatives that confuse the current model" during training? ANCE uses a recent checkpoint of the model itself (slide 30).

1. Train the model about every 1,000 iterations 2. Save a checkpoint frozen copy of the weights 3. Retrieve with the checkpoint re-encode corpus, ANN search 4. Keep top-ranked, non-relevant = new hard negatives feed back as negatives repeat throughout training
The asynchronous loop of ANCE: save a checkpoint, retrieve with it, use high-ranked but irrelevant documents as negatives, continue training, and do it again.
  1. Save a checkpoint of the model regularly during training (for example, every thousand iterations).
  2. Use the previous checkpoint to embed the corpus. Retrieve documents for the training queries with approximate nearest-neighbour search ("ANN" in the name).
  3. From the results, take the documents that have a high rank but are not relevant. These documents are the errors of the current model. Use them as negatives.
  4. Do this again during all of training. Refresh the negatives when the model improves.
Watch outTrade-off. Tests show that this method gives better dense retrievers than the in-batch and BM25 negatives of DPR. But training costs much more. The system must compute the corpus representations again from each new checkpoint. The query-time cost stays the same, because the result is still a plain bi-encoder.
PracticeWhy are BM25 hard negatives "hard" for a lexical reason but possibly "easy" for a dense retriever? How does ANCE correct this?
AnswerBM25 negatives share keywords with the query. This makes them hard for a model that matches terms. A dense retriever compares meanings. In its embedding space, many of these documents are already far from the query. Thus, they teach it little. ANCE finds negatives with a recent checkpoint of the dense model itself. It uses the irrelevant documents that this model gives a high rank. It refreshes them during training.
PracticeDoes ANCE make retrieval at query time slower than DPR? Explain.
AnswerNo. ANCE changes only how training selects negatives. The trained model is still a bi-encoder with one [CLS] vector for each text and a dot product. Thus, the query-time cost is the same as DPR. The extra cost (to encode the corpus again at each checkpoint) occurs during training.

10. Contriever: unsupervised dense retrieval

Sources: slides 31–32 · Video 5 (Contriever) · In-class slide 11

Izacard et al., "Unsupervised Dense Information Retrieval with Contrastive Learning", 2021.

Design choiceContriever
BackboneBERT encoder
RepresentationMean of token embeddings
SimilarityCosine similarity
LossNegative log likelihood (InfoNCE, the same loss as DPR and ANCE)
New ideaUnsupervised training: no labelled data necessary

DPR and ANCE need relevance judgments from human annotators. The main idea of Contriever is to make its own training signal from a raw text collection:

Key ideaThe unsupervised method of Contriever. You have a document collection D={d1,d2,…,dn}D = \{d_1, d_2, \dots, d_n\}. For each document did_i: Then train with the usual InfoNCE loss. One span is the "query". The other span from the same document is the positive. Spans from other documents are the negatives.
positive pair document dᵢ span s¹ᵢ span s²ᵢ negative pair another document dj (j ≠ i) span sᵏj span length: 5%–50% of the document (2–20 tokens for a 40-token document)
Contriever makes its own training pairs. Two spans of one document are a positive pair. A span from a different document gives a negative pair.

Worked example 6Building Contriever training pairs from raw text

Use a document of 40 tokens: "The Great Barrier Reef stretches over 2,300 km off the coast of Queensland. It is the world's largest coral reef system, composed of nearly 3,000 individual reefs. Rising sea temperatures have caused repeated mass bleaching events in recent years."

1
Permitted span lengths: between 5% and 50% of the document length: shortest:0.05×40=2 tokenslongest:0.50×40=20 tokens \begin{aligned} \text{shortest:}\quad 0.05 \times 40 &= 2 \text{ tokens} \\ \text{longest:}\quad 0.50 \times 40 &= 20 \text{ tokens} \end{aligned}
2
Select two spans from this document. They are random continuous sequences in the permitted length range. s1s_1 = "stretches over 2,300 km off the coast of Queensland" (9 tokens). s2s_2 = "repeated mass bleaching events" (4 tokens). These spans are a positive pair. They come from the same document. Thus, the method assumes that they are related (both are about the Reef).
3
Negatives: select a span from a different document. For example, sjks_j^k = "the quarterly earnings exceeded analyst expectations" from a finance article. The pair (s1,sjk)(s_1, s_j^k) is a negative pair.
4
Train: use s1s_1 as the query and s2s_2 as the positive. Use the spans of other documents as negatives (in practice, the other spans in the batch). Minimise InfoNCE with cosine similarity on mean-pooled embeddings.
ResultThe method makes an unlimited number of (query, positive, negatives) triples from any text collection. The full process needs no human relevance labels.
PracticeA document has 200 tokens. Which span lengths can Contriever select? Why can two spans from the same document sometimes be a bad positive pair?
AnswerSpans can have these lengths: shortest:0.05×200=10 tokenslongest:0.5×200=100 tokens \begin{aligned} \text{shortest:}\quad 0.05 \times 200 &= 10 \text{ tokens} \\ \text{longest:}\quad 0.5 \times 200 &= 100 \text{ tokens} \end{aligned} The method assumes that spans of one document are related. A long document can change topic (for example, a news summary). Then the two spans can be about unrelated subjects. This gives a noisy positive.

11. ColBERT: late interaction with MaxSim

Sources: slides 33–40 · Video 6 (ColBERT) · In-class slides 12–13

Khattab and Zaharia, "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT", SIGIR 2020. ColBERT is the only multi-vector model of this week. It is the best-known model of the late-interaction family.

Design choiceColBERT
BackboneBERT encoder (query encoder and document encoder)
RepresentationAll token embeddings (L2-normalised), for each query and each document
SimilarityMaxSim (late interaction)
LossNegative log likelihood (the same family as DPR, ANCE and Contriever)

11.1 The MaxSim scoring function, step by step

score(q,d)=∑i=1mmax⁡j=1,…,nEqi⋅Edj \mathrm{score}(q,d) = \sum_{i=1}^{m} \max_{j = 1,\dots,n} E_{q_i} \cdot E_{d_j}
Where
mm
the number of query tokens
nn
the number of document tokens
EqiE_{q_i}
the L2-normalised embedding of query token ii
EdjE_{d_j}
the L2-normalised embedding of document token jj
Eqi⋅EdjE_{q_i} \cdot E_{d_j}
their dot product. It is a cosine, because both vectors have length 1. In-class slide 13 writes it as EqiEdjTE_{q_i}E_{d_j}^{T}.
max⁡j\max_{j}
the best match for query token ii in the full document
Read it aloud The score of d for q is the sum, for i from 1 to m, of the maximum over j from 1 to n of E-q-i dot E-d-j.
Formula in plain words For each query token, compute its similarity with each document token. Keep only the maximum. This is the best match of that query token in the full document. Then add these maximum values. Thus, each query token can find its own match at any position. A new document token with a strong match increases the score only if it is better than the previous best match. Additional weak tokens in a long document have no effect, because the max ignores them. Longer queries have more terms in the sum. Thus, compare scores only for the same query.

The lecture shows the method in five steps for query tokens tq1,…,tqmt_{q_1}, \dots, t_{q_m} and document tokens td1,…,tdnt_{d_1}, \dots, t_{d_n} (slides 34–38):

  1. Step 1: take query token tq1t_{q_1}. Compute the dot product of its embedding with the first document token embedding td1t_{d_1}. This gives one similarity score.
  2. Step 2: for the same tq1t_{q_1}, compute its similarity with each other document token td2,td3,…,tdnt_{d_2}, t_{d_3}, \dots, t_{d_n}. This gives a full list of scores for tq1t_{q_1}.
  3. Step 3: from the list, keep the maximum. This is the best match of tq1t_{q_1} in the full document. A strong match with any part of the document is sufficient. The method does not calculate an average.
  4. Step 4: do steps 1–3 again for each remaining query token tq2,…,tqmt_{q_2}, \dots, t_{q_m}. This gives one maximum for each query token.
  5. Step 5: add the maximum values. Each query token adds its best match score to the total.
similarity matrix S: query tokens × document tokens d₁ d₂ d₃ d₄ max per row q₁ q₂ q₃ 0.90.20.40.1 0.30.80.50.2 0.10.40.20.6 0.90.80.6 Σ = 2.3 score(q, d) each row keeps only its best match (max), then the row maxima are summed
MaxSim on an example with 3 query tokens and 4 document tokens. Take the maximum of each row. Then add the maximum values. Worked example 7 uses the same numbers.

Worked example 7ColBERT MaxSim by hand

These values are illustrative. A query has m=3m = 3 token embeddings. A document has n=4n = 4. All token vectors are L2-normalised. Thus, each dot product is a cosine similarity. The pair similarities Sij=Eqi⋅EdjS_{ij} = E_{q_i} \cdot E_{d_j} are:

S=(0.90.20.40.10.30.80.50.20.10.40.20.6) S = \begin{pmatrix} 0.9 & 0.2 & 0.4 & 0.1 \\ 0.3 & 0.8 & 0.5 & 0.2 \\ 0.1 & 0.4 & 0.2 & 0.6 \end{pmatrix}
1
Query token 1 (row 1): the scores against the 4 document tokens are 0.9, 0.2, 0.4, 0.1. The best match in the document: max⁡(0.9,0.2,0.4,0.1)=0.9\max(0.9, 0.2, 0.4, 0.1) = 0.9.
2
Query token 2 (row 2): max⁡(0.3,0.8,0.5,0.2)=0.8\max(0.3, 0.8, 0.5, 0.2) = 0.8. The best match is document token 2.
3
Query token 3 (row 3): max⁡(0.1,0.4,0.2,0.6)=0.6\max(0.1, 0.4, 0.2, 0.6) = 0.6. The best match is document token 4.
4
Add the maximum values: score(q,d)=0.9+0.8+0.6=2.3 \mathrm{score}(q,d) = 0.9 + 0.8 + 0.6 = 2.3
5
Count the work: this needs m×n=3×4=12m \times n = 3 \times 4 = 12 dot products for one document. A bi-encoder such as DPR needs exactly 1. For a real query of 32 tokens and a passage of 300 tokens, ColBERT needs 9,600 dot products for each document.
Resultscore(q,d)=2.3\mathrm{score}(q,d) = 2.3. Each query token found its best match at a different position in the document. Single-vector models lose this detailed token-level signal.

11.2 Why it works, and what it costs

Key ideaColBERT computes detailed interactions between all pairs of query tokens and document tokens. MaxSim combines them. The interaction stays late. The system computes and stores the document token embeddings offline. The query and the document meet only at the cheap dot-product step. ColBERT keeps more information. Thus, it is often more effective than single-vector retrievers.
Watch outThe problems of ColBERT (slide 40).

Worked example 8Index size: bi-encoder against ColBERT

Illustrative setting: 1,000,000 passages. Each vector has 768 dimensions, stored as 4-byte floats. ColBERT keeps 200 token vectors for each passage (in the 100–500 range of slide 40). Real systems make token vectors smaller and compress them. The lecture does not cover this. The important result here is the ratio.

1
One vector: 768×4=3,072768 \times 4 = 3{,}072 bytes.
2
Bi-encoder index: 1,000,000×3,072≈3.11{,}000{,}000 \times 3{,}072 \approx 3.1 GB.
3
ColBERT index: 1,000,000×200×3,072≈6141{,}000{,}000 \times 200 \times 3{,}072 \approx 614 GB.
4
Score one passage for a query of 32 tokens: the bi-encoder needs 1 dot product. ColBERT needs 32×200=6,40032 \times 200 = 6{,}400 dot products.
ResultIn this setting, ColBERT needs approximately 200 times the storage (614 GB against 3.1 GB). It also needs 6,400 times the similarity operations for each passage. This is the cost of its better effectiveness.
PracticeWhy does ColBERT use the maximum over document tokens instead of the mean? What does an average do to a long document with one very relevant sentence?
AnswerThe maximum gives a query token credit for one strong match in any part of the document. Relevance needs this. An average mixes the strong match with the many unrelated tokens of a long document. Thus, a long document with one very relevant sentence gets a low score. The average penalises the document for its length.
PracticeThe row of one query token in the similarity matrix is 0.2, 0.7, 0.7, 0.1. What does this token add to the MaxSim score? Is it important that the maximum occurs two times?
AnswerIt adds 0.7. MaxSim takes the maximum one time for each query token. Thus, a second match with the same value adds nothing.

12. E5-Mistral: LLM embeddings and synthetic data

Sources: slides 41–43 · Video 7 (E5-Mistral) · In-class slide 16

Wang et al., "Improving Text Embeddings with Large Language Models", 2023 (arXiv 2401.00368). E5 is a family of embedding models. The original E5 embeddings use BERT. E5-Mistral replaces the encoder backbone with a decoder-only LLM (Mistral-7B). It uses LLMs in two different ways.

12.1 Use 1: synthetic training data generated by an LLM

  1. Brainstorm tasks: tell an LLM (GPT-4) to make a list of possible retrieval tasks. Examples: "search web pages about a topic", "find academic references for an assignment", "find a tutorial video".
  2. Generate triples: for each task, tell the LLM to generate a ⟨query, positive document, hard-negative document⟩ triple. The prompt gives format instructions from the task definition. Do this many times. The result is a large synthetic training set without human annotation.
1. Brainstorm GPT-4 lists retrieval tasks 2. Generate ⟨q, p⁺, hard p⁻⟩ for each task Synthetic data no human annotation Fine-tune + MS MARCO labels contrastive loss
Use 1: an LLM writes the training data (slide 41, part A).

12.2 Use 2: the LLM as the embedding backbone

"Instruct: {task} Query: {q}" + [EOS] Mistral-7B decoder-only LLM last hidden state of [EOS] dense vector → ANN index documents take the same path without the instruction prefix: document text + [EOS] → embedding of [EOS]
The embedding path of E5-Mistral. A query with an instruction (or a plain document) and an added [EOS] token goes through the LLM. The [EOS] vector of the last layer is the embedding (slide 42).
  1. Change the query to include an instruction. The instruction is the task definition from the data generation step: q_inst = "Instruct: {task_definition} \n Query: {q}"
  2. Add [EOS] to the end of the query and of the document. Put each text into the LLM. Use the [EOS] vector of the last layer as the embedding. With causal attention, the [EOS] representation depends on all tokens before it. Thus, it describes the full input (Section 4.2).
  3. Train with the standard contrastive (InfoNCE) loss. Use in-batch negatives and the generated hard negatives.

12.3 Training stages and the key observation

Key ideaKey observation. BERT-based backbones need the contrastive pre-training stage. For E5-Mistral, this stage has almost no effect on quality. The LLM had a long auto-regressive pre-training. This training already gave it good text representations. Only fine-tuning is necessary. Fine-tuning adds mainly task knowledge (how to embed for retrieval). The LLM already has the text knowledge.
PracticeGive the two different ways in which E5-Mistral uses LLMs. Which one is still necessary if you have much human-labelled training data?
Answer(A) An LLM (GPT-4) generates synthetic training triples. (B) An LLM (Mistral-7B) is the embedding backbone. With much labelled data, use A becomes optional (RepLLaMA shows this). Use B is the model itself. It is always necessary.
PracticeWhy does E5-Mistral add [EOS] to queries and to documents?
AnswerWith causal attention, only the last position attends to the full input. The added [EOS] gives a fixed last position. Its last-layer vector describes the full text. The same method for queries and documents keeps them in the same representation space.

13. LLM2Vec: turning any decoder into an encoder

Sources: slides 44–46 · Video 8 (LLM2Vec) · In-class slide 16

BehnamGhader et al., "LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders", 2024. LLM2Vec is a general method. It changes any decoder-only LLM into a strong text encoder in three steps. A fourth step is optional:

Step 1 bidirectional attention Step 2 masked next-token prediction (MNTP) Step 3 unsupervised SimCSE Step 4 (optional) supervised contrastive result: a bidirectional text encoder built from a decoder-only LLM
The LLM2Vec method as a pipeline (slides 44–46).
  1. Enable bidirectional attention. Replace the causal attention mask with a matrix of ones. Then each token can attend to all other tokens. The decoder becomes bidirectional, like an encoder. But this step alone decreases the embedding performance. The weights of the model never learned to attend to future tokens.
  2. Masked next-token prediction (MNTP). This step teaches the model to use its new bidirectional attention. Mask one token in the sequence. Train the model to predict it. The special rule: to predict the masked token at position ii, compute the loss from the logits at position i−1i - 1. This agrees with the next-token habit of the decoder. Slide 45 gives the reason for this step. Decoder pre-training has no next-sentence-prediction objective. Thus, the model never learned to capture the context of the full sequence.
  3. Unsupervised contrastive learning (SimCSE). Put the same input sequence into the model two times, with two different random dropout masks. The dropout makes the two outputs a little different. The result is two "views" of one sentence. Train the model to make the similarity of the two views high. Train it to make the similarity to the other sequences in the batch low. This is an InfoNCE-type loss (Section 7.1). No labels are necessary.
  4. (Optional) Supervised contrastive learning on labelled retrieval data or sentence-similarity data. This gives more improvement.
causal mask (before) all-ones mask (after) thecatsatdown thecatsatdown thecatsatdown thecatsatdown 1000 1100 1110 1111 1111 1111 1111 1111 step 1 row i sees columns ≤ i (10 ones) every token sees every token (16 ones) Step 1 alone lowers quality: the weights never learned to use future tokens
LLM2Vec step 1 changes the causal mask (row = the token that attends) to all ones. Step 2 (MNTP) then teaches the weights to use the new connections.

Worked example 9What changes when the mask flips, and how MNTP works

Use the 4-token sequence the cat sat down.

1
Causal attention (original decoder). Row ii of the mask shows what token ii can see. Token 2 ("cat") sees only ⟨"the", "cat"⟩. Thus, its hidden vector has no information about "sat" or "down". This is the mask matrix (1 = can attend): Mcausal=(1000110011101111) M_{\text{causal}} = \begin{pmatrix} 1 & 0 & 0 & 0 \\ 1 & 1 & 0 & 0 \\ 1 & 1 & 1 & 0 \\ 1 & 1 & 1 & 1 \end{pmatrix}
2
Change to bidirectional (LLM2Vec step 1). The mask becomes a matrix of ones. It has 16 ones instead of 10. Now the vector of "cat" can also contain its future context: Mbi=(1111111111111111) M_{\text{bi}} = \begin{pmatrix} 1 & 1 & 1 & 1 \\ 1 & 1 & 1 & 1 \\ 1 & 1 & 1 & 1 \\ 1 & 1 & 1 & 1 \end{pmatrix} The pretrained weights never used these new connections. Thus, quality decreases at first.
3
MNTP (step 2). Mask position 3: the cat [MASK] down. The model must predict "sat". The loss comes from the logits at position 3−1=23 - 1 = 2 (the representation of "cat"). To solve this task, the weights must use attention to future tokens ("down" is to the right of the mask). This makes the bidirectional attention useful.
4
SimCSE (step 3). Encode the cat sat down two times with different dropout. This gives vectors zz and z′z'. Train with a contrastive loss. The loss makes ϕ(z,z′)\phi(z, z') high. It makes ϕ(z,zother)\phi(z, z_{\text{other}}) low for each other sentence in the batch. The model learns sentence embeddings with no labels.
ResultAfter the three steps, an LLM that only generated text operates as a strong bidirectional text encoder. The paper title calls it a "secretly powerful text encoder".
PracticeWhy does bidirectional attention alone (LLM2Vec step 1) decrease performance? Which step corrects this?
AnswerThe decoder weights learned only with causal attention. They never learned to use information from future tokens. The new connections give the layers inputs that they never saw in training. Step 2, masked next-token prediction, changes the weights. Then the model learns to use its new bidirectional attention.
PracticeA sequence has 6 tokens. How many ones are in the causal mask? How many are in the mask of ones?
Answercausal:1+2+⋯+6=21mask of ones:6×6=36 \begin{aligned} \text{causal:}\quad 1 + 2 + \dots + 6 &= 21 \\ \text{mask of ones:}\quad 6 \times 6 &= 36 \end{aligned}

14. RepLLaMA: supervised LLM retrieval with LoRA

Sources: slides 47–48 · Video 9 (RepLLaMA) · In-class slide 16

Ma et al., "Fine-Tuning LLaMA for Multi-Stage Text Retrieval", 2023. RepLLaMA uses the same architecture as E5-Mistral: a decoder-only backbone, a query with an instruction, an added [EOS], and the last hidden state of [EOS] as the embedding. Its backbone is Llama-2-7B.

Design choiceRepLLaMA
BackboneLlama-2-7B (decoder-only, open source)
Representation[EOS] token embedding (last hidden state)
SimilarityDot product
LossNegative log likelihood / InfoNCE (the same as DPR and ANCE)
TrainingOnly supervised labels (as DPR) + LoRA adapters

Two practical results make it different:

The lecture gives the name LoRA without the maths. For background, this is the standard LoRA update for one weight matrix:

W=W0+BA,B∈Rd×r,A∈Rr×k,r≪min⁡(d,k) W = W_0 + BA,\qquad B \in \mathbb{R}^{d \times r},\quad A \in \mathbb{R}^{r \times k},\quad r \ll \min(d,k)
Where
W0W_0
the pretrained d×kd \times k weight matrix of one projection layer. It stays frozen.
A, BA,\ B
the two small adapter matrices that training changes
rr
the rank of the adapter (for example, 8 or 16). It is much smaller than dd and kk.
BABA
their product: a d×kd \times k low-rank correction that the model adds to W0W_0
WW
the weight that the forward pass uses
Read it aloud W equals W-zero plus B multiplied by A. B is a d by r matrix. A is an r by k matrix. r is much smaller than the smaller of d and k.
Formula in plain words Keep the very large pretrained matrix fixed. Learn only a correction that comes from two thin matrices. The number of trained weights for each matrix decreases from d×kd \times k to r(d+k)r(d + k). Thus, memory and training time decrease very much. A larger rank rr lets the correction make more change (nearer to full fine-tuning), at a higher cost. A smaller rr costs less but permits less change.
x W₀ (frozen) d × k pretrained weights A B + h trainable: r(d + k) weights
LoRA keeps the pretrained weights frozen. It trains a small low-rank side path and adds its output.

Worked example 10How much smaller is a LoRA update?

Illustrative: one 4096×40964096 \times 4096 projection matrix (the hidden size of Llama-2-7B is 4096) and LoRA rank r=8r = 8.

1
Full fine-tuning of this matrix: 4096×4096=16,777,2164096 \times 4096 = 16{,}777{,}216 weights.
2
LoRA: AA has 8×40968 \times 4096 weights and BB has 4096×84096 \times 8 weights. Thus, r(d+k)=8×8192=65,536r(d + k) = 8 \times 8192 = 65{,}536 weights.
3
Ratio: 65,53616,777,216≈0.0039 \frac{65{,}536}{16{,}777{,}216} \approx 0.0039
ResultLoRA trains approximately 0.39% of the weights of that matrix. For this reason, RepLLaMA can fine-tune a model with 7 billion parameters on retrieval labels at a practical cost.
Key ideaE5-Mistral and RepLLaMA in one line. They use the same embedding method (instruction + [EOS], last hidden state, contrastive loss). E5-Mistral puts its effort into synthetic training data from an LLM. RepLLaMA puts its effort into efficient supervised fine-tuning (LoRA on labelled data only).
PracticeWhat do E5-Mistral and RepLLaMA share? On which design dimension are they most different?
AnswerThey share a decoder-only LLM backbone, the query with an instruction, the [EOS] last-hidden-state embedding and a contrastive (InfoNCE) loss. They are most different in the training signal (positive/negative sampling). E5-Mistral generates synthetic triples with GPT-4 (and uses MS MARCO). RepLLaMA trains only on supervised labels and uses LoRA for efficiency.
PracticeDoes LoRA make the encoding of a RepLLaMA query faster at search time?
AnswerNo. The lecture gives LoRA as an efficiency for training. It decreases the number of weights that fine-tuning changes. At search time, the query still goes through the full model with 7B parameters.

15. Model comparison summary

Sources: slides 18–48 · Videos 2–9 · In-class slides 10–12, 16

ModelBackboneRepresentationSimilarityLossNegatives / training signal
DPR (2020)Encoder[CLS] embeddingDot productInfoNCE (negative log likelihood)Random + BM25 hard + in-batch
RepBERT (2020)EncoderMean of token embeddingsInner productMulti-label margin lossIn-batch
ANCE (2021)Encoder[CLS] embeddingInner productInfoNCESelf-mined hard negatives with ANN over training checkpoints
Contriever (2021)EncoderMean of token embeddingsCosine similarityInfoNCEUnsupervised: two spans of the same document = positive pair
ColBERT (2020)Encoder (multi-vector)All token embeddingsMaxSim (late interaction)InfoNCEStandard triples (as in DPR-style training)
E5-Mistral (2023/24)Decoder, Mistral-7B[EOS] last hidden state, query with instructionCosine/dot (contrastive)InfoNCEGPT-4 synthetic ⟨q, p⁺, hard p⁻⟩ + MS MARCO; in-batch + hard
LLM2Vec (2024)Decoder (any LLM)Mean pooling over the bidirectional LLMCosine (contrastive)SimCSE contrastive (+ optional supervised)Unsupervised: dropout views of the same text (+ optional labels)
RepLLaMA (2023)Decoder, Llama-2-7B[EOS] last hidden state, query with instructionDot productInfoNCEOnly supervised labels (MS MARCO), trained with LoRA adapters
Dense retrievers Encoder backbone (BERT) Decoder backbone (LLM) Single vector DPR, ANCE, RepBERT, Contriever Multi vector ColBERT late interaction Single vector E5-Mistral, LLM2Vec, RepLLaMA
The eight models of this week, by backbone and by number of vectors for each text (slide 3 and slide 17).
PracticeWhich two models in the table use cosine similarity on mean-pooled vectors? What else do they share?
AnswerContriever and LLM2Vec. Both can train without human relevance labels. Contriever uses spans of the same document as positives. LLM2Vec uses two dropout views of the same text (SimCSE).
PracticeWhich model is the only one of this week that trains with a margin loss instead of InfoNCE? Which model is the only multi-vector model?
AnswerRepBERT (multi-label margin loss). ColBERT (all token embeddings, MaxSim).

16. Course context from the in-person lecture

Sources: In-class slides 2–7 · slides 4–6

The in-person session started with the "historical view of ranking models" timeline of the course. It shows how the weeks connect. It also shows which topics the exam covers:

1950s–90s term-based lexical models Week 3 Weeks 7, 11, 12 2000s–10s feature-based learning to rank (LTR) 2010s word2vec, RNN, LSTM not covered Week 4 2017–18 Transformer: BERT, GPT WE ARE HERE 2018–22 dense retrievers, learned sparse, rerankers Weeks 5, 6 (+ 8) Parts of 5, 9, 10 2022 onwards AI age: ChatGPT, Mistral, Claude, Qwen The future LLM rerankers, RAG, search agents Parts of 5, 9, 10 "The field is moving very fast: we focus on core ideas and the most impactful techniques."
The ranking-models timeline from the in-person lecture (in-class slide 3), with the course weeks on it. Week 5 (dense retrievers) and Week 6 (learned sparse) are in the "new inventions" period. The LLM-based models of this week are already in the AI age.

The lecturer also told how the weeks build on each other. Weeks 1, 2 and 4 give preliminary knowledge. Weeks 3, 5, 6, 7 and 8 are the core models. This week is a core week. Weeks 9–10 cover new research. Weeks 11–13 are also important. In the cost table of relevance models, the last empty slot is for learned sparse retrievers, the topic of next week. The session also gave the course survey result (average 4.54 out of 5 from 24 responses). Students asked that lectures also include concepts. For this reason, the session included the transformer recap in Section 17.

16.1 The models at a glance: years and one-line innovations

The in-person slides show each model as a card: representation, similarity and loss. Each card also has a one-line "innovation" that makes the model important. Use this table as a revision checklist:

Model (year)Card (representation · similarity · loss)Innovation in one line
DPR (2020)[CLS] embedding · dot product · InfoNCEThe influential dual-encoder model.
RepBERT (2020)Mean embedding · dot product · multi-label margin lossStrong early BERT dual-encoder baseline. Mean pooling + margin loss is better than BM25 on MS MARCO.
ANCE (2021)[CLS] embedding · dot product · InfoNCEAsynchronous negative sampling from an index that changes during training.
Contriever (2021)Mean embedding · cosine similarity · InfoNCETrains without labels on very large corpora with contrastive learning.
ColBERT (2020)All token embeddings · MaxSim · InfoNCEEfficient late interaction: token-level query and document embeddings, scored with MaxSim.
E5-Mistral (2024)[EOS] embedding · dot product · InfoNCESupervised synthetic data from GPT-4 + MS MARCO labels.
LLM2Vec (2024)Mean pooling · dot product · InfoNCEBidirectional attention + masked next-token prediction + unsupervised SimCSE. Supervised tuning is optional.
RepLLaMA (2023/24)[EOS] embedding · dot product · InfoNCELLaMA bi-encoder trained on MS MARCO with hard and in-batch negatives.
Watch outThe representation of LLM2Vec. The in-person model card gives mean pooling for LLM2Vec. After its three steps, the LLM is bidirectional. Each token vector then contains the full context. Thus, an average of the token vectors is useful again (as in RepBERT and Contriever). E5-Mistral and RepLLaMA keep causal attention. Thus, they use the [EOS] last-token embedding, the only position that sees the full input. The model cards and the main slides also give different similarity functions for some models (for example, dot product on the card and cosine in the comparison table). Both answers are correct if you can explain the choice.

Remember these group remarks from the lecture. DPR and RepBERT are the early series (2020). The card of ANCE shows that negative sampling is important during training. The card of Contriever shows that these models can train without supervised labels.

PracticeWhich weeks did the lecturer call the "core models"? Which two of them cover neural first-stage retrievers?
AnswerWeeks 3, 5, 6, 7 and 8. Week 5 (dense, embedding-based retrievers) and Week 6 (learned sparse retrievers).

17. Transformer recap from the in-person session

Sources: In-class slides 9, 14–15 · Week 4 lecture notes

The survey feedback said: "lecture should include concepts as well". Thus, the in-person session taught the transformer pipeline from Week 4 again before the exercises. All topics of this week use it. This is a short version of the recap:

αij=softmaxj ⁣(qi⋅kjd),oi=∑jαij vj \alpha_{ij} = \mathrm{softmax}_j\!\left(\frac{\mathbf{q}_i \cdot \mathbf{k}_j}{\sqrt{d}}\right),\qquad \mathbf{o}_i = \sum_{j} \alpha_{ij}\, \mathbf{v}_j
Where
qi\mathbf{q}_i
the query vector of token ii (the token that attends)
kj, vj\mathbf{k}_j,\ \mathbf{v}_j
the key vector and the value vector of token jj
dd
the dimension of the query and key vectors. d\sqrt{d} keeps the dot products in a stable range.
softmaxj\mathrm{softmax}_j
normalises over all tokens jj that token ii can see. The weights are positive and their sum is 1.
αij\alpha_{ij}
the attention weight that token ii gives to token jj
oi\mathbf{o}_i
the output vector of token ii
Read it aloud Alpha i j equals the softmax over j of q-i dot k-j, divided by the square root of d. o-i equals the sum over j of alpha i j multiplied by v-j.
Formula in plain words Token ii compares its query with each key. This shows how important each other token is. Softmax changes these comparisons into weights with a sum of 1. The output is the weighted sum of the value vectors. A key that is very similar to the query gets a large weight. It pulls the output towards its value. In an encoder, jj goes over all tokens. In a decoder, jj goes only over j≤ij \le i (the causal mask of Section 13). The weights depend on qi\mathbf{q}_i. Thus, the attention of token ii to token jj is usually different from the attention of jj to ii.
token E([CLS]), 768 + segment E(A), 768 + position E(pos 0), 768 = input vector still 768 values element-wise addition: dimension 1 + dimension 1 + dimension 1, … example: the [CLS] token at position 0 of sentence A
The input vector of one token is the sum of three embeddings of the same size (the answer to the in-class exercise).
PracticeBERT-base uses embeddings with 768 dimensions. What is the dimension of the input vector of the [CLS] token after the model adds its token, segment and position embeddings?
Answer768. The model adds the three vectors element by element. Thus, the dimension stays the same. The model does not stack them (stacking gives 3 × 768).

18. Q&A from the in-person lecture

Sources: In-class slides 21–22

The discussion was the most useful part of the live session. These are the questions (from students or from in-class exercises) and the answers, in a clear form.

18.1 Transformer mechanics

Q: Is the self-attention matrix symmetric?
A: No. Attention has a direction. The attention of token ii to token jj can be different from the attention of jj to ii. The lecturer said: "I care about that token; that token doesn't have to care about me." Each row comes from a different query vector. Thus, usually Aij≠AjiA_{ij} \ne A_{ji}.

Q (in-class MCQ): Which statements about the transformer are correct?
A: "The encoder uses masked attention to prevent looking at future tokens": false. This describes the decoder. The encoder must see the full sentence to capture its meaning. "The decoder uses bidirectional attention": false. The two are the wrong way around again. "The encoder processes input tokens in parallel using bidirectional attention": true. This parallel work is the reason why transformers use GPUs well. "Both encoder and decoder use the same type of attention": false. One is bidirectional and one is causal.

Q: Does BERT have more layers than GPT? Do they use different tokenisation?
A: The number of layers is a free design choice. In practice, BERT (approximately 110M parameters) is much smaller than modern GPT-style models. But you can build an encoder with more layers. Generation is the harder task. For this reason, decoder models grew to billions or trillions of parameters (the "scaling law"). Both families use word-piece-style tokenisation. The difference is the trained vocabulary (size and contents). Each vocabulary comes from a different collection with different goals.

Q: In the exercise, how do the three embeddings combine? Into a matrix?
A: The model adds the vectors: first dimension + first dimension + first dimension, and so on. Each token gets one vector with the model dimension.

Q: What does the linear layer of the decoder do?
A: It is a learned n×mn \times m projection from the model dimension nn to the vocabulary size mm. Multiply the hidden vector of the last token by it. This gives a score (logit) for each vocabulary token. Softmax changes these scores into probabilities between 0 and 1. The model selects the next token from the top of this distribution. Deployed systems use sampling from the distribution. They do not always take the argmax. Thus, the same prompt can give different generations.

18.2 Using encoders and decoders for ranking (in-class Exercises 1 and 2)

Q: How can you get a text representation from a transformer encoder? How do you compute similarity?
A: There are three methods. This is the answer to Exercise 1. Use the [CLS] vector (DPR, ANCE). Calculate the mean of the token vectors (RepBERT, Contriever). Or keep more than one token vector (ColBERT). For the single-vector methods, similarity is one dot product or cosine between the query vector and the document vector. For the multi-vector method, similarity is late interaction. Compare each query token embedding with each document token embedding. Take the maximum for each query token. Add the maximum values (MaxSim). A student gave this method in class. The lecturer said that ColBERT started from almost the same idea.

Q: How can a transformer decoder compute query–document similarity? (Exercise 2. The class gave many answers. All are correct.)

Decoder LLM 1. Embedder last-token vector, dot product E5-Mistral, RepLLaMA 2. Generative ranker prompt: pointwise, pairwise or listwise ordering 3. Expansion (doc2query) generate terms at indexing time, then BM25 matches 4. Output probabilities vocabulary distribution as term weights (Week 6 preview)
The four answers to Exercise 2 from the class. A decoder can embed, rank with a prompt, expand documents, or give term weights.

Q: Why does expansion occur at indexing time and not at query time?
A: Latency. At query time, the user waits. Long generation at that time makes the experience bad. The system processes documents offline at indexing time. There, extra minutes cost nothing. The same reason applies to the offline document embeddings of a bi-encoder.

18.3 Beyond this week

Q: Can an encoder backbone generate text?
A: Yes. This is the idea of diffusion language models, a very new research area. The encoder learned masked-token prediction. Thus, add, for example, 256 [MASK] tokens to the prompt. Predict them all at the same time (parallel work, good for GPUs). Do not predict them one at a time. The problem: tokens that the model predicts separately can be incoherent. The solution comes from image diffusion. Run some denoising passes. This needs far fewer passes than 256 auto-regressive steps. The lecturer said that his own recent work applies diffusion language models to retrieval.

Q (assignment): What must a gain–loss plot look like?
A: One bar for each query. The bar is the evaluation score of System A minus the score of System B on that query. Usually, sort the bars from the largest gain to the largest loss. Then you can quickly see if A is always better than B. More importantly, you can see which queries A loses on. Analyse those queries. The BM25 baseline in the assignment must be your own implementation. Read the criteria. If the criteria are not clear, ask on the discussion board.

PracticeGive two ways to use a decoder LLM for ranking without running it at query time over each document.
Answer(1) Use it as an embedder. The system computes document embeddings offline and encodes only the query online. (2) Use it for document expansion (doc2query) at indexing time. Then rank with BM25. (Term weights from output probabilities, the preview for Week 6, can also be computed offline for documents.)

19. In-class exercises and where the solutions live

Sources: In-class slides 21–25 · Practical solution pages for Exercises 3–5

The session had five exercises. Exercises 1–2 are conceptual. The Q&A section above answers them. Exercises 3–5 are calculation or challenge questions. Their full step-by-step solutions are in the Practical/ folder:

ExerciseQuestionAnswerSolution
1: Encoder-basedMethods to get a text representation from an encoder; which tokens; how to compute similarity[CLS] / mean / all tokens; dot or cosine, or MaxSimSection 18.2; Sections 4.1 and 11
2: Decoder-basedHow can a decoder compute query–document similarity?Last-token embedding; LLM as ranker; doc2query; representations from output probabilitiesSection 18.2; Section 4.2
3: InfoNCE losssim(q, p⁺) = 0.9; negatives 0.3, 0.1, 0.4L=−ln⁡(0.3839)≈0.96\mathcal{L} = -\ln(0.3839) \approx 0.96worked solution
4: Margin lossRel(q, p⁺) = 1.5; negatives 0.3, 0.1, 0.9; margin 1L=0.43≈0.133\mathcal{L} = \frac{0.4}{3} \approx 0.133worked solution
5: ChallengeFind negatives that are harder than the top BM25 documentsANCE: self-mined negatives, refreshed from checkpointsworked solution

Worked example 11Exercise 3 in brief: InfoNCE with three negatives

From in-class slide 23: sim(q,p+)=0.9\mathrm{sim}(q,p^+) = 0.9. Negatives: 0.3, 0.1, 0.40.3,\ 0.1,\ 0.4.

1
Calculate the exponentials: e0.9≈2.4596e0.3≈1.3499e0.1≈1.1052e0.4≈1.4918 \begin{aligned} e^{0.9} &\approx 2.4596 \\ e^{0.3} &\approx 1.3499 \\ e^{0.1} &\approx 1.1052 \\ e^{0.4} &\approx 1.4918 \end{aligned}
2
Bottom of the fraction: 2.4596+1.3499+1.1052+1.4918=6.40652.4596 + 1.3499 + 1.1052 + 1.4918 = 6.4065.
3
Probability of the positive: 2.45966.4065≈0.3839 \frac{2.4596}{6.4065} \approx 0.3839
L=−ln⁡(0.3839)≈0.957 \mathcal{L} = -\ln(0.3839) \approx 0.957
ResultL≈0.96\mathcal{L} \approx 0.96. The positive is only a little higher than the hardest negative (0.9 against 0.4). Thus, the model gives it only 38% of the probability mass, and the loss is high. The full solution page gives more checks.

Worked example 12Exercise 4 in brief: margin loss with three negatives

From in-class slide 24: Rel(q,p+)=1.5\mathrm{Rel}(q,p^+) = 1.5. Negatives: 0.3, 0.1, 0.90.3,\ 0.1,\ 0.9. Margin: 1.

1
Gaps: 1.5−0.3=1.21.5−0.1=1.41.5−0.9=0.6 \begin{aligned} 1.5 - 0.3 &= 1.2 \\ 1.5 - 0.1 &= 1.4 \\ 1.5 - 0.9 &= 0.6 \end{aligned}
2
Hinge for each pair: max⁡(0,1−1.2)=0max⁡(0,1−1.4)=0max⁡(0,1−0.6)=0.4 \begin{aligned} \max(0, 1 - 1.2) &= 0 \\ \max(0, 1 - 1.4) &= 0 \\ \max(0, 1 - 0.6) &= 0.4 \end{aligned}
3
Calculate the average over the 3 pairs:
L=0+0+0.43≈0.133 \mathcal{L} = \frac{0 + 0 + 0.4}{3} \approx 0.133
ResultL≈0.133\mathcal{L} \approx 0.133. Only the hardest negative (0.9) is inside the margin. Thus, all of the gradient pushes this negative away.
PracticeIn Exercise 4, how low must the score of the hardest negative go to make the loss zero?
AnswerThe gap must be 1 or more. Thus, Rel(q,p3−)≤1.5−1=0.5\mathrm{Rel}(q,p_3^-) \le 1.5 - 1 = 0.5. At 0.5 or less, all pairs reach the margin and the loss is 0.
PracticeExercise 5: training with BM25 hard negatives improves slowly. Give a method to find harder negatives.
AnswerUse the dense model itself, as ANCE does. Save a checkpoint at regular intervals. Encode the corpus again with it. Retrieve documents for each training query. Use the top documents without a relevant label as new negatives. Refresh them during training. These documents confuse the current model. Thus, they give the strongest learning signal. The cost is that the corpus must be encoded again many times.

20. Summary and study checklist

Takeaways

Study checklist. You must be able to:

21. Video index

YouTube playlist for Week 5: IR Week 5. The transcripts are in week_5_video_transcripts.md. The slides of the in-person lecture are in Lecture-Slides/. The file INFS7410_S2_2026_STL-transcript.txt in this folder is a copy of the Week 4 lecture transcript. These notes do not use it.

VideoTopicLengthSections here
Video 1Overview: embedding-based retrievers18:341–6
Video 2DPR (Dense Passage Retrieval)11:105, 7
Video 3RepBERT04:268
Video 4ANCE03:439
Video 5Contriever03:0610
Video 6ColBERT07:115.2, 11
Video 7E5-Mistral07:5812
Video 8LLM2Vec04:3413
Video 9RepLLaMA03:0914

22. References