INFS7410 · Information Retrieval and Web Search · Week 5

Embedding-based Retrieval Models (Dense Retrievers)

How transformer encoders and decoders become search engines: bi-encoders (DPR, RepBERT, ANCE, Contriever), late interaction (ColBERT) and LLM-based embedders (E5-Mistral, LLM2Vec, RepLLaMA). For each model the notes cover the representation, the similarity function, the training loss (InfoNCE or margin loss) and the negative-sampling strategy. Lecturer: Dr Shuai Wang, The University of Queensland.

1. The big picture: from lexical to dense retrieval

Sources: slides 2–3, 8 · Video 1 (Overview: embedding-based retrievers) · In-class slides 4, 7

Where we are in the course. Week 3 covered term-based lexical models such as BM25, which score a document by counting how its terms overlap with the query's terms. Week 4 covered the transformer architecture: encoders (BERT) read a whole sequence with bidirectional attention and produce one contextual vector per token; decoders (GPT) read left to right with causal attention and predict the next token. This week the two threads meet: transformers embed queries and documents as vectors, and retrieval ranks documents by vector similarity in place of term overlap.

Watch outWhy lexical models fall short. BM25 can only match documents that share exact words with the query. It suffers from vocabulary mismatch: "car" and "automobile" share no terms, so a lexical model sees zero similarity between them even though they mean the same thing. Dense retrievers fix this by mapping both words to nearby points in a learned vector space.
Lexical match (BM25) q: car d: automobile ≠ shared terms: none term-overlap score = 0 vocabulary mismatch Dense match (embeddings) η(car) η(automobile) small angle: high similarity
Lexical models need shared terms; dense retrievers compare learned vectors, so synonyms with no shared terms can still score highly.

This week's models, organised by their backbone architecture:

BackboneRepresentation styleModels
Transformer encoderSingle vector: one vector per query or documentDPR, ANCE, RepBERT, Contriever
Transformer encoderMulti vector: one vector per tokenColBERT
Transformer decoder (LLM)Single vector: last-token vectorE5-Mistral, LLM2Vec, RepLLaMA
Key ideaBoth the encoder and the decoder side of the transformer can be repurposed for dense retrieval, even though the decoder was originally built for text generation. The early work used encoders because they already read the whole text at once; later work showed that the internal layers of large decoders also hold rich text representations.
PracticeA user searches for "automobile insurance" and the best document only says "car cover". Why does BM25 miss it, and what does a dense retriever do differently?
Think about what BM25 counts, and where a dense retriever places each text.
AnswerBM25 scores term overlap, and the query and document share no terms, so the document scores zero for those terms (vocabulary mismatch). A dense retriever embeds both texts into a learned vector space where synonyms land close together, so the similarity of the two vectors is high even without shared words.

2. The table of relevance models: costs

Sources: slides 4–6 · Video 1 (Overview: embedding-based retrievers) · In-class slide 6

Recall the "table of relevance models" from Week 3. Each family of models is characterised by two costs:

Model familyExamplesScorer costEncoder cost
Term-based lexical modelsBM25, TF-IDFO(<1)O(<1): cheap inverted-index lookupsO(∣d∣)O(|d|): linear scan to count terms
Single-vector embedding-based retrieversDPR, ANCE, RepBERT, Contriever, E5, LLM2Vec, RepLLaMAO(1)O(1): one dot product or cosineO(∣d∣2)O(|d|^2): self-attention over all token pairs
Multi-vector embedding-based retrieversColBERTO(m×n)O(m \times n): every query token against every document tokenO(∣d∣2)O(|d|^2): same transformer encoding
Learned sparse retrieversnext week (Week 6)(next week)(next week)

How to read the cost terms:

Offline, once per document (indexing time) Documents whole collection Encoder η O(|d|²) per document Vector index stored document vectors Online, for every query (the user is waiting) Query q short text Encoder η one short pass Similarity φ O(1) per document Ranked list
The quadratic encoder cost is paid offline for documents; at query time only one short query encoding and cheap similarity scores remain.
Key ideaSingle-vector retrievers are fast in practice because the expensive O(∣d∣2)O(|d|^2) encoding of documents happens once, offline. At query time the system encodes only the (short) query and then runs millions of cheap O(1)O(1) similarity computations against pre-computed document vectors.
PracticeWhy is the scorer cost of a bi-encoder O(1) per document while its encoder cost is O(|d|²), and why does the quadratic part leave query latency almost untouched?
AnswerScoring compares two fixed-size vectors with one dot product or cosine, whatever the document length. Encoding runs self-attention over all token pairs, which is quadratic in length. Document encodings are computed offline and stored, so at query time only the short query is encoded (one pass) and the per-document work is the constant-cost similarity.
PracticeWhich slot in the relevance-model cost table stays empty this week, and when is it filled?
AnswerThe learned sparse retrievers slot. The lecturer left it for Week 6; those models are more complex to build even though they are often more efficient than the embedding-based approaches.

3. Dense representations: embedders and similarity

Sources: slide 7 · Video 1 (Overview: embedding-based retrievers) · In-class slide 8

Every dense retriever has two ingredients. First, an embedder η(⋅)\eta(\cdot): a neural network (BERT, an LLM, …) that maps a text to a fixed-size vector. The same kind of embedder is applied to the query qq and the document dd, so both land in the same vector space. Second, a similarity function ϕ(⋅,⋅)\phi(\cdot,\cdot): a cheap function (dot product, cosine similarity, …) that compares the two vectors.

score(q,d)=ϕ(η(q), η(d)) \mathrm{score}(q,d) = \phi\big(\eta(q),\ \eta(d)\big)
Where
q, dq,\ d
the query text and the document text
η(⋅)\eta(\cdot)
the embedder (eta): a neural network that maps a text to a fixed-size vector
η(q), η(d)\eta(q),\ \eta(d)
the query embedding and the document embedding (for example 768 numbers each)
ϕ(⋅,⋅)\phi(\cdot,\cdot)
the similarity function (phi), for example inner product or cosine similarity
score(q,d)\mathrm{score}(q,d)
one number used to rank dd for qq
Read it aloud The score of document d for query q is phi applied to eta of q and eta of d: the similarity between the embedding of the query and the embedding of the document.
Formula in plain words Run the query through the embedder, run the document through the embedder, then measure how close the two resulting vectors are. All the "understanding" lives in η\eta, which is learned during training; ϕ\phi is deliberately cheap so it can be applied to millions of stored document vectors. Ideally ϕ(η(q),η(d))\phi(\eta(q),\eta(d)) is large exactly when dd is relevant to qq: a better-trained embedder moves relevant pairs closer and irrelevant pairs further apart.
query q Embedder η η(q) [0.8, −1.2, …, 2.4, −0.3] document d Embedder η η(d) [0.5, 0.0, …, 2.6, −0.7] φ score
The dense retrieval recipe from slide 7: the same embedder maps query and document to low-dimensional vectors, and φ compares them.

The vectors are low-dimensional compared with a lexical vocabulary: typically 768 dimensions for BERT-based models and around 3,000–4,000 for decoder (LLM) based models, against tens of thousands of possible terms in a bag-of-words representation. Relevance is measured as geometric closeness in a learned semantic space, so "car" and "automobile" can end up with very similar embeddings even though they share no characters.

3.1 The two standard similarity functions

Dot product (inner product). You met vector operations in the Week 4 maths refresher.

ϕdot(q,d)=q⋅d=∑i=1kqi di \phi_{\text{dot}}(\mathbf{q},\mathbf{d}) = \mathbf{q}\cdot\mathbf{d} = \sum_{i=1}^{k} q_i\, d_i
Where
q, d\mathbf{q},\ \mathbf{d}
the query and document embeddings, η(q)\eta(q) and η(d)\eta(d)
kk
the number of dimensions of each vector (for example 768)
qi, diq_i,\ d_i
the ii-th entry of the query vector and of the document vector
∑i=1k\sum_{i=1}^{k}
add up over all kk dimensions
Read it aloud Phi-dot of q and d equals q dot d, which is the sum, for i from 1 to k, of q-i times d-i.
Formula in plain words Multiply the two vectors entry by entry, then add all the products: one number out, and bigger means more similar. Dimensions where both vectors are large and share a sign push the score up; dimensions with opposite signs pull it down. Because the raw entries are used, a longer vector (larger magnitude) gets a larger score even at the same direction, so the dot product mixes "direction" and "length".

Cosine similarity. This is the same cosine used by the vector space model in Week 3, now applied to learned embeddings.

ϕcos⁡(q,d)=q⋅d∥q∥ ∥d∥ \phi_{\cos}(\mathbf{q},\mathbf{d}) = \frac{\mathbf{q}\cdot\mathbf{d}}{\lVert\mathbf{q}\rVert\,\lVert\mathbf{d}\rVert}
Where
q⋅d\mathbf{q}\cdot\mathbf{d}
the dot product from the formula above
∥q∥\lVert\mathbf{q}\rVert
the length (L2 norm) of q\mathbf{q}: ∑iqi2\sqrt{\sum_i q_i^2}
∥d∥\lVert\mathbf{d}\rVert
the length (L2 norm) of d\mathbf{d}
Read it aloud Phi-cos of q and d equals q dot d divided by the norm of q times the norm of d.
Formula in plain words Take the dot product, then divide by the product of the two vector lengths. The division cancels the effect of magnitude, so what remains is the cosine of the angle between the vectors: 1 when they point the same way, 0 when they are at right angles, −1 when they point in opposite directions. Stretching either vector leaves the cosine unchanged; turning it away from the other vector lowers it. If both vectors are first scaled to length 1 (L2-normalised), dot product and cosine give the same number.
q d₁ d₂ = 2·d₁ θ q · d₁ = 8, q · d₂ = 16 dot product doubles with the length cos(q, d₁) = cos(q, d₂) ≈ 0.889 same angle θ, same cosine
Dot product responds to direction and length; cosine responds to direction only (numbers from Worked example 1).

Worked example 1Dot product against cosine similarity

Illustrative, tiny 3-dimensional embeddings for a query and two documents:

q=[2,1,2],d1=[1,2,2],d2=[2,4,4] \mathbf{q} = [2, 1, 2],\quad \mathbf{d}_1 = [1, 2, 2],\quad \mathbf{d}_2 = [2, 4, 4]
1
Dot product with d1\mathbf{d}_1: multiply entry by entry and sum. q⋅d1=(2)(1)+(1)(2)+(2)(2)=2+2+4=8 \mathbf{q}\cdot\mathbf{d}_1 = (2)(1) + (1)(2) + (2)(2) = 2 + 2 + 4 = 8
2
Norms. The norm is the square root of the sum of squared entries. ∥q∥=22+12+22=9=3∥d1∥=12+22+22=9=3 \begin{aligned} \lVert\mathbf{q}\rVert &= \sqrt{2^2+1^2+2^2} = \sqrt{9} = 3 \\ \lVert\mathbf{d}_1\rVert &= \sqrt{1^2+2^2+2^2} = \sqrt{9} = 3 \end{aligned}
3
Cosine similarity with d1\mathbf{d}_1: ϕcos⁡(q,d1)=83×3=89≈0.889 \phi_{\cos}(\mathbf{q},\mathbf{d}_1) = \frac{8}{3 \times 3} = \frac{8}{9} \approx 0.889
4
Now d2=2d1\mathbf{d}_2 = 2\mathbf{d}_1 (same direction, twice as long): q⋅d2=(2)(2)+(1)(4)+(2)(4)=16∥d2∥=4+16+16=6ϕcos⁡(q,d2)=163×6=89≈0.889 \begin{aligned} \mathbf{q}\cdot\mathbf{d}_2 &= (2)(2) + (1)(4) + (2)(4) = 16 \\ \lVert\mathbf{d}_2\rVert &= \sqrt{4+16+16} = 6 \\ \phi_{\cos}(\mathbf{q},\mathbf{d}_2) &= \frac{16}{3 \times 6} = \frac{8}{9} \approx 0.889 \end{aligned}
ResultThe dot product doubles (8 → 16) when the document vector doubles in length, while the cosine similarity stays at 0.889. Dot product is sensitive to magnitude; cosine responds only to direction. This is why models that use the raw dot product (DPR, ANCE, RepLLaMA) and models that use cosine (Contriever) can behave differently, and why ColBERT L2-normalises its token vectors before taking dot products (which makes them equal to cosines).
PracticeTwo documents have embeddings pointing in exactly the same direction, but one vector is three times as long. Which similarity function ranks them equally, and which prefers the longer vector?
AnswerCosine similarity ranks them equally, because it divides out both lengths and keeps only the angle. The dot product gives the longer vector three times the score, because it scales with magnitude.

4. Two backbones: encoder vs decoder

Sources: slides 8–11, 14–16 · Video 1 (Overview: embedding-based retrievers) · In-class slides 9, 14–15

4.1 Encoder-only (BERT-style): one vector per token, then pool

A transformer encoder (Week 4) reads the whole input at once with bidirectional attention and outputs one contextual vector per token. Retrieval needs a representation of the whole text, so a model must decide how to go from many token vectors to a text representation. There are three standard strategies:

input text: [CLS] deep learning for search … Transformer encoder (BERT) v[CLS] v₁ v₂ v₃ v₄ … one vector per token A: [CLS] vector keep only v[CLS] DPR · ANCE B: mean vector average all token vectors RepBERT · Contriever C: all vectors keep every token vector ColBERT
An encoder produces one vector per token. Single-vector models compress these into the [CLS] vector (DPR, ANCE) or the mean of all token vectors (RepBERT, Contriever); ColBERT keeps them all.

Mean pooling, written as a formula:

ηmean(t)=1n∑i=1nvi \eta_{\text{mean}}(t) = \frac{1}{n}\sum_{i=1}^{n} \mathbf{v}_i
Where
tt
the text being embedded (a query or a document)
nn
the number of tokens in tt
vi\mathbf{v}_i
the encoder's output vector for token ii (for BERT, 768 numbers)
ηmean(t)\eta_{\text{mean}}(t)
the single text vector, with the same number of dimensions as each vi\mathbf{v}_i
Read it aloud Eta-mean of t equals one over n times the sum, for i from 1 to n, of v-i.
Formula in plain words Add up the token vectors dimension by dimension, then divide by the number of tokens. Every token contributes equally, so the result summarises the whole text with every position counted. Dividing by nn keeps long and short texts on the same scale: without it, longer texts would get larger vectors and larger dot products. The price is that a single strongly relevant token is diluted by all the others.

Worked example 2Mean pooling

Illustrative token vectors for a 3-token text (dimension 3 for readability):

v1=[0.2,0.8,−0.4],v2=[0.6,0.0,0.2],v3=[0.4,0.4,0.8] \mathbf{v}_1 = [0.2, 0.8, -0.4],\quad \mathbf{v}_2 = [0.6, 0.0, 0.2],\quad \mathbf{v}_3 = [0.4, 0.4, 0.8]
1
Sum the vectors coordinate by coordinate: v1+v2+v3=[0.2+0.6+0.4, 0.8+0.0+0.4, −0.4+0.2+0.8]=[1.2, 1.2, 0.6] \mathbf{v}_1+\mathbf{v}_2+\mathbf{v}_3 = [0.2+0.6+0.4,\ 0.8+0.0+0.4,\ -0.4+0.2+0.8] = [1.2,\ 1.2,\ 0.6]
2
Divide by the number of tokens (3): mean=13[1.2, 1.2, 0.6]=[0.4, 0.4, 0.2] \text{mean} = \frac{1}{3}[1.2,\ 1.2,\ 0.6] = [0.4,\ 0.4,\ 0.2]
ResultThe single vector [0.4, 0.4, 0.2] now represents the whole text. This is exactly what RepBERT and Contriever store per document.

4.2 Decoder-only (GPT-style): discard generation, keep the last-token vector

A decoder predicts the next token given the input, using causal attention (each token only sees earlier tokens). It was designed for generation, yet its internal layers still learn rich representations of the text. The trick for retrieval:

input: what is dense retrieval [EOS] Transformer decoder (LLM) h(what) h(is) … h(last token) next-token prediction head discarded for retrieval text embedding stored in an ANN index
Decoder-based embedding: drop the next-token prediction part and use the hidden state of the last token (usually an appended [EOS] token) as the embedding of the whole text. Used by E5-Mistral and RepLLaMA.
Key ideaWhy the last token? Under causal attention, the last token is the only position that has attended to every previous token in the sequence, so its hidden state is the only one that "has seen" the entire input. That makes it the natural single-vector summary. This is why E5-Mistral and RepLLaMA append an [EOS] token to the text and take the last layer's vector at that position.
PracticeWhy would mean pooling over a causal decoder's token vectors be a weaker summary than the last-token vector?
What has the first token seen under causal attention?
AnswerUnder causal attention each token vector only reflects the tokens to its left, so early tokens carry almost no information about the rest of the text. Averaging mixes these partial views in with equal weight. The last token has attended to everything, so its vector is the only complete summary. (LLM2Vec, in Section 13, first makes attention bidirectional, which is why mean pooling becomes sensible there.)
PracticeCompute the mean-pooled vector of the token vectors [1, 0], [0, 1] and [2, 2].
AnswerSum the vectors, then divide by the number of tokens: sum=[3, 3]mean=sum3=[1, 1] \begin{aligned} \text{sum} &= [3,\ 3] \\ \text{mean} &= \frac{\text{sum}}{3} = [1,\ 1] \end{aligned}

5. Bi-encoders vs late interaction

Sources: slides 12–13, 17 · Video 1 (Overview: embedding-based retrievers) · Video 2 (DPR) · In-class slides 12–13

5.1 Single-vector retrievers: the bi-encoder architecture

Single-vector models use a bi-encoder (dual encoder): one encoder for the query and one for the document (often sharing weights), each producing a single vector. The score is one similarity computation.

query q (online, at search time) document d (offline, pre-computed) Query encoder η Document encoder η η(q): 1 vector η(d): 1 vector φ = dot or cosine score(q, d): one O(1) step
The bi-encoder: query and document are encoded independently into single vectors and only meet at the final cheap similarity step. DPR uses [CLS] embeddings and a dot product, as on slide 12.
Key ideaWhy bi-encoders scale to the web. Because the two texts are encoded independently, all document embeddings can be pre-computed offline and stored in a vector (ANN, approximate nearest neighbour) index. At query time only the query is encoded, one transformer pass, and then millions of cheap dot products run with efficient vector operations. The expensive part stays off the query-time path.

5.2 Multi-vector retrievers: late interaction

ColBERT keeps every token vector of both query and document, and lets them interact only at scoring time: hence late interaction. This captures richer, token-level matching signals and is often more effective, but scoring is more expensive (O(m×n)O(m \times n) in place of O(1)O(1)) and storage grows from 1 vector to hundreds of vectors per document. Section 11 walks through its MaxSim scoring step by step.

query token vectors document token vectors E(q₁) E(q₂) E(q₃) E(d₁) E(d₂) E(d₃) E(d₄) E(d₅) score = sum of the highlighted maxima
Late interaction compares every query token with every document token (grey lines), keeps each query token's best match (green) and sums those maxima.
PropertyBi-encoder (single vector)Late interaction (multi vector)
Vectors stored per document1about 100–500 (one per token)
Scoring per documentO(1)O(1): one dot product or cosineO(m×n)O(m \times n): MaxSim over all token pairs
Matching signalwhole-text summarytoken-level matches
Typical trade-offfaster, more storage-efficientoften more effective, more compute and storage
PracticeWhy can a bi-encoder pre-compute document vectors offline, and what would break this if the query and document were fed into one encoder together (a cross-encoder)?
AnswerIn a bi-encoder the document vector depends only on the document, so it can be computed once and reused for every query. If query and document were encoded together, every representation would depend on the query, so nothing could be pre-computed and each query would need one full transformer pass per candidate document.

6. The five dimensions that differentiate models

Sources: slides 17–18 · Video 1 (Overview: embedding-based retrievers)

Every model this week is a different answer to the same five design questions. Keeping these in mind makes the whole week easy to organise:

Design questionPossible answers seen this week
1. Model backbone: encoder or decoder?Encoder (DPR, ANCE, RepBERT, Contriever, ColBERT) or decoder LLM (E5-Mistral, LLM2Vec, RepLLaMA)
2. Representation: how is a text turned into vector(s)?[CLS] embedding · mean of token embeddings · all token embeddings · last-token/[EOS] embedding
3. Similarity function: how are vectors compared?Dot product · cosine similarity · late interaction (MaxSim)
4. Loss function: how is ranking learned in fine-tuning?Negative log likelihood (InfoNCE) · multi-label margin loss · SimCSE-style contrastive
5. Positive/negative sampling: where does the training signal come from?Random negatives · BM25 hard negatives · in-batch negatives · self-mined (ANCE checkpoint) negatives · same-document spans (Contriever) · LLM-generated synthetic triples (E5-Mistral)

To train any of these models you need, for each training query, a positive document (known relevant) and a set of negative documents to contrast against. The model learns to score the positive above the negatives; once it can do that, its embeddings are useful for dense retrieval.

query q positive p⁺ negatives p₁⁻ … pₙ⁻ Encoder η shared weights s⁺ = sim(q, p⁺) sj⁻ = sim(q, pj⁻) Loss InfoNCE or margin gradient update: raise s⁺, lower every sj⁻
Every model this week is fine-tuned in this loop; they differ in the encoder, the similarity, the loss and where the negatives come from.
PracticeANCE and DPR share the backbone, representation, similarity and loss. On which of the five dimensions do they differ?
AnswerDimension 5, positive/negative sampling. DPR uses random, BM25 hard and in-batch negatives; ANCE mines hard negatives with its own recent checkpoints.

7. DPR: Dense Passage Retrieval

Sources: slides 19–25 · Video 2 (DPR) · In-class slides 10, 17–20

Karpukhin et al., "Dense Passage Retrieval for Open-Domain Question Answering", EMNLP 2020. DPR is the representative bi-encoder: it demonstrated that a trained dense retriever could match and outperform BM25. Its key design choices:

Design choiceDPR
BackboneBERT encoder (bi-encoder: separate query and passage encoders)
Representation[CLS] token embedding
SimilarityDot product (simple, faster than cosine)
LossNegative log likelihood (InfoNCE)
NegativesRandom + BM25 hard + in-batch

7.1 The InfoNCE loss

The goal of the loss is to maximise the probability that the model picks the relevant passage out of a set of candidates. The in-class slide 17 summarises it as "encourages the positive passage to receive the highest score among candidates".

L(q,p+,p1−,…,pn−)=−log⁡esim(q,p+)esim(q,p+)+∑j=1nesim(q,pj−) \mathcal{L}(q, p^+, p_1^-, \dots, p_n^-) = -\log \frac{e^{\mathrm{sim}(q,p^+)}}{e^{\mathrm{sim}(q,p^+)} + \sum_{j=1}^{n} e^{\mathrm{sim}(q,p_j^-)}}
Where
qq
the query embedding (DPR: the query's [CLS] vector)
p+p^+
the positive (relevant) passage embedding
pj−p_j^-
the jj-th negative (irrelevant) passage embedding
nn
the number of negatives for this query
sim(q,p)=q⋅p\mathrm{sim}(q,p) = q \cdot p
the dot-product similarity
exe^{x}
the exponential function; it makes every term positive
−log⁡-\log
the negative natural logarithm, which turns a probability into a loss
L\mathcal{L}
the loss for one training query; training minimises its average
Read it aloud The loss for query q, positive p-plus and negatives p-one-minus to p-n-minus equals minus the log of a fraction: e to the similarity of q and p-plus, divided by e to the similarity of q and p-plus plus the sum, for j from 1 to n, of e to the similarity of q and p-j-minus.
Formula in plain words The numerator is the "score mass" of the correct passage. The denominator adds that same mass to the mass of every negative, so the fraction is a softmax: the probability that the model picks the correct passage out of all the candidates. The exponential amplifies differences: a positive at 0.8 and a negative at 0.2 differ by 0.6, but e0.8≈2.23e^{0.8} \approx 2.23 against e0.2≈1.22e^{0.2} \approx 1.22 sharpens the preference. The −log⁡-\log turns that probability into a loss: −log⁡(1)=0-\log(1) = 0 (perfect confidence, no loss), −log⁡(0.5)≈0.69-\log(0.5) \approx 0.69, and the loss grows without bound as the probability of the correct passage falls towards 0. Raising sim(q,p+)\mathrm{sim}(q,p^+) or lowering any negative's similarity lowers the loss, so training pulls query and relevant passage together and pushes irrelevant ones away. More (or harder) negatives make the denominator larger and the task harder.

Worked example 3Computing the DPR InfoNCE loss

Illustrative dot-product similarities for one query with one positive and two negatives:

sim(q,p+)=1.2,sim(q,p1−)=0.4,sim(q,p2−)=−0.3 \mathrm{sim}(q,p^+) = 1.2,\quad \mathrm{sim}(q,p_1^-) = 0.4,\quad \mathrm{sim}(q,p_2^-) = -0.3
1
Exponentiate each similarity (every term becomes positive and gaps are amplified): e1.2≈3.3201e0.4≈1.4918e−0.3≈0.7408 \begin{aligned} e^{1.2} &\approx 3.3201 \\ e^{0.4} &\approx 1.4918 \\ e^{-0.3} &\approx 0.7408 \end{aligned}
2
Form the softmax probability of picking the positive: positive mass over total mass. P(p+∣q)=3.32013.3201+1.4918+0.7408=3.32015.5527≈0.598 P(p^+ \mid q) = \frac{3.3201}{3.3201 + 1.4918 + 0.7408} = \frac{3.3201}{5.5527} \approx 0.598
3
Take the negative log: L=−log⁡(0.598)≈0.514 \mathcal{L} = -\log(0.598) \approx 0.514
4
Sanity check the behaviour. If training raises sim(q,p+)\mathrm{sim}(q,p^+) to 3.0 with the negatives unchanged, then P≈e3e3+1.4918+0.7408≈0.90L≈0.105 \begin{aligned} P &\approx \frac{e^{3}}{e^{3} + 1.4918 + 0.7408} \approx 0.90 \\ \mathcal{L} &\approx 0.105 \end{aligned} The result is higher confidence in the positive and a much lower loss.
ResultL≈0.514\mathcal{L} \approx 0.514. The model is only about 60% "sure" of the correct passage, so there is still a meaningful gradient pushing η(q)\eta(q) towards η(p+)\eta(p^+) and away from the negatives.

7.2 The three-part negative sampling strategy

DPR's success depends heavily on which negatives it contrasts against. It combines three sources of increasing difficulty:

whole corpus BM25 top-1000 q p⁺ Three kinds of negatives Random (easy) sampled anywhere in the corpus, e.g. a cooking recipe for an ML query BM25 hard top-ranked by BM25 but not relevant: shares keywords, wrong meaning In-batch (medium) other queries' positives in the batch: already encoded, so free DPR in practice: combine all three.
DPR's negative sampling (slides 22–25). Easy negatives are random corpus documents; hard negatives come from BM25's top results (lexically similar, semantically wrong); in-batch negatives are the other queries' positives, reused for free.
  1. Random negatives (easy). Sample passages uniformly from the whole corpus. For a query about machine learning, a random negative might be a cooking recipe. Useful for basic discrimination, but the model quickly finds them trivial and stops learning subtle distinctions; real search involves much harder cases.
  2. BM25 hard negatives. Retrieve the BM25 top-kk (for example top 1000) for the query, exclude anything labelled relevant, and sample negatives from that set. These passages share keywords with the query but are semantically irrelevant: query "how does machine learning work" against passage "machine learning applications in healthcare". They force the model to learn the difference between lexical similarity and true semantic relevance.
  3. In-batch negatives (medium, free). In a training batch with queries q1,q2,q3q_1, q_2, q_3 and positives p1+,p2+,p3+p_1^+, p_2^+, p_3^+: for q1q_1, reuse p2+p_2^+ and p3+p_3^+ as negatives. They are already encoded for the other queries, so they cost nothing extra. Because they come from the same dataset and domain, they are harder than random negatives: "medium" difficulty. Slide 25 credits them with efficiency and a domain-specific gain.
p₁⁺ p₂⁺ p₃⁺ p₄⁺ q₁ q₂ q₃ q₄ +−−− −+−− −−+− −−−+ each row qᵢ: 1 positive (+) B − 1 in-batch negatives (−) one softmax (InfoNCE) per row B queries × B passages: B² scores from only 2B encodings
In-batch negatives with B = 4: the diagonal holds each query's positive and the rest of its row holds free negatives.

Worked example 4Counting negatives in a DPR training batch

A training batch contains B=8B = 8 queries. Each query comes with 1 labelled positive passage and 1 BM25 hard negative, and all passages in the batch are shared as in-batch negatives. How many negatives does each query get, and how many passage encodings does the batch need?

1
Passages encoded in the batch: 8 positives + 8 BM25 hard negatives = 16 passage encodings (plus 8 query encodings).
2
Negatives available to one query, say q1q_1: the other queries' positives (8−1=78 - 1 = 7), plus all 8 hard negatives in the batch (its own and the other queries', all reusable) = 7+8=157 + 8 = 15 negatives.
3
Compare with naive sampling: giving each of the 8 queries its own 15 negatives would need 8×15=1208 \times 15 = 120 negative encodings in place of 16 total passage encodings. The in-batch trick gives each query 15 contrastive terms for the price of 2 passages per query.
ResultEach query trains against 15 negatives while the batch only encodes 16 passages. This computational efficiency is why in-batch negatives are used by nearly every model this week.
PracticeCompute the InfoNCE loss for sim(q, p⁺) = 2.0 and two negatives with similarities 1.0 and 0.0.
Follow Worked example 3: exponentiate, form the softmax probability of the positive, take −log.
AnswerExponentiate, form the probability of the positive, then take the negative log: e2≈7.389e1≈2.718e0=1P=7.38911.107≈0.665L=−log⁡(0.665)≈0.408 \begin{aligned} e^{2} &\approx 7.389 \\ e^{1} &\approx 2.718 \\ e^{0} &= 1 \\ P &= \frac{7.389}{11.107} \approx 0.665 \\ \mathcal{L} &= -\log(0.665) \approx 0.408 \end{aligned}
PracticeWhy are in-batch negatives called "medium" difficulty, harder than random negatives but easier than BM25 hard negatives?
AnswerThey are real positives for other training queries, so they come from the same dataset and domain as the true positive and look like plausible answers, which makes them harder than arbitrary corpus text. They were chosen for a different query, so they rarely share the specific keywords of this query, which keeps them easier than BM25's top-ranked lexical matches.

8. RepBERT: mean pooling and margin loss

Sources: slides 26–27 · Video 3 (RepBERT) · In-class slides 10, 17

Zhan et al., "RepBERT: Contextualized Text Embeddings for First-Stage Retrieval", arXiv 2020. RepBERT differs from DPR in two ways: how it pools token vectors, and how it computes the training loss.

Design choiceRepBERT
BackboneBERT encoder
RepresentationMean of all token embeddings (Section 4.1)
SimilarityInner (dot) product
LossMulti-label margin loss
NegativesIn-batch (for efficiency)

8.1 Multi-label margin loss

The in-class slide 17 states the aim: relevant documents should score higher than irrelevant documents by at least a fixed margin (here 1).

L=1∣D+∣ ∣D−∣∑di+∈D+∑dj−∈D−max⁡ ⁣(0, 1−(Rel(q,di+)−Rel(q,dj−))) \mathcal{L} = \frac{1}{|D^+|\,|D^-|} \sum_{d_i^+ \in D^+} \sum_{d_j^- \in D^-} \max\!\Big(0,\ 1 - \big(\mathrm{Rel}(q,d_i^+) - \mathrm{Rel}(q,d_j^-)\big)\Big)
Where
qq
the query
D+, D−D^+,\ D^-
the positive (relevant) and negative (irrelevant) documents for qq in the batch
∣D+∣ ∣D−∣|D^+|\,|D^-|
the number of positive–negative pairs, used to average
di+, dj−d_i^+,\ d_j^-
one positive document and one negative document
Rel(q,d)\mathrm{Rel}(q,d)
the relevance score: the dot product of the mean-pooled query and document vectors
11
the margin: the minimum gap wanted between a positive and a negative score
max⁡(0,⋅)\max(0,\cdot)
clips each pair's loss at zero (a hinge)
Read it aloud The loss equals one over the number of positives times the number of negatives, times the double sum over every positive d-i-plus and every negative d-j-minus of the maximum of zero and one minus the difference between Rel of q and d-i-plus and Rel of q and d-j-minus.
Formula in plain words For every pair of one positive and one negative document, measure how much the positive out-scores the negative, subtract that gap from 1, and clip at zero. If the positive already scores at least 1 full point higher than the negative, the pair contributes zero loss: it is satisfied. If the gap is smaller than 1 (or the negative even wins), the pair contributes the amount by which it falls short of the margin, so badly ordered pairs cost more. Then average over all positive–negative pairs. A larger margin would demand wider separation before a pair stops counting.
Watch outSlide 27 writes the normaliser as 1n\frac{1}{n} with the index range 1≤i≤m, m<j≤n1 \le i \le m,\ m < j \le n (the first mm documents are positives, the rest negatives). The in-class Exercise 4 solution divides by the number of positive–negative pairs, which is the form shown above. Clip each pair at zero before averaging.
−1 0 1 2 0 1 2 loss score gap Δ = Rel(q, d⁺) − Rel(q, d⁻) margin = 1 Case B: Δ = −0.6, loss 1.6 Case A: Δ = 0.7, loss 0.3 Δ = 1.3, loss 0
The margin (hinge) loss of one positive–negative pair falls as the positive's lead grows and switches off once the lead reaches the margin of 1.
Key ideaMargin loss and InfoNCE follow different philosophies. InfoNCE is probabilistic: it pushes the softmax probability of the positive towards 1, so every score always contributes a little gradient. Margin loss is threshold-based: it directly enforces a minimum separation (margin = 1) between relevant and irrelevant scores, and once a pair is separated by the margin it is ignored entirely. This "stop caring once it is good enough" property can lead to more stable training in some scenarios.

Worked example 5Margin loss on the two lecture cases

Margin = 1. Compute the loss contribution of a single positive–negative pair in each situation (values from slide 27).

1
Case A: the model is nearly right. Rel(q,d+)=0.8\mathrm{Rel}(q,d^+) = 0.8, Rel(q,d−)=0.1\mathrm{Rel}(q,d^-) = 0.1. The score difference 0.8−0.1=0.70.8 - 0.1 = 0.7 is below the margin of 1, so LA=max⁡(0, 1−0.7)=max⁡(0, 0.3)=0.3 \mathcal{L}_A = \max(0,\ 1 - 0.7) = \max(0,\ 0.3) = 0.3 A relatively small loss: the pair is close to the target separation, so it needs only a gentle push.
2
Case B: the model has it backwards. Rel(q,d+)=0.2\mathrm{Rel}(q,d^+) = 0.2, Rel(q,d−)=0.8\mathrm{Rel}(q,d^-) = 0.8 (the negative out-scores the positive). The difference is 0.2−0.8=−0.60.2 - 0.8 = -0.6, so LB=max⁡(0, 1−(−0.6))=max⁡(0, 1.6)=1.6 \mathcal{L}_B = \max(0,\ 1 - (-0.6)) = \max(0,\ 1.6) = 1.6 A much larger loss: the ranking error gets a strong corrective push.
3
Check the satisfied case. If Rel(q,d+)=1.5\mathrm{Rel}(q,d^+) = 1.5 and Rel(q,d−)=0.2\mathrm{Rel}(q,d^-) = 0.2, the difference is 1.3≥11.3 \ge 1, so L=max⁡(0, 1−1.3)=max⁡(0, −0.3)=0\mathcal{L} = \max(0,\ 1 - 1.3) = \max(0,\ -0.3) = 0. The pair contributes nothing; training effort goes to the pairs that still violate the margin.
ResultL=0.3\mathcal{L} = 0.3 for the near-correct case, 1.61.6 for the inverted case and 00 once the margin is met. The loss scales with how badly each pair is ordered, exactly the behaviour wanted from a ranking loss.
PracticeA RepBERT pair has Rel(q, d⁺) = 0.9 and Rel(q, d⁻) = 0.4. What is the margin loss with margin 1?
AnswerThe gap is 0.5, so the loss is max⁡(0, 1−0.5)=0.5\max(0,\ 1 - 0.5) = 0.5.
PracticeA query has one positive (score 1.0) and two negatives (scores 0.5 and −0.5). A classmate sums the raw values 1 − gap for both pairs and reports 0.5 + (−0.5) = 0. What went wrong?
AnswerThe clip at zero was skipped. The pairs and the averaged loss work out as follows: first pair:max⁡(0,1−0.5)=0.5second pair:max⁡(0,1−1.5)=0average:L=0.52=0.25 \begin{aligned} \text{first pair:}\quad \max(0, 1 - 0.5) &= 0.5 \\ \text{second pair:}\quad \max(0, 1 - 1.5) &= 0 \\ \text{average:}\quad \mathcal{L} &= \frac{0.5}{2} = 0.25 \end{aligned} A satisfied pair contributes exactly 0 and can never cancel another pair's loss.

9. ANCE: self-mined hard negatives

Sources: slides 28–30 · Video 4 (ANCE) · In-class slide 11 · In-class slide 25 (Exercise 5)

Xiong et al., "Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval", ICLR 2021. ANCE keeps DPR's representation, similarity and loss, and innovates purely on where the negatives come from.

Design choiceANCE
BackboneBERT encoder
Representation[CLS] token embedding
SimilarityInner (dot) product
LossNegative log likelihood (InfoNCE, Section 7.1)
NegativesSelf-mined hard negatives via regular checkpoints (Negative Contrastive Estimation)

9.1 The problem with BM25 negatives

DPR's hard negatives are hard for BM25. The model being trained is a dense retriever, and what confuses BM25 need not confuse the dense embedding space. Picture the embeddings on a 2-D map (slide 29):

dense embedding space (2-D sketch) q query vector relevant documents: near q BM25 negatives: mostly far easy for the dense model random negatives: far, mixed ANCE negatives: ranked high by the dense model itself: close to q, yet irrelevant Hard for this model ≠ hard for BM25
In the dense space, most BM25-retrieved negatives and random negatives sit far from the query, so the dense retriever separates them without learning much. The valuable negatives (blue) are the irrelevant documents the dense model itself currently ranks highly.

9.2 Mining negatives from checkpoints

How do we get "negatives that fool the current model" while the model is still training? ANCE's answer: ask a recent checkpoint of the model itself (slide 30).

1. Train the model about every 1,000 iterations 2. Save a checkpoint frozen copy of the weights 3. Retrieve with the checkpoint re-encode corpus, ANN search 4. Keep top-ranked, non-relevant = new hard negatives feed back as negatives repeat throughout training
ANCE's asynchronous loop: checkpoint, retrieve with the checkpoint, take highly ranked but non-relevant documents as negatives, continue training, repeat.
  1. Checkpoint the model regularly as it trains (for example every thousand iterations).
  2. Use the previous checkpoint to embed the corpus and retrieve for the training queries, with approximate nearest-neighbour search (the "ANN" in the name).
  3. From the results, take documents that are ranked highly but not relevant. These are exactly the errors the current model makes; use them as negatives.
  4. Iterate this process across the whole of training, refreshing the negatives as the model improves.
Watch outTrade-off. Empirically this yields better dense retrievers than DPR's in-batch + BM25 negatives. The price is much higher training cost: the corpus representations must be recomputed from each new checkpoint again and again during training. Query-time cost stays the same, because the result is still a plain bi-encoder.
PracticeWhy are BM25 hard negatives "hard" for a lexical reason but potentially "easy" for a dense retriever, and how does ANCE fix this?
AnswerBM25 negatives share keywords with the query, which makes them hard for a term-matching model. A dense retriever compares meanings, and in its embedding space many of these documents already sit far from the query, so they teach it little. ANCE mines negatives with a recent checkpoint of the dense model itself: the irrelevant documents that this model ranks highly, refreshed as training proceeds.
PracticeDoes ANCE make query-time retrieval slower than DPR? Explain.
AnswerNo. ANCE changes only how training negatives are chosen. The trained model is still a bi-encoder with one [CLS] vector per text and a dot product, so query-time cost matches DPR. The extra cost (re-encoding the corpus at each checkpoint) is paid during training.

10. Contriever: unsupervised dense retrieval

Sources: slides 31–32 · Video 5 (Contriever) · In-class slide 11

Izacard et al., "Unsupervised Dense Information Retrieval with Contrastive Learning", 2021.

Design choiceContriever
BackboneBERT encoder
RepresentationMean of token embeddings
SimilarityCosine similarity
LossNegative log likelihood (InfoNCE, the same loss as DPR and ANCE)
InnovationUnsupervised training: no labelled data needed

DPR and ANCE need human-annotated query–document relevance judgments. Contriever's key innovation is to manufacture its own training signal from any raw text collection:

Key ideaContriever's unsupervised recipe. Given a document collection D={d1,d2,…,dn}D = \{d_1, d_2, \dots, d_n\}, for each document did_i: Then train with the usual InfoNCE loss: one span plays the role of the "query", the other span from the same document is the positive, and spans of other documents are the negatives.
positive pair document dᵢ span s¹ᵢ span s²ᵢ negative pair another document dj (j ≠ i) span sᵏj span length: 5%–50% of the document (2–20 tokens for a 40-token document)
Contriever builds its own training pairs: two spans of one document are a positive pair, a span from another document is a negative.

Worked example 6Building Contriever training pairs from raw text

Take a document of 40 tokens: "The Great Barrier Reef stretches over 2,300 km off the coast of Queensland. It is the world's largest coral reef system, composed of nearly 3,000 individual reefs. Rising sea temperatures have caused repeated mass bleaching events in recent years."

1
Allowed span lengths: between 5% and 50% of the document length: shortest:0.05×40=2 tokenslongest:0.50×40=20 tokens \begin{aligned} \text{shortest:}\quad 0.05 \times 40 &= 2 \text{ tokens} \\ \text{longest:}\quad 0.50 \times 40 &= 20 \text{ tokens} \end{aligned}
2
Sample two spans from this document (random contiguous subsequences within that length range): s1s_1 = "stretches over 2,300 km off the coast of Queensland" (9 tokens) and s2s_2 = "repeated mass bleaching events" (4 tokens). These form a positive pair: same document, so they are assumed to be semantically related (both are about the Reef).
3
Negatives: a span sampled from a different document, for example sjks_j^k = "the quarterly earnings exceeded analyst expectations" from a finance article. The pair (s1,sjk)(s_1, s_j^k) is a negative pair.
4
Train: treat s1s_1 as the query, s2s_2 as the positive and other documents' spans (in practice, the other spans in the batch) as negatives, and minimise InfoNCE with cosine similarity on mean-pooled embeddings.
ResultAn unlimited supply of (query, positive, negatives) triples generated automatically from any text collection, with no human relevance labels anywhere in the pipeline.
PracticeA document has 200 tokens. What span lengths can Contriever sample, and why could two spans from the same document occasionally make a poor positive pair?
AnswerSpans can have these lengths: shortest:0.05×200=10 tokenslongest:0.5×200=100 tokens \begin{aligned} \text{shortest:}\quad 0.05 \times 200 &= 10 \text{ tokens} \\ \text{longest:}\quad 0.5 \times 200 &= 100 \text{ tokens} \end{aligned} The recipe assumes that spans of one document are related; a long document that changes topic (for example a news digest) can give two spans about unrelated things, which makes a noisy positive.

11. ColBERT: late interaction with MaxSim

Sources: slides 33–40 · Video 6 (ColBERT) · In-class slides 12–13

Khattab and Zaharia, "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT", SIGIR 2020. ColBERT is the week's one multi-vector model and the most prominent representative of the late-interaction family.

Design choiceColBERT
BackboneBERT encoder (query encoder and document encoder)
RepresentationAll token embeddings (L2-normalised), per query and per document
SimilarityMaxSim (late interaction)
LossNegative log likelihood (the same family as DPR, ANCE and Contriever)

11.1 The MaxSim scoring function, step by step

score(q,d)=∑i=1mmax⁡j=1,…,nEqi⋅Edj \mathrm{score}(q,d) = \sum_{i=1}^{m} \max_{j = 1,\dots,n} E_{q_i} \cdot E_{d_j}
Where
mm
the number of query tokens
nn
the number of document tokens
EqiE_{q_i}
the L2-normalised embedding of query token ii
EdjE_{d_j}
the L2-normalised embedding of document token jj
Eqi⋅EdjE_{q_i} \cdot E_{d_j}
their dot product (a cosine, because both vectors have length 1); in-class slide 13 writes it EqiEdjTE_{q_i}E_{d_j}^{T}
max⁡j\max_{j}
the best match for query token ii anywhere in the document
Read it aloud The score of d for q is the sum, for i from 1 to m, of the maximum over j from 1 to n of E-q-i dot E-d-j.
Formula in plain words For each query token, compute its similarity with every document token and keep only the maximum: the best match that query token found anywhere in the document. Then add up those per-token maxima. Each query token therefore gets its own chance to be matched, at whatever position suits it. Adding a strongly matching document token raises the score only if it beats that query token's previous best; extra weak tokens in a long document change nothing, because the max ignores them. Longer queries have more terms in the sum, so scores are compared within one query.

The lecture walks this through in five steps for query tokens tq1,…,tqmt_{q_1}, \dots, t_{q_m} and document tokens td1,…,tdnt_{d_1}, \dots, t_{d_n} (slides 34–38):

  1. Step 1: take query token tq1t_{q_1}; compute the dot product of its embedding with the first document token embedding td1t_{d_1}, giving one similarity score.
  2. Step 2: still for tq1t_{q_1}, compute its similarity with every other document token td2,td3,…,tdnt_{d_2}, t_{d_3}, \dots, t_{d_n}, giving a full list of scores for tq1t_{q_1}.
  3. Step 3: from that list, keep the maximum: the best match tq1t_{q_1} found anywhere in the document. If a query token matches strongly with any part of the document, that is what matters; the scores are left un-averaged.
  4. Step 4: repeat steps 1–3 for every remaining query token tq2,…,tqmt_{q_2}, \dots, t_{q_m}, giving one maximum per query token.
  5. Step 5: sum the maxima. Each query token contributes its best matching score to the total.
similarity matrix S: query tokens × document tokens d₁ d₂ d₃ d₄ max per row q₁ q₂ q₃ 0.90.20.40.1 0.30.80.50.2 0.10.40.20.6 0.90.80.6 Σ = 2.3 score(q, d) each row keeps only its best match (max), then the row maxima are summed
MaxSim on a 3-query-token × 4-document-token example: maximum along each row, then the sum of the maxima. Worked example 7 uses the same numbers.

Worked example 7ColBERT MaxSim by hand

Illustrative values. A query has m=3m = 3 token embeddings and a document n=4n = 4. All token vectors are L2-normalised, so each dot product behaves like a cosine similarity. The pairwise similarities Sij=Eqi⋅EdjS_{ij} = E_{q_i} \cdot E_{d_j} are:

S=(0.90.20.40.10.30.80.50.20.10.40.20.6) S = \begin{pmatrix} 0.9 & 0.2 & 0.4 & 0.1 \\ 0.3 & 0.8 & 0.5 & 0.2 \\ 0.1 & 0.4 & 0.2 & 0.6 \end{pmatrix}
1
Query token 1 (row 1): scores against the 4 document tokens are 0.9, 0.2, 0.4, 0.1. Its best match anywhere in the document: max⁡(0.9,0.2,0.4,0.1)=0.9\max(0.9, 0.2, 0.4, 0.1) = 0.9.
2
Query token 2 (row 2): max⁡(0.3,0.8,0.5,0.2)=0.8\max(0.3, 0.8, 0.5, 0.2) = 0.8; its best match is document token 2.
3
Query token 3 (row 3): max⁡(0.1,0.4,0.2,0.6)=0.6\max(0.1, 0.4, 0.2, 0.6) = 0.6; its best match is document token 4.
4
Sum the maxima: score(q,d)=0.9+0.8+0.6=2.3 \mathrm{score}(q,d) = 0.9 + 0.8 + 0.6 = 2.3
5
Count the work: this took m×n=3×4=12m \times n = 3 \times 4 = 12 dot products for one document, where a bi-encoder like DPR needs exactly 1. For a realistic 32-token query against a 300-token passage that is 9,600 dot products per document.
Resultscore(q,d)=2.3\mathrm{score}(q,d) = 2.3. Each query token found its own best match at a different position in the document; this token-level flexibility is exactly the fine-grained signal that single-vector models compress away.

11.2 Why it works, and what it costs

Key ideaColBERT computes fine-grained interactions between all query-token/document-token pairs, aggregated via MaxSim, while keeping the interaction late: document token embeddings are pre-computed and stored offline, and query and document only meet at the cheap dot-product stage. It captures a richer amount of information, so it is often more effective than single-vector retrievers.
Watch outColBERT's problems (slide 40).

Worked example 8Index size: bi-encoder against ColBERT

Illustrative setting: 1,000,000 passages; every vector has 768 dimensions stored as 4-byte floats; ColBERT keeps 200 token vectors per passage (inside the 100–500 range on slide 40). Real systems shrink and compress token vectors, which the lecture does not cover; the point here is the ratio.

1
One vector: 768×4=3,072768 \times 4 = 3{,}072 bytes.
2
Bi-encoder index: 1,000,000×3,072≈3.11{,}000{,}000 \times 3{,}072 \approx 3.1 GB.
3
ColBERT index: 1,000,000×200×3,072≈6141{,}000{,}000 \times 200 \times 3{,}072 \approx 614 GB.
4
Scoring one passage for a 32-token query: 1 dot product for the bi-encoder; 32×200=6,40032 \times 200 = 6{,}400 dot products for ColBERT.
ResultColBERT needs about 200 times the storage (614 GB against 3.1 GB) and 6,400 times the similarity operations per passage in this setting. This is the cost side of its better effectiveness.
PracticeIn ColBERT, why take the max over document tokens and leave the mean aside? What would averaging do to a long document that contains one highly relevant sentence?
AnswerThe max rewards a query token for finding one strong match anywhere in the document, which is what relevance needs. Averaging would dilute that strong match with the many unrelated tokens of a long document, so a long document with one highly relevant sentence would score low, penalising it for its length.
PracticeA query token row of the similarity matrix is 0.2, 0.7, 0.7, 0.1. What does it contribute to the MaxSim score, and does it matter that the maximum appears twice?
AnswerIt contributes 0.7. The maximum is taken once per query token, so a second equally good match adds nothing.

12. E5-Mistral: LLM embeddings and synthetic data

Sources: slides 41–43 · Video 7 (E5-Mistral) · In-class slide 16

Wang et al., "Improving Text Embeddings with Large Language Models", 2023 (arXiv 2401.00368). E5 is a family of embedding models. The original E5 embeddings are BERT-based; E5-Mistral replaces that encoder backbone with a decoder-only LLM (Mistral-7B) and uses LLMs in two distinct ways.

12.1 Use 1: synthetic training data generated by an LLM

  1. Brainstorm tasks: prompt an LLM (GPT-4) to brainstorm a list of potential retrieval tasks, for example "search web pages about a topic", "find academic references for an assignment", "find a tutorial video".
  2. Generate triples: for each brainstormed task, prompt the LLM to generate a ⟨query, positive document, hard-negative document⟩ triple, with formatting instructions based on that task definition. Repeat many times to build a large synthetic training set with no human annotation.
1. Brainstorm GPT-4 lists retrieval tasks 2. Generate ⟨q, p⁺, hard p⁻⟩ for each task Synthetic data no human annotation Fine-tune + MS MARCO labels contrastive loss
Use 1: an LLM writes the training data (slide 41, part A).

12.2 Use 2: the LLM as the embedding backbone

"Instruct: {task} Query: {q}" + [EOS] Mistral-7B decoder-only LLM last hidden state of [EOS] dense vector → ANN index documents take the same path without the instruction prefix: document text + [EOS] → embedding of [EOS]
E5-Mistral's embedding path: an instruction-prefixed query (or a plain document) with an appended [EOS] token goes through the LLM; the last layer's [EOS] vector is the embedding (slide 42).
  1. Modify the query to include an instruction, the task definition brainstormed during data generation: q_inst = "Instruct: {task_definition} \n Query: {q}"
  2. Append [EOS] to the end of the query and of the document, feed each into the LLM, and take the last layer's [EOS] vector as the embedding. The [EOS] representation is conditioned on all previous tokens (causal attention), so it summarises the entire input (Section 4.2).
  3. Train with the standard contrastive (InfoNCE) loss over in-batch negatives and the generated hard negatives.

12.3 Training stages and the key observation

Key ideaKey observation. For E5-Mistral, the contrastive pre-training stage, essential for BERT-based backbones, has negligible impact on quality. The extensive auto-regressive pre-training that created the LLM already taught it good text representations; only fine-tuning is needed, and what it adds is mainly task knowledge (how to embed for retrieval) on top of existing text knowledge.
PracticeName the two separate ways E5-Mistral uses LLMs, and say which one would still be needed if you had plenty of human-labelled training data.
Answer(A) An LLM (GPT-4) generates synthetic training triples; (B) an LLM (Mistral-7B) is the embedding backbone. With plenty of labelled data, use A becomes optional (RepLLaMA shows this), while use B is the model itself and is always needed.
PracticeWhy does E5-Mistral append [EOS] to both queries and documents?
AnswerWith causal attention the final position is the only one that has attended to the whole input. Appending [EOS] gives a fixed final position whose last-layer vector summarises the full text, and using it for both sides keeps queries and documents in the same representation space.

13. LLM2Vec: turning any decoder into an encoder

Sources: slides 44–46 · Video 8 (LLM2Vec) · In-class slide 16

BehnamGhader et al., "LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders", 2024. LLM2Vec is a general recipe to transform any decoder-only LLM into a strong text encoder in three steps, plus an optional fourth:

Step 1 bidirectional attention Step 2 masked next-token prediction (MNTP) Step 3 unsupervised SimCSE Step 4 (optional) supervised contrastive result: a bidirectional text encoder built from a decoder-only LLM
LLM2Vec's recipe as a pipeline (slides 44–46).
  1. Enable bidirectional attention. Replace the causal attention mask with an all-ones matrix, so each token can attend to every other token: the decoder becomes bidirectional like an encoder. Doing only this decreases embedding performance, because the model's weights were never trained to attend to future tokens.
  2. Masked next-token prediction (MNTP). Make the model aware of its new bidirectional attention: mask a token in the sequence and train the model to predict it. The twist: when predicting the masked token at position ii, the loss is computed from the logits at position i−1i - 1, staying faithful to the decoder's next-token habit. Slide 45 adds the motivation: decoder pre-training has no next-sentence-prediction objective, so the model was never explicitly trained to capture the context of the entire sequence.
  3. Unsupervised contrastive learning (SimCSE). Pass the same input sequence through the model twice with independently sampled dropout masks. Dropout randomness makes the two outputs slightly different: two "views" of one sentence. Train to maximise the similarity between the two views and minimise similarity to the other sequences in the batch (an InfoNCE-style loss, Section 7.1). No labels required.
  4. (Optional) Supervised contrastive learning on labelled retrieval or sentence-similarity data for a further boost.
causal mask (before) all-ones mask (after) thecatsatdown thecatsatdown thecatsatdown thecatsatdown 1000 1100 1110 1111 1111 1111 1111 1111 step 1 row i sees columns ≤ i (10 ones) every token sees every token (16 ones) Step 1 alone lowers quality: the weights never learned to use future tokens
LLM2Vec step 1 flips the causal mask (row = token doing the attending) to all ones; step 2 (MNTP) then teaches the weights to use the new connections.

Worked example 9What changes when the mask flips, and how MNTP works

Take the 4-token sequence the cat sat down.

1
Causal attention (original decoder). Row ii of the mask says what token ii may look at. Token 2 ("cat") sees only ⟨"the", "cat"⟩, so its hidden vector cannot contain any information about "sat" or "down". As a mask matrix (1 = may attend): Mcausal=(1000110011101111) M_{\text{causal}} = \begin{pmatrix} 1 & 0 & 0 & 0 \\ 1 & 1 & 0 & 0 \\ 1 & 1 & 1 & 0 \\ 1 & 1 & 1 & 1 \end{pmatrix}
2
Flip to bidirectional (LLM2Vec step 1). The mask becomes the all-ones matrix, 16 ones in place of 10, and now "cat"'s vector can also encode its future context: Mbi=(1111111111111111) M_{\text{bi}} = \begin{pmatrix} 1 & 1 & 1 & 1 \\ 1 & 1 & 1 & 1 \\ 1 & 1 & 1 & 1 \\ 1 & 1 & 1 & 1 \end{pmatrix} The pretrained weights have never used those new connections, so quality initially drops.
3
MNTP (step 2). Mask position 3: the cat [MASK] down. The model must predict "sat", and the loss is computed from the logits at position 3−1=23 - 1 = 2 (the representation of "cat"). Solving this forces the weights to exploit attention to future tokens ("down" is to the right of the mask), making the bidirectional attention useful.
4
SimCSE (step 3). Encode the cat sat down twice with different dropout, giving vectors zz and z′z'. Train with a contrastive loss so that ϕ(z,z′)\phi(z, z') is high while ϕ(z,zother)\phi(z, z_{\text{other}}) is low for every other sentence in the batch: sentence-level embedding ability, learned with zero labels.
ResultAfter the three steps, a generation-only LLM behaves like a strong bidirectional text encoder: the "secretly powerful text encoder" of the paper's title.
PracticeWhy does enabling bidirectional attention alone (LLM2Vec step 1) hurt performance, and which step repairs it?
AnswerThe decoder's weights were trained only with causal attention, so they have never learned how to use information from future tokens; suddenly exposing those connections gives the layers inputs they were never trained on. Step 2, masked next-token prediction, adapts the weights so the model learns to use its new bidirectional attention.
PracticeFor a 6-token sequence, how many ones are in the causal mask and in the all-ones mask?
Answercausal:1+2+⋯+6=21all-ones:6×6=36 \begin{aligned} \text{causal:}\quad 1 + 2 + \dots + 6 &= 21 \\ \text{all-ones:}\quad 6 \times 6 &= 36 \end{aligned}

14. RepLLaMA: supervised LLM retrieval with LoRA

Sources: slides 47–48 · Video 9 (RepLLaMA) · In-class slide 16

Ma et al., "Fine-Tuning LLaMA for Multi-Stage Text Retrieval", 2023. RepLLaMA uses the same architectural setup as E5-Mistral (decoder-only backbone, instruction-prefixed query, appended [EOS], embedding = last hidden state of [EOS]) with Llama-2-7B as the backbone.

Design choiceRepLLaMA
BackboneLlama-2-7B (decoder-only, open source)
Representation[EOS] token embedding (last hidden state)
SimilarityDot product
LossNegative log likelihood / InfoNCE (the same as DPR and ANCE)
TrainingSupervised labels only (like DPR) + LoRA adapters

Two practical insights distinguish it:

The lecture names LoRA without the maths. As background, the standard LoRA update for one weight matrix is:

W=W0+BA,B∈Rd×r,A∈Rr×k,r≪min⁡(d,k) W = W_0 + BA,\qquad B \in \mathbb{R}^{d \times r},\quad A \in \mathbb{R}^{r \times k},\quad r \ll \min(d,k)
Where
W0W_0
the pretrained d×kd \times k weight matrix of one projection layer, kept frozen
A, BA,\ B
the two small trainable adapter matrices
rr
the rank of the adapter (for example 8 or 16), much smaller than dd and kk
BABA
their product: a d×kd \times k low-rank correction added to W0W_0
WW
the effective weight used in the forward pass
Read it aloud W equals W-zero plus B times A, where B is a d by r matrix, A is an r by k matrix, and r is much smaller than the smaller of d and k.
Formula in plain words Keep the huge pretrained matrix fixed and learn only a correction built from two thin matrices. The trainable count per matrix drops from d×kd \times k to r(d+k)r(d + k), so memory and training time fall sharply. A larger rank rr lets the correction express more change (closer to full fine-tuning) at a higher cost; a smaller rr is cheaper but more restricted.
x W₀ (frozen) d × k pretrained weights A B + h trainable: r(d + k) weights
LoRA keeps the pretrained weights frozen and trains a small low-rank side path whose output is added in.

Worked example 10How much smaller is a LoRA update?

Illustrative: one 4096×40964096 \times 4096 projection matrix (Llama-2-7B's hidden size is 4096) and LoRA rank r=8r = 8.

1
Full fine-tuning of this matrix: 4096×4096=16,777,2164096 \times 4096 = 16{,}777{,}216 weights.
2
LoRA: AA has 8×40968 \times 4096 and BB has 4096×84096 \times 8 weights, so r(d+k)=8×8192=65,536r(d + k) = 8 \times 8192 = 65{,}536 weights.
3
Ratio: 65,53616,777,216≈0.0039 \frac{65{,}536}{16{,}777{,}216} \approx 0.0039
ResultLoRA trains about 0.39% of the weights of that matrix, which is why RepLLaMA can fine-tune a 7-billion-parameter model on retrieval labels at a practical cost.
Key ideaE5-Mistral and RepLLaMA in one line. Same embedding mechanics (instruction + [EOS], last hidden state, contrastive loss); E5-Mistral invests in LLM-generated synthetic training data, RepLLaMA invests in parameter-efficient supervised fine-tuning (LoRA on labelled data only).
PracticeWhat do E5-Mistral and RepLLaMA share, and on which design dimension do they differ most?
AnswerThey share a decoder-only LLM backbone, the instruction-prefixed query, the [EOS] last-hidden-state embedding and a contrastive (InfoNCE) loss. They differ most in the training signal (positive/negative sampling): E5-Mistral generates synthetic triples with GPT-4 (plus MS MARCO), while RepLLaMA trains on supervised labels only, using LoRA for efficiency.
PracticeDoes LoRA make a RepLLaMA query faster to encode at search time?
AnswerNo. The lecture presents LoRA as a training-time efficiency: it reduces the number of weights updated during fine-tuning. At search time the query still passes through the full 7B-parameter model.

15. Model comparison summary

Sources: slides 18–48 · Videos 2–9 · In-class slides 10–12, 16

ModelBackboneRepresentationSimilarityLossNegatives / training signal
DPR (2020)Encoder[CLS] embeddingDot productInfoNCE (negative log likelihood)Random + BM25 hard + in-batch
RepBERT (2020)EncoderMean of token embeddingsInner productMulti-label margin lossIn-batch
ANCE (2021)Encoder[CLS] embeddingInner productInfoNCESelf-mined hard negatives via ANN over training checkpoints
Contriever (2021)EncoderMean of token embeddingsCosine similarityInfoNCEUnsupervised: two spans of the same document = positive pair
ColBERT (2020)Encoder (multi-vector)All token embeddingsMaxSim (late interaction)InfoNCEStandard triples (as in DPR-style training)
E5-Mistral (2023/24)Decoder, Mistral-7B[EOS] last hidden state, instruction-prefixed queryCosine/dot (contrastive)InfoNCEGPT-4 synthetic ⟨q, p⁺, hard p⁻⟩ + MS MARCO; in-batch + hard
LLM2Vec (2024)Decoder (any LLM)Mean pooling over the bidirectional LLMCosine (contrastive)SimCSE contrastive (+ optional supervised)Unsupervised: dropout views of the same text (+ optional labels)
RepLLaMA (2023)Decoder, Llama-2-7B[EOS] last hidden state, instruction-prefixed queryDot productInfoNCESupervised labels only (MS MARCO), trained with LoRA adapters
Dense retrievers Encoder backbone (BERT) Decoder backbone (LLM) Single vector DPR, ANCE, RepBERT, Contriever Multi vector ColBERT late interaction Single vector E5-Mistral, LLM2Vec, RepLLaMA
This week's eight models by backbone and number of vectors per text (slide 3 and slide 17).
PracticeWhich two models in the table use cosine similarity on mean-pooled vectors, and what else do they have in common?
AnswerContriever and LLM2Vec. Both can be trained without human relevance labels: Contriever uses spans of the same document as positives, LLM2Vec uses two dropout views of the same text (SimCSE).
PracticeWhich model is the only one this week trained with a margin loss rather than InfoNCE, and which is the only multi-vector model?
AnswerRepBERT (multi-label margin loss); ColBERT (all token embeddings, MaxSim).

16. Course context from the in-person lecture

Sources: In-class slides 2–7 · slides 4–6

The in-person session opened by placing this week on the course's "historical view of ranking models" timeline, which shows how the weeks fit together and what is examinable where:

1950s–90s term-based lexical models Week 3 Weeks 7, 11, 12 2000s–10s feature-based learning to rank (LTR) 2010s word2vec, RNN, LSTM not covered Week 4 2017–18 Transformer: BERT, GPT WE ARE HERE 2018–22 dense retrievers, learned sparse, rerankers Weeks 5, 6 (+ 8) Parts of 5, 9, 10 2022 onwards AI age: ChatGPT, Mistral, Claude, Qwen The future LLM rerankers, RAG, search agents Parts of 5, 9, 10 "The field is moving very fast: we focus on core ideas and the most impactful techniques."
The ranking-models timeline from the in-person lecture (in-class slide 3), with course weeks mapped onto it. Week 5 (dense retrievers) and Week 6 (learned sparse) sit in the "new inventions" era; the LLM-based models of this week already reach into the AI age.

The lecturer also stressed how the weeks build on each other: weeks 1, 2 and 4 are preliminary knowledge; weeks 3, 5, 6, 7 and 8 are the core models (this week is one of the core weeks); weeks 9–10 cover new research advances; weeks 11–13 remain important. In the relevance-model cost table, the last empty slot next to this week's models belongs to learned sparse retrievers, next week's topic. The session also reported the course survey (average 4.54 out of 5 from 24 responses) and the request that lectures include concepts as well, which led to the transformer recap in Section 17.

16.1 The models at a glance: years and one-line innovations

The in-person slides summarised every model as a card: representation, similarity, loss, plus the one-line "innovation" that makes it worth knowing. This table is a good revision checklist:

Model (year)Card (representation · similarity · loss)Innovation one-liner
DPR (2020)[CLS] embedding · dot product · InfoNCEThe influential dual-encoder model.
RepBERT (2020)Mean embedding · dot product · multi-label margin lossStrong early BERT dual-encoder baseline; mean pooling + margin loss beats BM25 on MS MARCO.
ANCE (2021)[CLS] embedding · dot product · InfoNCEAsynchronous negative sampling from an evolving index during training.
Contriever (2021)Mean embedding · cosine similarity · InfoNCETrained without labels on massive corpora with contrastive learning.
ColBERT (2020)All token embeddings · MaxSim · InfoNCEEfficient late interaction: token-level query/document embeddings scored with MaxSim.
E5-Mistral (2024)[EOS] embedding · dot product · InfoNCEGPT-4-generated synthetic supervised data + MS MARCO labels.
LLM2Vec (2024)Mean pooling · dot product · InfoNCEBidirectional attention + masked next-token prediction + unsupervised SimCSE; supervised tuning optional.
RepLLaMA (2023/24)[EOS] embedding · dot product · InfoNCELLaMA bi-encoder trained on MS MARCO with hard and in-batch negatives.
Watch outLLM2Vec's representation. The in-person model card lists LLM2Vec with mean pooling. After its three steps make the LLM bidirectional, every token vector carries full context, so averaging them (as RepBERT and Contriever do for BERT) becomes meaningful again. E5-Mistral and RepLLaMA keep causal attention and therefore use the [EOS] last-token embedding, the only position that has seen the whole input. The model cards and the main slides also differ on a few similarity functions (for example dot product on the card, cosine in the comparison table); either is accepted as long as you can explain the choice.

Grouping remarks from the lecture worth remembering: DPR and RepBERT are the early series (2020); ANCE's card carries the takeaway that negative sampling matters during training; and Contriever's card carries the takeaway that these models can be trained without supervised labels.

PracticeWhich weeks did the lecturer call the "core models", and which two of them cover neural first-stage retrievers?
AnswerWeeks 3, 5, 6, 7 and 8. Week 5 (dense, embedding-based retrievers) and Week 6 (learned sparse retrievers).

17. Transformer recap from the in-person session

Sources: In-class slides 9, 14–15 · Week 4 lecture notes

Responding to survey feedback ("lecture should include concepts as well"), the in-person session re-taught the transformer pipeline from Week 4 before the exercises. A condensed version of that recap, since everything this week builds on it:

αij=softmaxj ⁣(qi⋅kjd),oi=∑jαij vj \alpha_{ij} = \mathrm{softmax}_j\!\left(\frac{\mathbf{q}_i \cdot \mathbf{k}_j}{\sqrt{d}}\right),\qquad \mathbf{o}_i = \sum_{j} \alpha_{ij}\, \mathbf{v}_j
Where
qi\mathbf{q}_i
the query vector of token ii (the token doing the attending)
kj, vj\mathbf{k}_j,\ \mathbf{v}_j
the key and value vectors of token jj
dd
the dimension of the query and key vectors; d\sqrt{d} keeps the dot products in a stable range
softmaxj\mathrm{softmax}_j
normalises over all tokens jj that token ii may see, so the weights are positive and sum to 1
αij\alpha_{ij}
the attention weight token ii gives to token jj
oi\mathbf{o}_i
the output vector of token ii
Read it aloud Alpha i j equals the softmax over j of q-i dot k-j divided by the square root of d; and o-i equals the sum over j of alpha i j times v-j.
Formula in plain words Token ii compares its query with every key to decide how much each other token matters, turns those comparisons into weights that sum to 1, and outputs the weighted sum of the value vectors. A key that matches the query closely gets a large weight and pulls the output towards its value. In an encoder jj ranges over all tokens; in a decoder only over j≤ij \le i (the causal mask of Section 13). Because the weights depend on qi\mathbf{q}_i, token ii's attention to jj generally differs from jj's attention to ii.
token E([CLS]), 768 + segment E(A), 768 + position E(pos 0), 768 = input vector still 768 values element-wise addition: dimension 1 + dimension 1 + dimension 1, … example: the [CLS] token at position 0 of sentence A
A transformer's input vector for one token is the sum of three embeddings of the same size (the in-class exercise answer).
PracticeBERT-base uses 768-dimensional embeddings. What is the dimension of the input vector for the [CLS] token after adding its token, segment and position embeddings?
Answer768. The three vectors are added element by element, so the dimension stays the same; stacking them (which would give 3 × 768) is what the model avoids.

18. Q&A from the in-person lecture

Sources: In-class slides 21–22

The most valuable part of the live session was the discussion. These are the questions asked (by students or as in-class exercises) with the answers, cleaned up.

18.1 Transformer mechanics

Q: Is the self-attention matrix symmetric?
A: No. Attention is directional: the attention token ii pays to token jj need not equal the attention jj pays to ii. In the lecturer's words, "I care about that token; that token doesn't have to care about me." Each row is computed from a different query vector, so Aij≠AjiA_{ij} \ne A_{ji} in general.

Q (in-class MCQ): Which statements about the transformer are correct?
A: "The encoder uses masked attention to prevent looking at future tokens": false (that describes the decoder; the encoder must see the whole sentence to capture its meaning). "The decoder uses bidirectional attention": false (swapped again). "The encoder processes input tokens in parallel using bidirectional attention": true (and this parallel processing is exactly why transformers exploit GPUs so well). "Both encoder and decoder use the same type of attention": false (bidirectional against causal).

Q: Is it true that BERT has more layers than GPT? Do they use different tokenisation?
A: Layer count is a free design choice. In practice BERT (about 110M parameters) is far smaller than modern GPT-style models, but you could build an encoder with more layers. Generation is the harder task, which is why decoder models grew to billions or trillions of parameters (the "scaling law"). Both families use word-piece-style tokenisation; what differs is the trained vocabulary (size and contents), because each is estimated from a different collection with different goals.

Q: In the exercise, how do the three embeddings combine: into a matrix?
A: By vector addition: first dimension + first dimension + first dimension, and so on. Each token ends with a single vector of the model dimension.

Q: What exactly does the decoder's linear layer do?
A: It is a learned n×mn \times m projection from the model dimension nn to the vocabulary size mm. Multiplying the last token's hidden vector by it gives a score (logit) for every vocabulary token; softmax converts these to probabilities between 0 and 1; the next token is picked from the top of that distribution. Because deployed systems sample from the distribution (always taking the argmax would be deterministic), the same prompt can produce different generations.

18.2 Using encoders and decoders for ranking (in-class Exercises 1 and 2)

Q: For a transformer encoder, what are the ways to obtain a text representation, and how is similarity computed?
A: Three ways, the answer to Exercise 1: use the [CLS] vector (DPR, ANCE), mean-pool the token vectors (RepBERT, Contriever), or keep multiple token vectors (ColBERT). For the single-vector options, similarity is one dot product or cosine between the query vector and the document vector. For the multi-vector option, similarity is late interaction: compare every query token embedding with every document token embedding, take each query token's maximum, and sum the maxima (MaxSim). A student proposed exactly this pairwise-then-aggregate scheme in class, and the lecturer confirmed it is essentially how ColBERT started.

Q: How can a transformer decoder be used to compute query–document similarity? (Exercise 2; several answers were collected in class, all valid.)

Decoder LLM 1. Embedder last-token vector, dot product E5-Mistral, RepLLaMA 2. Generative ranker prompt: pointwise, pairwise or listwise ordering 3. Expansion (doc2query) generate terms at indexing time, then BM25 matches 4. Output probabilities vocabulary distribution as term weights (Week 6 preview)
The four answers to Exercise 2 collected in class: a decoder can embed, rank by prompting, expand documents, or supply term weights.

Q: Why is expansion done at indexing time rather than query time?
A: Latency. The user is waiting at query time, so heavy generation there hurts the experience; documents are processed offline at indexing time where extra minutes cost nothing. This is the same logic as pre-computing document embeddings in a bi-encoder.

18.3 Beyond this week

Q: Can an encoder backbone be used for generation?
A: Yes: this is the idea behind diffusion language models, a very recent research direction. The encoder was trained on masked-token prediction, so append, say, 256 [MASK] tokens to the prompt and predict them all simultaneously (parallel, GPU-friendly) in place of one at a time. The catch is that independently predicted tokens can be incoherent; the fix, borrowed from image diffusion, is to run a few denoising/refinement passes, still far fewer passes than generating 256 tokens autoregressively. The lecturer noted his own recent work applies diffusion language models to retrieval.

Q (assignment logistics): What should a gain–loss plot look like?
A: One bar per query: the evaluation score of System A minus System B on that query. Typically sorted from largest gain to largest loss, so you can see at a glance whether A consistently beats B and, more importantly, which queries A loses on; those are the queries to analyse. The BM25 baseline in the assignment should be your own implementation (check the criteria; ask on the discussion board if unclear).

PracticeGive two ways to use a decoder LLM for ranking that avoid running it at query time over every document.
Answer(1) Use it as an embedder: document embeddings are computed offline and only the query is encoded online. (2) Use it for document expansion (doc2query) at indexing time, then rank with BM25. (Output-probability term weights, previewed for Week 6, can also be computed offline for documents.)

19. In-class exercises and where the solutions live

Sources: In-class slides 21–25 · Practical solution pages for Exercises 3–5

The session ran five exercises. Exercises 1–2 are conceptual and answered in the Q&A section above; Exercises 3–5 are computational or challenge questions with full step-by-step solutions saved in the Practical/ folder:

ExerciseQuestionAnswerSolution
1: Encoder-basedWays to obtain a text representation from an encoder; which tokens; how similarity is computed[CLS] / mean / all tokens; dot or cosine, or MaxSimSection 18.2; Sections 4.1 and 11
2: Decoder-basedHow can a decoder compute query–document similarity?Last-token embedding; LLM as ranker; doc2query; output-probability representationsSection 18.2; Section 4.2
3: InfoNCE losssim(q, p⁺) = 0.9; negatives 0.3, 0.1, 0.4L=−ln⁡(0.3839)≈0.96\mathcal{L} = -\ln(0.3839) \approx 0.96worked solution
4: Margin lossRel(q, p⁺) = 1.5; negatives 0.3, 0.1, 0.9; margin 1L=0.43≈0.133\mathcal{L} = \frac{0.4}{3} \approx 0.133worked solution
5: ChallengeMine negatives harder than top-ranked BM25 documentsANCE: self-mined, checkpoint-refreshed negativesworked solution

Worked example 11Exercise 3 in brief: InfoNCE with three negatives

From in-class slide 23: sim(q,p+)=0.9\mathrm{sim}(q,p^+) = 0.9; negatives 0.3, 0.1, 0.40.3,\ 0.1,\ 0.4.

1
Exponentiate: e0.9≈2.4596e0.3≈1.3499e0.1≈1.1052e0.4≈1.4918 \begin{aligned} e^{0.9} &\approx 2.4596 \\ e^{0.3} &\approx 1.3499 \\ e^{0.1} &\approx 1.1052 \\ e^{0.4} &\approx 1.4918 \end{aligned}
2
Denominator: 2.4596+1.3499+1.1052+1.4918=6.40652.4596 + 1.3499 + 1.1052 + 1.4918 = 6.4065.
3
Probability of the positive: 2.45966.4065≈0.3839 \frac{2.4596}{6.4065} \approx 0.3839
L=−ln⁡(0.3839)≈0.957 \mathcal{L} = -\ln(0.3839) \approx 0.957
ResultL≈0.96\mathcal{L} \approx 0.96. The positive is only slightly ahead of the hardest negative (0.9 against 0.4), so the model gives it just 38% of the probability mass and the loss is high. The full solution page adds further checks.

Worked example 12Exercise 4 in brief: margin loss with three negatives

From in-class slide 24: Rel(q,p+)=1.5\mathrm{Rel}(q,p^+) = 1.5; negatives 0.3, 0.1, 0.90.3,\ 0.1,\ 0.9; margin 1.

1
Gaps: 1.5−0.3=1.21.5−0.1=1.41.5−0.9=0.6 \begin{aligned} 1.5 - 0.3 &= 1.2 \\ 1.5 - 0.1 &= 1.4 \\ 1.5 - 0.9 &= 0.6 \end{aligned}
2
Per-pair hinge: max⁡(0,1−1.2)=0max⁡(0,1−1.4)=0max⁡(0,1−0.6)=0.4 \begin{aligned} \max(0, 1 - 1.2) &= 0 \\ \max(0, 1 - 1.4) &= 0 \\ \max(0, 1 - 0.6) &= 0.4 \end{aligned}
3
Average over the 3 pairs:
L=0+0+0.43≈0.133 \mathcal{L} = \frac{0 + 0 + 0.4}{3} \approx 0.133
ResultL≈0.133\mathcal{L} \approx 0.133. Only the hardest negative (0.9) violates the margin, so all of the gradient goes into pushing it away.
PracticeIn Exercise 4, how low would the hardest negative's score need to fall for the loss to reach zero?
AnswerThe gap must be at least 1, so Rel(q,p3−)≤1.5−1=0.5\mathrm{Rel}(q,p_3^-) \le 1.5 - 1 = 0.5. At 0.5 or below every pair meets the margin and the loss is 0.
PracticeExercise 5: training with BM25 hard negatives is slow to improve. Describe a way to mine harder negatives.
AnswerUse the dense model itself, as ANCE does: periodically save a checkpoint, re-encode the corpus with it, retrieve for each training query, and take the top-ranked documents that are not labelled relevant as new negatives. Refresh them as training continues. These are the documents that currently fool the model, so they give the strongest learning signal (at the cost of repeated corpus re-encoding).

20. Summary and study checklist

Takeaways

Study checklist. You should be able to:

21. Video index

Week 5 YouTube playlist: IR Week 5. Transcripts are in week_5_video_transcripts.md. The in-person lecture slides are in Lecture-Slides/. (The file INFS7410_S2_2026_STL-transcript.txt in this folder is a copy of the Week 4 lecture transcript, so these notes do not cite it.)

VideoTopicLengthSections here
Video 1Overview: embedding-based retrievers18:341–6
Video 2DPR (Dense Passage Retrieval)11:105, 7
Video 3RepBERT04:268
Video 4ANCE03:439
Video 5Contriever03:0610
Video 6ColBERT07:115.2, 11
Video 7E5-Mistral07:5812
Video 8LLM2Vec04:3413
Video 9RepLLaMA03:0914

22. References