Embedding-based Retrieval Models (Dense Retrievers)
How transformer encoders and decoders become search engines: bi-encoders (DPR, RepBERT, ANCE, Contriever), late interaction (ColBERT) and LLM-based embedders (E5-Mistral, LLM2Vec, RepLLaMA). For each model the notes cover the representation, the similarity function, the training loss (InfoNCE or margin loss) and the negative-sampling strategy. Lecturer: Dr Shuai Wang, The University of Queensland.
1. The big picture: from lexical to dense retrieval
Sources: slides 2–3, 8 · Video 1 (Overview: embedding-based retrievers) · In-class slides 4, 7
Where we are in the course. Week 3 covered term-based lexical models such as BM25, which score a document by counting how its terms overlap with the query's terms. Week 4 covered the transformer architecture: encoders (BERT) read a whole sequence with bidirectional attention and produce one contextual vector per token; decoders (GPT) read left to right with causal attention and predict the next token. This week the two threads meet: transformers embed queries and documents as vectors, and retrieval ranks documents by vector similarity in place of term overlap.
This week's models, organised by their backbone architecture:
| Backbone | Representation style | Models |
|---|---|---|
| Transformer encoder | Single vector: one vector per query or document | DPR, ANCE, RepBERT, Contriever |
| Transformer encoder | Multi vector: one vector per token | ColBERT |
| Transformer decoder (LLM) | Single vector: last-token vector | E5-Mistral, LLM2Vec, RepLLaMA |
PracticeA user searches for "automobile insurance" and the best document only says "car cover". Why does BM25 miss it, and what does a dense retriever do differently?
2. The table of relevance models: costs
Sources: slides 4–6 · Video 1 (Overview: embedding-based retrievers) · In-class slide 6
Recall the "table of relevance models" from Week 3. Each family of models is characterised by two costs:
- Scorer cost: the work needed, per document, to turn representations into a relevance score once the query arrives (query time).
- Encoder cost: the work needed to turn a text into its representation (mostly done offline for documents).
| Model family | Examples | Scorer cost | Encoder cost |
|---|---|---|---|
| Term-based lexical models | BM25, TF-IDF | : cheap inverted-index lookups | : linear scan to count terms |
| Single-vector embedding-based retrievers | DPR, ANCE, RepBERT, Contriever, E5, LLM2Vec, RepLLaMA | : one dot product or cosine | : self-attention over all token pairs |
| Multi-vector embedding-based retrievers | ColBERT | : every query token against every document token | : same transformer encoding |
| Learned sparse retrievers | next week (Week 6) | (next week) | (next week) |
How to read the cost terms:
- : the encoding work grows with the square of the document length, because self-attention compares every token with every other token.
- : the scoring work per document is constant, a single vector operation, however long the document was.
- : the work grows with the query length times the document length, because ColBERT compares all token pairs.
PracticeWhy is the scorer cost of a bi-encoder O(1) per document while its encoder cost is O(|d|²), and why does the quadratic part leave query latency almost untouched?
PracticeWhich slot in the relevance-model cost table stays empty this week, and when is it filled?
3. Dense representations: embedders and similarity
Sources: slide 7 · Video 1 (Overview: embedding-based retrievers) · In-class slide 8
Every dense retriever has two ingredients. First, an embedder : a neural network (BERT, an LLM, …) that maps a text to a fixed-size vector. The same kind of embedder is applied to the query and the document , so both land in the same vector space. Second, a similarity function : a cheap function (dot product, cosine similarity, …) that compares the two vectors.
- the query text and the document text
- the embedder (eta): a neural network that maps a text to a fixed-size vector
- the query embedding and the document embedding (for example 768 numbers each)
- the similarity function (phi), for example inner product or cosine similarity
- one number used to rank for
The vectors are low-dimensional compared with a lexical vocabulary: typically 768 dimensions for BERT-based models and around 3,000–4,000 for decoder (LLM) based models, against tens of thousands of possible terms in a bag-of-words representation. Relevance is measured as geometric closeness in a learned semantic space, so "car" and "automobile" can end up with very similar embeddings even though they share no characters.
3.1 The two standard similarity functions
Dot product (inner product). You met vector operations in the Week 4 maths refresher.
- the query and document embeddings, and
- the number of dimensions of each vector (for example 768)
- the -th entry of the query vector and of the document vector
- add up over all dimensions
Cosine similarity. This is the same cosine used by the vector space model in Week 3, now applied to learned embeddings.
- the dot product from the formula above
- the length (L2 norm) of :
- the length (L2 norm) of
Worked example 1Dot product against cosine similarity
Illustrative, tiny 3-dimensional embeddings for a query and two documents:
PracticeTwo documents have embeddings pointing in exactly the same direction, but one vector is three times as long. Which similarity function ranks them equally, and which prefers the longer vector?
4. Two backbones: encoder vs decoder
Sources: slides 8–11, 14–16 · Video 1 (Overview: embedding-based retrievers) · In-class slides 9, 14–15
4.1 Encoder-only (BERT-style): one vector per token, then pool
A transformer encoder (Week 4) reads the whole input at once with bidirectional attention and outputs one contextual vector per token. Retrieval needs a representation of the whole text, so a model must decide how to go from many token vectors to a text representation. There are three standard strategies:
- [CLS] token vector: BERT's special first token was designed to represent the entire sequence. Simple and efficient. Used by DPR and ANCE.
- Mean of all token vectors: average the information across all positions; potentially captures more of the sequence. Used by RepBERT and Contriever.
- All token vectors: keep everything. This preserves the most fine-grained information but needs a custom similarity function, because one dot product is no longer enough. Used by ColBERT.
Mean pooling, written as a formula:
- the text being embedded (a query or a document)
- the number of tokens in
- the encoder's output vector for token (for BERT, 768 numbers)
- the single text vector, with the same number of dimensions as each
Worked example 2Mean pooling
Illustrative token vectors for a 3-token text (dimension 3 for readability):
4.2 Decoder-only (GPT-style): discard generation, keep the last-token vector
A decoder predicts the next token given the input, using causal attention (each token only sees earlier tokens). It was designed for generation, yet its internal layers still learn rich representations of the text. The trick for retrieval:
PracticeWhy would mean pooling over a causal decoder's token vectors be a weaker summary than the last-token vector?
PracticeCompute the mean-pooled vector of the token vectors [1, 0], [0, 1] and [2, 2].
5. Bi-encoders vs late interaction
Sources: slides 12–13, 17 · Video 1 (Overview: embedding-based retrievers) · Video 2 (DPR) · In-class slides 12–13
5.1 Single-vector retrievers: the bi-encoder architecture
Single-vector models use a bi-encoder (dual encoder): one encoder for the query and one for the document (often sharing weights), each producing a single vector. The score is one similarity computation.
5.2 Multi-vector retrievers: late interaction
ColBERT keeps every token vector of both query and document, and lets them interact only at scoring time: hence late interaction. This captures richer, token-level matching signals and is often more effective, but scoring is more expensive ( in place of ) and storage grows from 1 vector to hundreds of vectors per document. Section 11 walks through its MaxSim scoring step by step.
| Property | Bi-encoder (single vector) | Late interaction (multi vector) |
|---|---|---|
| Vectors stored per document | 1 | about 100–500 (one per token) |
| Scoring per document | : one dot product or cosine | : MaxSim over all token pairs |
| Matching signal | whole-text summary | token-level matches |
| Typical trade-off | faster, more storage-efficient | often more effective, more compute and storage |
PracticeWhy can a bi-encoder pre-compute document vectors offline, and what would break this if the query and document were fed into one encoder together (a cross-encoder)?
6. The five dimensions that differentiate models
Sources: slides 17–18 · Video 1 (Overview: embedding-based retrievers)
Every model this week is a different answer to the same five design questions. Keeping these in mind makes the whole week easy to organise:
| Design question | Possible answers seen this week |
|---|---|
| 1. Model backbone: encoder or decoder? | Encoder (DPR, ANCE, RepBERT, Contriever, ColBERT) or decoder LLM (E5-Mistral, LLM2Vec, RepLLaMA) |
| 2. Representation: how is a text turned into vector(s)? | [CLS] embedding · mean of token embeddings · all token embeddings · last-token/[EOS] embedding |
| 3. Similarity function: how are vectors compared? | Dot product · cosine similarity · late interaction (MaxSim) |
| 4. Loss function: how is ranking learned in fine-tuning? | Negative log likelihood (InfoNCE) · multi-label margin loss · SimCSE-style contrastive |
| 5. Positive/negative sampling: where does the training signal come from? | Random negatives · BM25 hard negatives · in-batch negatives · self-mined (ANCE checkpoint) negatives · same-document spans (Contriever) · LLM-generated synthetic triples (E5-Mistral) |
To train any of these models you need, for each training query, a positive document (known relevant) and a set of negative documents to contrast against. The model learns to score the positive above the negatives; once it can do that, its embeddings are useful for dense retrieval.
PracticeANCE and DPR share the backbone, representation, similarity and loss. On which of the five dimensions do they differ?
7. DPR: Dense Passage Retrieval
Sources: slides 19–25 · Video 2 (DPR) · In-class slides 10, 17–20
Karpukhin et al., "Dense Passage Retrieval for Open-Domain Question Answering", EMNLP 2020. DPR is the representative bi-encoder: it demonstrated that a trained dense retriever could match and outperform BM25. Its key design choices:
| Design choice | DPR |
|---|---|
| Backbone | BERT encoder (bi-encoder: separate query and passage encoders) |
| Representation | [CLS] token embedding |
| Similarity | Dot product (simple, faster than cosine) |
| Loss | Negative log likelihood (InfoNCE) |
| Negatives | Random + BM25 hard + in-batch |
7.1 The InfoNCE loss
The goal of the loss is to maximise the probability that the model picks the relevant passage out of a set of candidates. The in-class slide 17 summarises it as "encourages the positive passage to receive the highest score among candidates".
- the query embedding (DPR: the query's [CLS] vector)
- the positive (relevant) passage embedding
- the -th negative (irrelevant) passage embedding
- the number of negatives for this query
- the dot-product similarity
- the exponential function; it makes every term positive
- the negative natural logarithm, which turns a probability into a loss
- the loss for one training query; training minimises its average
Worked example 3Computing the DPR InfoNCE loss
Illustrative dot-product similarities for one query with one positive and two negatives:
7.2 The three-part negative sampling strategy
DPR's success depends heavily on which negatives it contrasts against. It combines three sources of increasing difficulty:
- Random negatives (easy). Sample passages uniformly from the whole corpus. For a query about machine learning, a random negative might be a cooking recipe. Useful for basic discrimination, but the model quickly finds them trivial and stops learning subtle distinctions; real search involves much harder cases.
- BM25 hard negatives. Retrieve the BM25 top- (for example top 1000) for the query, exclude anything labelled relevant, and sample negatives from that set. These passages share keywords with the query but are semantically irrelevant: query "how does machine learning work" against passage "machine learning applications in healthcare". They force the model to learn the difference between lexical similarity and true semantic relevance.
- In-batch negatives (medium, free). In a training batch with queries and positives : for , reuse and as negatives. They are already encoded for the other queries, so they cost nothing extra. Because they come from the same dataset and domain, they are harder than random negatives: "medium" difficulty. Slide 25 credits them with efficiency and a domain-specific gain.
Worked example 4Counting negatives in a DPR training batch
A training batch contains queries. Each query comes with 1 labelled positive passage and 1 BM25 hard negative, and all passages in the batch are shared as in-batch negatives. How many negatives does each query get, and how many passage encodings does the batch need?
PracticeCompute the InfoNCE loss for sim(q, p⁺) = 2.0 and two negatives with similarities 1.0 and 0.0.
PracticeWhy are in-batch negatives called "medium" difficulty, harder than random negatives but easier than BM25 hard negatives?
8. RepBERT: mean pooling and margin loss
Sources: slides 26–27 · Video 3 (RepBERT) · In-class slides 10, 17
Zhan et al., "RepBERT: Contextualized Text Embeddings for First-Stage Retrieval", arXiv 2020. RepBERT differs from DPR in two ways: how it pools token vectors, and how it computes the training loss.
| Design choice | RepBERT |
|---|---|
| Backbone | BERT encoder |
| Representation | Mean of all token embeddings (Section 4.1) |
| Similarity | Inner (dot) product |
| Loss | Multi-label margin loss |
| Negatives | In-batch (for efficiency) |
8.1 Multi-label margin loss
The in-class slide 17 states the aim: relevant documents should score higher than irrelevant documents by at least a fixed margin (here 1).
- the query
- the positive (relevant) and negative (irrelevant) documents for in the batch
- the number of positive–negative pairs, used to average
- one positive document and one negative document
- the relevance score: the dot product of the mean-pooled query and document vectors
- the margin: the minimum gap wanted between a positive and a negative score
- clips each pair's loss at zero (a hinge)
Worked example 5Margin loss on the two lecture cases
Margin = 1. Compute the loss contribution of a single positive–negative pair in each situation (values from slide 27).
PracticeA RepBERT pair has Rel(q, d⁺) = 0.9 and Rel(q, d⁻) = 0.4. What is the margin loss with margin 1?
PracticeA query has one positive (score 1.0) and two negatives (scores 0.5 and −0.5). A classmate sums the raw values 1 − gap for both pairs and reports 0.5 + (−0.5) = 0. What went wrong?
9. ANCE: self-mined hard negatives
Sources: slides 28–30 · Video 4 (ANCE) · In-class slide 11 · In-class slide 25 (Exercise 5)
Xiong et al., "Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval", ICLR 2021. ANCE keeps DPR's representation, similarity and loss, and innovates purely on where the negatives come from.
| Design choice | ANCE |
|---|---|
| Backbone | BERT encoder |
| Representation | [CLS] token embedding |
| Similarity | Inner (dot) product |
| Loss | Negative log likelihood (InfoNCE, Section 7.1) |
| Negatives | Self-mined hard negatives via regular checkpoints (Negative Contrastive Estimation) |
9.1 The problem with BM25 negatives
DPR's hard negatives are hard for BM25. The model being trained is a dense retriever, and what confuses BM25 need not confuse the dense embedding space. Picture the embeddings on a 2-D map (slide 29):
9.2 Mining negatives from checkpoints
How do we get "negatives that fool the current model" while the model is still training? ANCE's answer: ask a recent checkpoint of the model itself (slide 30).
- Checkpoint the model regularly as it trains (for example every thousand iterations).
- Use the previous checkpoint to embed the corpus and retrieve for the training queries, with approximate nearest-neighbour search (the "ANN" in the name).
- From the results, take documents that are ranked highly but not relevant. These are exactly the errors the current model makes; use them as negatives.
- Iterate this process across the whole of training, refreshing the negatives as the model improves.
PracticeWhy are BM25 hard negatives "hard" for a lexical reason but potentially "easy" for a dense retriever, and how does ANCE fix this?
PracticeDoes ANCE make query-time retrieval slower than DPR? Explain.
10. Contriever: unsupervised dense retrieval
Sources: slides 31–32 · Video 5 (Contriever) · In-class slide 11
Izacard et al., "Unsupervised Dense Information Retrieval with Contrastive Learning", 2021.
| Design choice | Contriever |
|---|---|
| Backbone | BERT encoder |
| Representation | Mean of token embeddings |
| Similarity | Cosine similarity |
| Loss | Negative log likelihood (InfoNCE, the same loss as DPR and ANCE) |
| Innovation | Unsupervised training: no labelled data needed |
DPR and ANCE need human-annotated query–document relevance judgments. Contriever's key innovation is to manufacture its own training signal from any raw text collection:
- Sample two random spans and from , each a contiguous subsequence of tokens, with length drawn between 5% and 50% of the document length.
- Positive pair: . Two spans of the same document are assumed to be semantically related.
- Negative pairs: with . Spans from different documents are assumed to be less related.
Worked example 6Building Contriever training pairs from raw text
Take a document of 40 tokens: "The Great Barrier Reef stretches over 2,300 km off the coast of Queensland. It is the world's largest coral reef system, composed of nearly 3,000 individual reefs. Rising sea temperatures have caused repeated mass bleaching events in recent years."
PracticeA document has 200 tokens. What span lengths can Contriever sample, and why could two spans from the same document occasionally make a poor positive pair?
11. ColBERT: late interaction with MaxSim
Sources: slides 33–40 · Video 6 (ColBERT) · In-class slides 12–13
Khattab and Zaharia, "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT", SIGIR 2020. ColBERT is the week's one multi-vector model and the most prominent representative of the late-interaction family.
| Design choice | ColBERT |
|---|---|
| Backbone | BERT encoder (query encoder and document encoder) |
| Representation | All token embeddings (L2-normalised), per query and per document |
| Similarity | MaxSim (late interaction) |
| Loss | Negative log likelihood (the same family as DPR, ANCE and Contriever) |
11.1 The MaxSim scoring function, step by step
- the number of query tokens
- the number of document tokens
- the L2-normalised embedding of query token
- the L2-normalised embedding of document token
- their dot product (a cosine, because both vectors have length 1); in-class slide 13 writes it
- the best match for query token anywhere in the document
The lecture walks this through in five steps for query tokens and document tokens (slides 34–38):
- Step 1: take query token ; compute the dot product of its embedding with the first document token embedding , giving one similarity score.
- Step 2: still for , compute its similarity with every other document token , giving a full list of scores for .
- Step 3: from that list, keep the maximum: the best match found anywhere in the document. If a query token matches strongly with any part of the document, that is what matters; the scores are left un-averaged.
- Step 4: repeat steps 1–3 for every remaining query token , giving one maximum per query token.
- Step 5: sum the maxima. Each query token contributes its best matching score to the total.
Worked example 7ColBERT MaxSim by hand
Illustrative values. A query has token embeddings and a document . All token vectors are L2-normalised, so each dot product behaves like a cosine similarity. The pairwise similarities are:
11.2 Why it works, and what it costs
- Massive storage overhead: about 100–500 vectors per document in place of 1, hundreds of times the index size of a bi-encoder.
- Scoring complexity: per document against for bi-encoders (with = query length, = document length), and MaxSim operations are more expensive than one dot product. Storage and scoring both need substantially more infrastructure.
Worked example 8Index size: bi-encoder against ColBERT
Illustrative setting: 1,000,000 passages; every vector has 768 dimensions stored as 4-byte floats; ColBERT keeps 200 token vectors per passage (inside the 100–500 range on slide 40). Real systems shrink and compress token vectors, which the lecture does not cover; the point here is the ratio.
PracticeIn ColBERT, why take the max over document tokens and leave the mean aside? What would averaging do to a long document that contains one highly relevant sentence?
PracticeA query token row of the similarity matrix is 0.2, 0.7, 0.7, 0.1. What does it contribute to the MaxSim score, and does it matter that the maximum appears twice?
12. E5-Mistral: LLM embeddings and synthetic data
Sources: slides 41–43 · Video 7 (E5-Mistral) · In-class slide 16
Wang et al., "Improving Text Embeddings with Large Language Models", 2023 (arXiv 2401.00368). E5 is a family of embedding models. The original E5 embeddings are BERT-based; E5-Mistral replaces that encoder backbone with a decoder-only LLM (Mistral-7B) and uses LLMs in two distinct ways.
12.1 Use 1: synthetic training data generated by an LLM
- Brainstorm tasks: prompt an LLM (GPT-4) to brainstorm a list of potential retrieval tasks, for example "search web pages about a topic", "find academic references for an assignment", "find a tutorial video".
- Generate triples: for each brainstormed task, prompt the LLM to generate a ⟨query, positive document, hard-negative document⟩ triple, with formatting instructions based on that task definition. Repeat many times to build a large synthetic training set with no human annotation.
12.2 Use 2: the LLM as the embedding backbone
- Modify the query to include an instruction, the task definition brainstormed during data generation:
q_inst = "Instruct: {task_definition} \n Query: {q}" - Append [EOS] to the end of the query and of the document, feed each into the LLM, and take the last layer's [EOS] vector as the embedding. The [EOS] representation is conditioned on all previous tokens (causal attention), so it summarises the entire input (Section 4.2).
- Train with the standard contrastive (InfoNCE) loss over in-batch negatives and the generated hard negatives.
12.3 Training stages and the key observation
- Contrastive pre-training (inherited from the original BERT-based E5): weakly supervised text pairs from filtered data, for example split a passage in two and treat one part as the query and the other as the relevant passage.
- Contrastive fine-tuning: LLM-generated synthetic data + MS MARCO labelled data (and, in one version, 18 further datasets).
PracticeName the two separate ways E5-Mistral uses LLMs, and say which one would still be needed if you had plenty of human-labelled training data.
PracticeWhy does E5-Mistral append [EOS] to both queries and documents?
13. LLM2Vec: turning any decoder into an encoder
Sources: slides 44–46 · Video 8 (LLM2Vec) · In-class slide 16
BehnamGhader et al., "LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders", 2024. LLM2Vec is a general recipe to transform any decoder-only LLM into a strong text encoder in three steps, plus an optional fourth:
- Enable bidirectional attention. Replace the causal attention mask with an all-ones matrix, so each token can attend to every other token: the decoder becomes bidirectional like an encoder. Doing only this decreases embedding performance, because the model's weights were never trained to attend to future tokens.
- Masked next-token prediction (MNTP). Make the model aware of its new bidirectional attention: mask a token in the sequence and train the model to predict it. The twist: when predicting the masked token at position , the loss is computed from the logits at position , staying faithful to the decoder's next-token habit. Slide 45 adds the motivation: decoder pre-training has no next-sentence-prediction objective, so the model was never explicitly trained to capture the context of the entire sequence.
- Unsupervised contrastive learning (SimCSE). Pass the same input sequence through the model twice with independently sampled dropout masks. Dropout randomness makes the two outputs slightly different: two "views" of one sentence. Train to maximise the similarity between the two views and minimise similarity to the other sequences in the batch (an InfoNCE-style loss, Section 7.1). No labels required.
- (Optional) Supervised contrastive learning on labelled retrieval or sentence-similarity data for a further boost.
Worked example 9What changes when the mask flips, and how MNTP works
Take the 4-token sequence the cat sat down.
the cat [MASK] down. The model must predict "sat", and the loss is computed from the logits at position (the representation of "cat"). Solving this forces the weights to exploit attention to future tokens ("down" is to the right of the mask), making the bidirectional attention useful.the cat sat down twice with different dropout, giving vectors and . Train with a contrastive loss so that is high while is low for every other sentence in the batch: sentence-level embedding ability, learned with zero labels.PracticeWhy does enabling bidirectional attention alone (LLM2Vec step 1) hurt performance, and which step repairs it?
PracticeFor a 6-token sequence, how many ones are in the causal mask and in the all-ones mask?
14. RepLLaMA: supervised LLM retrieval with LoRA
Sources: slides 47–48 · Video 9 (RepLLaMA) · In-class slide 16
Ma et al., "Fine-Tuning LLaMA for Multi-Stage Text Retrieval", 2023. RepLLaMA uses the same architectural setup as E5-Mistral (decoder-only backbone, instruction-prefixed query, appended [EOS], embedding = last hidden state of [EOS]) with Llama-2-7B as the backbone.
| Design choice | RepLLaMA |
|---|---|
| Backbone | Llama-2-7B (decoder-only, open source) |
| Representation | [EOS] token embedding (last hidden state) |
| Similarity | Dot product |
| Loss | Negative log likelihood / InfoNCE (the same as DPR and ANCE) |
| Training | Supervised labels only (like DPR) + LoRA adapters |
Two practical insights distinguish it:
- No synthetic data needed. RepLLaMA shows that with a strong LLM backbone you can skip E5-Mistral's GPT-4 synthetic-data pipeline entirely and train directly on supervised retrieval labels (as DPR does, for example MS MARCO), and it already works well.
- LoRA adapters for training-time efficiency. Instead of fine-tuning all 7 billion parameters, LoRA (Low-Rank Adaptation) trains only small adapter modules on the projection layers. This dramatically reduces training time and compute while keeping competitive retrieval effectiveness, which makes LLM-based retrievers practical to train.
The lecture names LoRA without the maths. As background, the standard LoRA update for one weight matrix is:
- the pretrained weight matrix of one projection layer, kept frozen
- the two small trainable adapter matrices
- the rank of the adapter (for example 8 or 16), much smaller than and
- their product: a low-rank correction added to
- the effective weight used in the forward pass
Worked example 10How much smaller is a LoRA update?
Illustrative: one projection matrix (Llama-2-7B's hidden size is 4096) and LoRA rank .
PracticeWhat do E5-Mistral and RepLLaMA share, and on which design dimension do they differ most?
PracticeDoes LoRA make a RepLLaMA query faster to encode at search time?
15. Model comparison summary
Sources: slides 18–48 · Videos 2–9 · In-class slides 10–12, 16
| Model | Backbone | Representation | Similarity | Loss | Negatives / training signal |
|---|---|---|---|---|---|
| DPR (2020) | Encoder | [CLS] embedding | Dot product | InfoNCE (negative log likelihood) | Random + BM25 hard + in-batch |
| RepBERT (2020) | Encoder | Mean of token embeddings | Inner product | Multi-label margin loss | In-batch |
| ANCE (2021) | Encoder | [CLS] embedding | Inner product | InfoNCE | Self-mined hard negatives via ANN over training checkpoints |
| Contriever (2021) | Encoder | Mean of token embeddings | Cosine similarity | InfoNCE | Unsupervised: two spans of the same document = positive pair |
| ColBERT (2020) | Encoder (multi-vector) | All token embeddings | MaxSim (late interaction) | InfoNCE | Standard triples (as in DPR-style training) |
| E5-Mistral (2023/24) | Decoder, Mistral-7B | [EOS] last hidden state, instruction-prefixed query | Cosine/dot (contrastive) | InfoNCE | GPT-4 synthetic ⟨q, p⁺, hard p⁻⟩ + MS MARCO; in-batch + hard |
| LLM2Vec (2024) | Decoder (any LLM) | Mean pooling over the bidirectional LLM | Cosine (contrastive) | SimCSE contrastive (+ optional supervised) | Unsupervised: dropout views of the same text (+ optional labels) |
| RepLLaMA (2023) | Decoder, Llama-2-7B | [EOS] last hidden state, instruction-prefixed query | Dot product | InfoNCE | Supervised labels only (MS MARCO), trained with LoRA adapters |
PracticeWhich two models in the table use cosine similarity on mean-pooled vectors, and what else do they have in common?
PracticeWhich model is the only one this week trained with a margin loss rather than InfoNCE, and which is the only multi-vector model?
16. Course context from the in-person lecture
Sources: In-class slides 2–7 · slides 4–6
The in-person session opened by placing this week on the course's "historical view of ranking models" timeline, which shows how the weeks fit together and what is examinable where:
The lecturer also stressed how the weeks build on each other: weeks 1, 2 and 4 are preliminary knowledge; weeks 3, 5, 6, 7 and 8 are the core models (this week is one of the core weeks); weeks 9–10 cover new research advances; weeks 11–13 remain important. In the relevance-model cost table, the last empty slot next to this week's models belongs to learned sparse retrievers, next week's topic. The session also reported the course survey (average 4.54 out of 5 from 24 responses) and the request that lectures include concepts as well, which led to the transformer recap in Section 17.
16.1 The models at a glance: years and one-line innovations
The in-person slides summarised every model as a card: representation, similarity, loss, plus the one-line "innovation" that makes it worth knowing. This table is a good revision checklist:
| Model (year) | Card (representation · similarity · loss) | Innovation one-liner |
|---|---|---|
| DPR (2020) | [CLS] embedding · dot product · InfoNCE | The influential dual-encoder model. |
| RepBERT (2020) | Mean embedding · dot product · multi-label margin loss | Strong early BERT dual-encoder baseline; mean pooling + margin loss beats BM25 on MS MARCO. |
| ANCE (2021) | [CLS] embedding · dot product · InfoNCE | Asynchronous negative sampling from an evolving index during training. |
| Contriever (2021) | Mean embedding · cosine similarity · InfoNCE | Trained without labels on massive corpora with contrastive learning. |
| ColBERT (2020) | All token embeddings · MaxSim · InfoNCE | Efficient late interaction: token-level query/document embeddings scored with MaxSim. |
| E5-Mistral (2024) | [EOS] embedding · dot product · InfoNCE | GPT-4-generated synthetic supervised data + MS MARCO labels. |
| LLM2Vec (2024) | Mean pooling · dot product · InfoNCE | Bidirectional attention + masked next-token prediction + unsupervised SimCSE; supervised tuning optional. |
| RepLLaMA (2023/24) | [EOS] embedding · dot product · InfoNCE | LLaMA bi-encoder trained on MS MARCO with hard and in-batch negatives. |
Grouping remarks from the lecture worth remembering: DPR and RepBERT are the early series (2020); ANCE's card carries the takeaway that negative sampling matters during training; and Contriever's card carries the takeaway that these models can be trained without supervised labels.
PracticeWhich weeks did the lecturer call the "core models", and which two of them cover neural first-stage retrievers?
17. Transformer recap from the in-person session
Sources: In-class slides 9, 14–15 · Week 4 lecture notes
Responding to survey feedback ("lecture should include concepts as well"), the in-person session re-taught the transformer pipeline from Week 4 before the exercises. A condensed version of that recap, since everything this week builds on it:
- Tokenisation (WordPiece). Text is split into sub-word tokens, each mapped to an integer ID. The vocabulary (about 30K for BERT; model-dependent) is learned from corpus statistics by merging frequently co-occurring character sequences, so "playing" may split into
play+##ing. Different model families (BERT, Llama, GPT) train different tokenisers on different collections, so the same text tokenises differently. - Input embeddings are added element by element. Each token's input vector = token embedding + segment embedding + position embedding: vector addition with the dimension unchanged. (This was an in-class exercise: the [CLS] final input embedding is its token embedding + segment-A embedding + position-0 embedding.) Position embeddings matter because "I asked him" and "he asked me" contain the same words in different orders; segment embeddings (BERT-specific, optional in other models) separate sentence A from sentence B.
- Special tokens. [CLS]: a classification/summary slot heavily used by retrievers; [SEP]: separates input parts; [PAD]: makes batch sequences equal length for parallel GPU processing; [MASK]: used in BERT's training. Decoders use [BOS]/[EOS] to mark sequence start and end; [EOS] is how generation knows when to stop, and it is the token whose vector E5-Mistral and RepLLaMA use as the embedding.
- Self-attention (Q, K, V). Each token's input vector is projected by three learned weight matrices into a query vector ("what am I looking for"), a key vector ("what do I offer to others") and a value vector ("how do I express myself"). The output dimension equals the input dimension, so the block can be stacked (for example 12 layers, with multiple heads per layer). The attention formula is below.
- Encoder and decoder attention. The encoder is bidirectional: every token attends to every other token, past and future, and all tokens are processed in parallel (this parallelism is what makes transformers fit GPUs so well). The decoder is causal: each token attends only to earlier tokens, because during generation future tokens do not exist yet.
- How a decoder generates. The last token's final-layer hidden vector (dimension , for example 768 for BERT-sized or 2048+ for Llama) is multiplied by a learned linear layer ( = vocabulary size), giving one logit per vocabulary token; softmax turns the logits into probabilities; a token is chosen (greedy or sampled; sampling is why the same ChatGPT prompt gives different answers); the chosen token is appended to the input and the loop repeats until [EOS].
- Pre-training. BERT: masked language modelling, which randomly masks a fraction of tokens and predicts them from both sides. GPT: next-token prediction over huge text collections. Both are unsupervised, which is what makes web-scale pre-training possible.
- the query vector of token (the token doing the attending)
- the key and value vectors of token
- the dimension of the query and key vectors; keeps the dot products in a stable range
- normalises over all tokens that token may see, so the weights are positive and sum to 1
- the attention weight token gives to token
- the output vector of token
PracticeBERT-base uses 768-dimensional embeddings. What is the dimension of the input vector for the [CLS] token after adding its token, segment and position embeddings?
18. Q&A from the in-person lecture
Sources: In-class slides 21–22
The most valuable part of the live session was the discussion. These are the questions asked (by students or as in-class exercises) with the answers, cleaned up.
18.1 Transformer mechanics
Q: Is the self-attention matrix symmetric?
A: No. Attention is directional: the attention token pays to token need not equal the attention pays to . In the lecturer's words, "I care about that token; that token doesn't have to care about me." Each row is computed from a different query vector, so in general.
Q (in-class MCQ): Which statements about the transformer are correct?
A: "The encoder uses masked attention to prevent looking at future tokens": false (that describes the decoder; the encoder must see the whole sentence to capture its meaning). "The decoder uses bidirectional attention": false (swapped again). "The encoder processes input tokens in parallel using bidirectional attention": true (and this parallel processing is exactly why transformers exploit GPUs so well). "Both encoder and decoder use the same type of attention": false (bidirectional against causal).
Q: Is it true that BERT has more layers than GPT? Do they use different tokenisation?
A: Layer count is a free design choice. In practice BERT (about 110M parameters) is far smaller than modern GPT-style models, but you could build an encoder with more layers. Generation is the harder task, which is why decoder models grew to billions or trillions of parameters (the "scaling law"). Both families use word-piece-style tokenisation; what differs is the trained vocabulary (size and contents), because each is estimated from a different collection with different goals.
Q: In the exercise, how do the three embeddings combine: into a matrix?
A: By vector addition: first dimension + first dimension + first dimension, and so on. Each token ends with a single vector of the model dimension.
Q: What exactly does the decoder's linear layer do?
A: It is a learned projection from the model dimension to the vocabulary size . Multiplying the last token's hidden vector by it gives a score (logit) for every vocabulary token; softmax converts these to probabilities between 0 and 1; the next token is picked from the top of that distribution. Because deployed systems sample from the distribution (always taking the argmax would be deterministic), the same prompt can produce different generations.
18.2 Using encoders and decoders for ranking (in-class Exercises 1 and 2)
Q: For a transformer encoder, what are the ways to obtain a text representation, and how is similarity computed?
A: Three ways, the answer to Exercise 1: use the [CLS] vector (DPR, ANCE), mean-pool the token vectors (RepBERT, Contriever), or keep multiple token vectors (ColBERT). For the single-vector options, similarity is one dot product or cosine between the query vector and the document vector. For the multi-vector option, similarity is late interaction: compare every query token embedding with every document token embedding, take each query token's maximum, and sum the maxima (MaxSim). A student proposed exactly this pairwise-then-aggregate scheme in class, and the lecturer confirmed it is essentially how ColBERT started.
Q: How can a transformer decoder be used to compute query–document similarity? (Exercise 2; several answers were collected in class, all valid.)
- As an embedder: cut off the generation head and use the internal vector of the last token as a text embedding, then score with dot product or cosine: exactly what E5-Mistral, LLM2Vec and RepLLaMA do.
- As a generative ranker: prompt the LLM directly: "here is a query and 10 documents, give me an order" (listwise), "is this document relevant: yes/no or a score" (pointwise), "which of these two documents is more relevant" (pairwise).
- For query/document expansion (doc2query): generate a pseudo-document from the query, or generate likely query terms for each document at indexing time, then let a lexical ranker (BM25) match in the enriched term space. This attacks vocabulary mismatch from the lexical side.
- Via output probabilities: use the model's probability distribution over the whole vocabulary as a representation of the text; terms the document makes probable get high weight. The lecturer flagged this as a preview of learned sparse retrievers, next week's topic.
Q: Why is expansion done at indexing time rather than query time?
A: Latency. The user is waiting at query time, so heavy generation there hurts the experience; documents are processed offline at indexing time where extra minutes cost nothing. This is the same logic as pre-computing document embeddings in a bi-encoder.
18.3 Beyond this week
Q: Can an encoder backbone be used for generation?
A: Yes: this is the idea behind diffusion language models, a very recent research direction. The encoder was trained on masked-token prediction, so append, say, 256 [MASK] tokens to the prompt and predict them all simultaneously (parallel, GPU-friendly) in place of one at a time. The catch is that independently predicted tokens can be incoherent; the fix, borrowed from image diffusion, is to run a few denoising/refinement passes, still far fewer passes than generating 256 tokens autoregressively. The lecturer noted his own recent work applies diffusion language models to retrieval.
Q (assignment logistics): What should a gain–loss plot look like?
A: One bar per query: the evaluation score of System A minus System B on that query. Typically sorted from largest gain to largest loss, so you can see at a glance whether A consistently beats B and, more importantly, which queries A loses on; those are the queries to analyse. The BM25 baseline in the assignment should be your own implementation (check the criteria; ask on the discussion board if unclear).
PracticeGive two ways to use a decoder LLM for ranking that avoid running it at query time over every document.
19. In-class exercises and where the solutions live
Sources: In-class slides 21–25 · Practical solution pages for Exercises 3–5
The session ran five exercises. Exercises 1–2 are conceptual and answered in the Q&A section above; Exercises 3–5 are computational or challenge questions with full step-by-step solutions saved in the Practical/ folder:
| Exercise | Question | Answer | Solution |
|---|---|---|---|
| 1: Encoder-based | Ways to obtain a text representation from an encoder; which tokens; how similarity is computed | [CLS] / mean / all tokens; dot or cosine, or MaxSim | Section 18.2; Sections 4.1 and 11 |
| 2: Decoder-based | How can a decoder compute query–document similarity? | Last-token embedding; LLM as ranker; doc2query; output-probability representations | Section 18.2; Section 4.2 |
| 3: InfoNCE loss | sim(q, p⁺) = 0.9; negatives 0.3, 0.1, 0.4 | worked solution | |
| 4: Margin loss | Rel(q, p⁺) = 1.5; negatives 0.3, 0.1, 0.9; margin 1 | worked solution | |
| 5: Challenge | Mine negatives harder than top-ranked BM25 documents | ANCE: self-mined, checkpoint-refreshed negatives | worked solution |
Worked example 11Exercise 3 in brief: InfoNCE with three negatives
From in-class slide 23: ; negatives .
Worked example 12Exercise 4 in brief: margin loss with three negatives
From in-class slide 24: ; negatives ; margin 1.
PracticeIn Exercise 4, how low would the hardest negative's score need to fall for the loss to reach zero?
PracticeExercise 5: training with BM25 hard negatives is slow to improve. Describe a way to mine harder negatives.
20. Summary and study checklist
- Dense retrieval = embed both texts with the same embedder, score with a cheap similarity function: . It solves vocabulary mismatch by measuring closeness in a learned semantic space in place of term overlap.
- Bi-encoders owe their query-time speed to pre-computing document embeddings offline; the transformer cost stays off the query-time path, leaving an scorer.
- Pooling choices: [CLS] (DPR, ANCE), mean (RepBERT, Contriever), all tokens (ColBERT), last/[EOS] token (decoder models: the only position that has seen the whole input under causal attention).
- Negative sampling is a first-class design decision: random = easy, in-batch = medium (and free), BM25 = lexically hard, ANCE checkpoint-mined = hard for the dense model itself. Harder, fresher negatives give better retrievers at higher training cost.
- ColBERT trades scoring and 1-vector storage for token-level MaxSim matching: usually more effective, dramatically more expensive in storage and scoring.
- LLMs can be retrievers: discard generation and take the last-token embedding (E5-Mistral, RepLLaMA), or convert the decoder into a true bidirectional encoder (LLM2Vec). Their autoregressive pre-training already provides strong text representations, so contrastive pre-training becomes unnecessary and fine-tuning mainly adds task knowledge.
- Labels are optional: Contriever (same-document spans) and LLM2Vec (dropout views) train contrastively without human judgments; E5-Mistral substitutes LLM-generated synthetic triples.
Study checklist. You should be able to:
- Fill in the relevance-model cost table (scorer and encoder cost) for lexical, single-vector and multi-vector models, and explain each entry.
- Compute a dot product, a cosine similarity and a mean-pooled vector by hand, and explain when dot product and cosine disagree.
- Write the InfoNCE loss, define every symbol, and compute it for given similarities (Worked examples 3 and 11).
- Write the multi-label margin loss, clip each pair at zero, and average over pairs (Worked examples 5 and 12).
- Describe DPR's three negative types and count in-batch negatives for a batch (Worked example 4).
- Explain why ANCE's checkpoint-mined negatives beat BM25 negatives for a dense model, and what they cost.
- Build Contriever training pairs from a raw document.
- Compute a ColBERT MaxSim score and its cost in dot products and storage (Worked examples 7 and 8).
- Explain why decoder embedders use the [EOS]/last token, and list LLM2Vec's three steps and why step 1 alone hurts.
- Compare E5-Mistral and RepLLaMA, including the role of synthetic data and LoRA.
- Place every model of the week on the five design dimensions (backbone, representation, similarity, loss, sampling).
21. Video index
Week 5 YouTube playlist: IR Week 5. Transcripts are in week_5_video_transcripts.md. The in-person lecture slides are in Lecture-Slides/. (The file INFS7410_S2_2026_STL-transcript.txt in this folder is a copy of the Week 4 lecture transcript, so these notes do not cite it.)
| Video | Topic | Length | Sections here |
|---|---|---|---|
| Video 1 | Overview: embedding-based retrievers | 18:34 | 1–6 |
| Video 2 | DPR (Dense Passage Retrieval) | 11:10 | 5, 7 |
| Video 3 | RepBERT | 04:26 | 8 |
| Video 4 | ANCE | 03:43 | 9 |
| Video 5 | Contriever | 03:06 | 10 |
| Video 6 | ColBERT | 07:11 | 5.2, 11 |
| Video 7 | E5-Mistral | 07:58 | 12 |
| Video 8 | LLM2Vec | 04:34 | 13 |
| Video 9 | RepLLaMA | 03:09 | 14 |
22. References
- Karpukhin et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020. (DPR)
- Zhan et al. (2020). RepBERT: Contextualized Text Embeddings for First-Stage Retrieval. arXiv 2020. (RepBERT)
- Xiong et al. (2021). Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. ICLR 2021. (ANCE)
- Izacard, Caron, Hosseini, Riedel, Bojanowski, Joulin and Grave (2021). Unsupervised Dense Information Retrieval with Contrastive Learning. arXiv 2112.09118. (Contriever)
- Khattab and Zaharia (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. SIGIR 2020. (ColBERT)
- Wang, Yang, Huang, Yang, Majumder and Wei (2023). Improving Text Embeddings with Large Language Models. arXiv 2401.00368. (E5-Mistral)
- BehnamGhader, Adlakha, Mosbach, Bahdanau, Chapados and Reddy (2024). LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders. arXiv 2404.05961. (LLM2Vec)
- Ma et al. (2023). Fine-Tuning LLaMA for Multi-Stage Text Retrieval. (RepLLaMA)