Embedding-based Retrieval Models (Dense Retrievers)
This document tells you how dense retrievers use transformer encoders and decoders to find relevant documents. It covers bi-encoders (DPR, RepBERT, ANCE, Contriever), late interaction (ColBERT) and LLM-based embedders (E5-Mistral, LLM2Vec, RepLLaMA). For each model, it gives the representation, the similarity function, the training loss and the negative sampling method. Written in Simplified Technical English. Lecturer: Dr Shuai Wang, The University of Queensland.
1. The big picture: from lexical to dense retrieval
Sources: slides 2–3, 8 · Video 1 (Overview: embedding-based retrievers) · In-class slides 4, 7
Where we are in the course. In Week 3 you learned term-based lexical models such as BM25. These models score a document by the terms it shares with the query. In Week 4 you learned the transformer architecture. The encoder (BERT) reads the full text with bidirectional attention. It makes one contextual vector for each token. The decoder (GPT) reads the text from left to right with causal attention. It predicts the next token. This week connects the two topics. Transformers turn queries and documents into vectors. The system then ranks documents by vector similarity.
The models of this week, in groups by backbone architecture:
| Backbone | Representation style | Models |
|---|---|---|
| Transformer encoder | Single vector: one vector for each query or document | DPR, ANCE, RepBERT, Contriever |
| Transformer encoder | Multi vector: one vector for each token | ColBERT |
| Transformer decoder (LLM) | Single vector: the vector of the last token | E5-Mistral, LLM2Vec, RepLLaMA |
PracticeA user searches for "automobile insurance". The best document only says "car cover". Why does BM25 miss it? What does a dense retriever do differently?
2. The table of relevance models: costs
Sources: slides 4–6 · Video 1 (Overview: embedding-based retrievers) · In-class slide 6
Recall the table of relevance models from Week 3. Two costs describe each model family:
- Scorer cost: the work, for each document, to compute a relevance score after the query arrives (query time).
- Encoder cost: the work to turn one text into its representation. For documents, this work occurs offline.
| Model family | Examples | Scorer cost | Encoder cost |
|---|---|---|---|
| Term-based lexical models | BM25, TF-IDF | : inverted-index lookups | : one pass to count terms |
| Single-vector embedding-based retrievers | DPR, ANCE, RepBERT, Contriever, E5, LLM2Vec, RepLLaMA | : one dot product or cosine | : attention over all token pairs |
| Multi-vector embedding-based retrievers | ColBERT | : each query token against each document token | : the same transformer encoding |
| Learned sparse retrievers | next week (Week 6) | (next week) | (next week) |
How to read the cost terms:
- : the encoding work increases with the square of the document length. Attention compares each token with each other token.
- : the scoring work for one document is constant. It is one vector operation for all document lengths.
- : the work increases with the query length multiplied by the document length. ColBERT compares all token pairs.
PracticeWhy is the scorer cost of a bi-encoder O(1) for each document, and its encoder cost O(|d|²)? Why does the quadratic part have almost no effect on query latency?
PracticeWhich slot in the cost table stays empty this week? When does the course fill it?
3. Dense representations: embedders and similarity
Sources: slide 7 · Video 1 (Overview: embedding-based retrievers) · In-class slide 8
Each dense retriever has two parts. The first part is an embedder . This is a neural network (BERT, an LLM, or other) that changes a text into a vector of fixed size. The same type of embedder applies to the query and the document . Thus, the two vectors are in the same space. The second part is a similarity function . This is a cheap function (dot product, cosine similarity, or other) that compares the two vectors.
- the query text and the document text
- the embedder (eta): a neural network that changes a text into a vector of fixed size
- the query embedding and the document embedding (for example, 768 numbers each)
- the similarity function (phi), for example inner product or cosine similarity
- one number that ranks for
The vectors have a low number of dimensions. BERT-based models use 768 dimensions. Decoder (LLM) based models use approximately 3,000 to 4,000 dimensions. A lexical bag-of-words vector has tens of thousands of dimensions. Dense retrieval measures relevance as distance in a learned semantic space. Thus, "car" and "automobile" can have almost equal embeddings, although they share no characters.
3.1 The two standard similarity functions
Dot product (also called inner product). The Week 4 maths refresher shows vector operations.
- the query embedding and the document embedding, and
- the number of dimensions of each vector (for example, 768)
- entry of the query vector and of the document vector
- add over all dimensions
Cosine similarity. The vector space model in Week 3 uses the same cosine. Here, it applies to learned embeddings.
- the dot product from the formula above
- the length (L2 norm) of :
- the length (L2 norm) of
Worked example 1Dot product against cosine similarity
These are illustrative, small 3-dimension embeddings for a query and two documents:
PracticeTwo document embeddings point in the same direction. One vector is three times as long. Which similarity function gives them the same rank? Which function prefers the longer vector?
4. Two backbones: encoder vs decoder
Sources: slides 8–11, 14–16 · Video 1 (Overview: embedding-based retrievers) · In-class slides 9, 14–15
4.1 Encoder-only (BERT-style): one vector per token, then pool
A transformer encoder (Week 4) reads the full input with bidirectional attention. It outputs one contextual vector for each token. Retrieval needs one representation of the full text. Thus, a model must change many token vectors into one text representation. There are three standard methods:
- [CLS] token vector: BERT has a special first token. Its purpose is to represent the full sequence. This method is simple and efficient. DPR and ANCE use it.
- Mean of all token vectors: calculate the average of all positions. This method can keep more information about the sequence. RepBERT and Contriever use it.
- All token vectors: keep all the vectors. This method keeps the most detailed information. But it needs a special similarity function, because one dot product is not sufficient. ColBERT uses it.
This is the formula for mean pooling:
- the text to embed (a query or a document)
- the number of tokens in
- the encoder output vector for token (for BERT, 768 numbers)
- the single text vector. It has the same number of dimensions as each .
Worked example 2Mean pooling
These are illustrative token vectors for a text with 3 tokens (3 dimensions, to make the example easy to read):
4.2 Decoder-only (GPT-style): discard generation, keep the last-token vector
A decoder predicts the next token from the input. It uses causal attention: each token sees only the tokens before it. The decoder was made for generation. But its internal layers also learn good representations of the text. This is the method for retrieval:
PracticeWhy is mean pooling over the token vectors of a causal decoder a weaker summary than the last-token vector?
PracticeCompute the mean-pooled vector of the token vectors [1, 0], [0, 1] and [2, 2].
5. Bi-encoders vs late interaction
Sources: slides 12–13, 17 · Video 1 (Overview: embedding-based retrievers) · Video 2 (DPR) · In-class slides 12–13
5.1 Single-vector retrievers: the bi-encoder architecture
Single-vector models use a bi-encoder (also called dual encoder). One encoder is for the query. One encoder is for the document. The two encoders often share weights. Each encoder makes one vector. The score is one similarity calculation.
5.2 Multi-vector retrievers: late interaction
ColBERT keeps all token vectors of the query and of the document. The vectors interact only at scoring time. This is the reason for the name late interaction. This method finds more detailed matches at token level. It is often more effective. But scoring is more expensive ( instead of ). Storage also increases from 1 vector to hundreds of vectors for each document. Section 11 shows the MaxSim scoring step by step.
| Property | Bi-encoder (single vector) | Late interaction (multi vector) |
|---|---|---|
| Vectors stored for each document | 1 | approximately 100–500 (one for each token) |
| Scoring for each document | : one dot product or cosine | : MaxSim over all token pairs |
| Match signal | summary of the full text | matches at token level |
| Usual trade-off | faster, less storage | often more effective, more computation and storage |
PracticeWhy can a bi-encoder compute document vectors offline? What prevents this if one encoder reads the query and the document together (a cross-encoder)?
6. The five dimensions that differentiate models
Sources: slides 17–18 · Video 1 (Overview: embedding-based retrievers)
Each model of this week gives a different answer to the same five design questions. These questions help you to organise the week:
| Design question | Possible answers in this week |
|---|---|
| 1. Model backbone: encoder or decoder? | Encoder (DPR, ANCE, RepBERT, Contriever, ColBERT) or decoder LLM (E5-Mistral, LLM2Vec, RepLLaMA) |
| 2. Representation: how does the model turn a text into one or more vectors? | [CLS] embedding · mean of token embeddings · all token embeddings · last-token/[EOS] embedding |
| 3. Similarity function: how does the model compare vectors? | Dot product · cosine similarity · late interaction (MaxSim) |
| 4. Loss function: how does the model learn ranking during fine-tuning? | Negative log likelihood (InfoNCE) · multi-label margin loss · SimCSE-style contrastive loss |
| 5. Positive/negative sampling: where does the training signal come from? | Random negatives · BM25 hard negatives · in-batch negatives · self-mined (ANCE checkpoint) negatives · spans of the same document (Contriever) · synthetic triples from an LLM (E5-Mistral) |
To train these models, each training query needs a positive document (known to be relevant). It also needs a set of negative documents for comparison. The model learns to give the positive a higher score than the negatives. When it can do this, its embeddings are useful for dense retrieval.
PracticeANCE and DPR have the same backbone, representation, similarity and loss. On which of the five dimensions are they different?
7. DPR: Dense Passage Retrieval
Sources: slides 19–25 · Video 2 (DPR) · In-class slides 10, 17–20
Karpukhin et al., "Dense Passage Retrieval for Open-Domain Question Answering", EMNLP 2020. DPR is the typical bi-encoder. It showed that a trained dense retriever can be equal to BM25 or better. These are its design choices:
| Design choice | DPR |
|---|---|
| Backbone | BERT encoder (bi-encoder: one query encoder and one passage encoder) |
| Representation | [CLS] token embedding |
| Similarity | Dot product (simple, faster than cosine) |
| Loss | Negative log likelihood (InfoNCE) |
| Negatives | Random + BM25 hard + in-batch |
7.1 The InfoNCE loss
The loss has one goal. It makes the probability high that the model selects the relevant passage from a set of candidates. In-class slide 17 says that it "encourages the positive passage to receive the highest score among candidates".
- the query embedding (DPR: the [CLS] vector of the query)
- the positive (relevant) passage embedding
- negative (irrelevant) passage embedding number
- the number of negatives for this query
- the dot-product similarity
- the exponential function. It makes all terms positive.
- the negative natural logarithm. It changes a probability into a loss.
- the loss for one training query. Training makes its average as small as possible.
Worked example 3Computing the DPR InfoNCE loss
These are illustrative dot-product similarities for one query with one positive and two negatives:
7.2 The three-part negative sampling strategy
The success of DPR depends much on the negatives that it uses for comparison. DPR uses three sources. Each source is more difficult than the one before it:
- Random negatives (easy). Select passages at random from the full corpus. For a query about machine learning, a random negative can be a cooking recipe. These negatives help with basic discrimination. But the model quickly finds them too easy. Then it stops learning small differences. Real search has much harder cases.
- BM25 hard negatives. Get the BM25 top- for the query (for example, the top 1000). Remove all passages with a relevant label. Select negatives from the remaining set. These passages share keywords with the query, but their meaning is not relevant. Example: the query "how does machine learning work" and the passage "machine learning applications in healthcare". These negatives teach the model the difference between lexical similarity and true semantic relevance.
- In-batch negatives (medium, free). A training batch has queries and positives . For , use and as negatives. The system already encoded these passages for the other queries. Thus, they cost nothing more. They come from the same dataset and domain. Thus, they are harder than random negatives: "medium" difficulty. Slide 25 says that they give efficiency and a gain for the domain.
Worked example 4Counting negatives in a DPR training batch
A training batch has queries. Each query has 1 positive passage with a label and 1 BM25 hard negative. The batch shares all passages as in-batch negatives. How many negatives does each query get? How many passage encodings does the batch need?
PracticeCompute the InfoNCE loss for sim(q, p⁺) = 2.0 and two negatives with similarities 1.0 and 0.0.
PracticeWhy are in-batch negatives "medium" difficulty? Why are they harder than random negatives and easier than BM25 hard negatives?
8. RepBERT: mean pooling and margin loss
Sources: slides 26–27 · Video 3 (RepBERT) · In-class slides 10, 17
Zhan et al., "RepBERT: Contextualized Text Embeddings for First-Stage Retrieval", arXiv 2020. RepBERT is different from DPR in two ways. It pools the token vectors differently. It also uses a different training loss.
| Design choice | RepBERT |
|---|---|
| Backbone | BERT encoder |
| Representation | Mean of all token embeddings (Section 4.1) |
| Similarity | Inner (dot) product |
| Loss | Multi-label margin loss |
| Negatives | In-batch (for efficiency) |
8.1 Multi-label margin loss
In-class slide 17 gives the aim. Relevant documents must get a higher score than irrelevant documents, by a minimum fixed margin (here 1).
- the query
- the positive (relevant) and negative (irrelevant) documents for in the batch
- the number of positive–negative pairs. The formula divides by it to get the average.
- one positive document and one negative document
- the relevance score: the dot product of the mean-pooled query vector and document vector
- the margin: the minimum gap between a positive score and a negative score
- sets the loss of each pair to zero when it is below zero (a hinge)
Worked example 5Margin loss on the two lecture cases
The margin is 1. Compute the loss of one positive–negative pair in each case (values from slide 27).
PracticeA RepBERT pair has Rel(q, d⁺) = 0.9 and Rel(q, d⁻) = 0.4. What is the margin loss with margin 1?
PracticeA query has one positive (score 1.0) and two negatives (scores 0.5 and −0.5). A student adds the raw values 1 − gap for the two pairs. The student gets 0.5 + (−0.5) = 0. What is the error?
9. ANCE: self-mined hard negatives
Sources: slides 28–30 · Video 4 (ANCE) · In-class slide 11 · In-class slide 25 (Exercise 5)
Xiong et al., "Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval", ICLR 2021. ANCE keeps the representation, similarity and loss of DPR. Its only new idea is the source of the negatives.
| Design choice | ANCE |
|---|---|
| Backbone | BERT encoder |
| Representation | [CLS] token embedding |
| Similarity | Inner (dot) product |
| Loss | Negative log likelihood (InfoNCE, Section 7.1) |
| Negatives | Self-mined hard negatives from regular checkpoints (Negative Contrastive Estimation) |
9.1 The problem with BM25 negatives
The hard negatives of DPR are hard for BM25. But the model in training is a dense retriever. A document that is difficult for BM25 can be easy in the dense embedding space. Slide 29 shows the embeddings on a 2-D map:
9.2 Mining negatives from checkpoints
How can you get "negatives that confuse the current model" during training? ANCE uses a recent checkpoint of the model itself (slide 30).
- Save a checkpoint of the model regularly during training (for example, every thousand iterations).
- Use the previous checkpoint to embed the corpus. Retrieve documents for the training queries with approximate nearest-neighbour search ("ANN" in the name).
- From the results, take the documents that have a high rank but are not relevant. These documents are the errors of the current model. Use them as negatives.
- Do this again during all of training. Refresh the negatives when the model improves.
PracticeWhy are BM25 hard negatives "hard" for a lexical reason but possibly "easy" for a dense retriever? How does ANCE correct this?
PracticeDoes ANCE make retrieval at query time slower than DPR? Explain.
10. Contriever: unsupervised dense retrieval
Sources: slides 31–32 · Video 5 (Contriever) · In-class slide 11
Izacard et al., "Unsupervised Dense Information Retrieval with Contrastive Learning", 2021.
| Design choice | Contriever |
|---|---|
| Backbone | BERT encoder |
| Representation | Mean of token embeddings |
| Similarity | Cosine similarity |
| Loss | Negative log likelihood (InfoNCE, the same loss as DPR and ANCE) |
| New idea | Unsupervised training: no labelled data necessary |
DPR and ANCE need relevance judgments from human annotators. The main idea of Contriever is to make its own training signal from a raw text collection:
- Select two random spans and from . Each span is a continuous sequence of tokens. Its length is between 5% and 50% of the document length.
- Positive pair: . The method assumes that two spans of the same document are related in meaning.
- Negative pairs: with . The method assumes that spans from different documents are less related.
Worked example 6Building Contriever training pairs from raw text
Use a document of 40 tokens: "The Great Barrier Reef stretches over 2,300 km off the coast of Queensland. It is the world's largest coral reef system, composed of nearly 3,000 individual reefs. Rising sea temperatures have caused repeated mass bleaching events in recent years."
PracticeA document has 200 tokens. Which span lengths can Contriever select? Why can two spans from the same document sometimes be a bad positive pair?
11. ColBERT: late interaction with MaxSim
Sources: slides 33–40 · Video 6 (ColBERT) · In-class slides 12–13
Khattab and Zaharia, "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT", SIGIR 2020. ColBERT is the only multi-vector model of this week. It is the best-known model of the late-interaction family.
| Design choice | ColBERT |
|---|---|
| Backbone | BERT encoder (query encoder and document encoder) |
| Representation | All token embeddings (L2-normalised), for each query and each document |
| Similarity | MaxSim (late interaction) |
| Loss | Negative log likelihood (the same family as DPR, ANCE and Contriever) |
11.1 The MaxSim scoring function, step by step
- the number of query tokens
- the number of document tokens
- the L2-normalised embedding of query token
- the L2-normalised embedding of document token
- their dot product. It is a cosine, because both vectors have length 1. In-class slide 13 writes it as .
- the best match for query token in the full document
The lecture shows the method in five steps for query tokens and document tokens (slides 34–38):
- Step 1: take query token . Compute the dot product of its embedding with the first document token embedding . This gives one similarity score.
- Step 2: for the same , compute its similarity with each other document token . This gives a full list of scores for .
- Step 3: from the list, keep the maximum. This is the best match of in the full document. A strong match with any part of the document is sufficient. The method does not calculate an average.
- Step 4: do steps 1–3 again for each remaining query token . This gives one maximum for each query token.
- Step 5: add the maximum values. Each query token adds its best match score to the total.
Worked example 7ColBERT MaxSim by hand
These values are illustrative. A query has token embeddings. A document has . All token vectors are L2-normalised. Thus, each dot product is a cosine similarity. The pair similarities are:
11.2 Why it works, and what it costs
- Very large storage: approximately 100–500 vectors for each document instead of 1. The index is hundreds of times larger than a bi-encoder index.
- Scoring complexity: for each document, against for bi-encoders ( = query length, = document length). MaxSim operations cost more than one dot product. Storage and scoring need much more infrastructure.
Worked example 8Index size: bi-encoder against ColBERT
Illustrative setting: 1,000,000 passages. Each vector has 768 dimensions, stored as 4-byte floats. ColBERT keeps 200 token vectors for each passage (in the 100–500 range of slide 40). Real systems make token vectors smaller and compress them. The lecture does not cover this. The important result here is the ratio.
PracticeWhy does ColBERT use the maximum over document tokens instead of the mean? What does an average do to a long document with one very relevant sentence?
PracticeThe row of one query token in the similarity matrix is 0.2, 0.7, 0.7, 0.1. What does this token add to the MaxSim score? Is it important that the maximum occurs two times?
12. E5-Mistral: LLM embeddings and synthetic data
Sources: slides 41–43 · Video 7 (E5-Mistral) · In-class slide 16
Wang et al., "Improving Text Embeddings with Large Language Models", 2023 (arXiv 2401.00368). E5 is a family of embedding models. The original E5 embeddings use BERT. E5-Mistral replaces the encoder backbone with a decoder-only LLM (Mistral-7B). It uses LLMs in two different ways.
12.1 Use 1: synthetic training data generated by an LLM
- Brainstorm tasks: tell an LLM (GPT-4) to make a list of possible retrieval tasks. Examples: "search web pages about a topic", "find academic references for an assignment", "find a tutorial video".
- Generate triples: for each task, tell the LLM to generate a ⟨query, positive document, hard-negative document⟩ triple. The prompt gives format instructions from the task definition. Do this many times. The result is a large synthetic training set without human annotation.
12.2 Use 2: the LLM as the embedding backbone
- Change the query to include an instruction. The instruction is the task definition from the data generation step:
q_inst = "Instruct: {task_definition} \n Query: {q}" - Add [EOS] to the end of the query and of the document. Put each text into the LLM. Use the [EOS] vector of the last layer as the embedding. With causal attention, the [EOS] representation depends on all tokens before it. Thus, it describes the full input (Section 4.2).
- Train with the standard contrastive (InfoNCE) loss. Use in-batch negatives and the generated hard negatives.
12.3 Training stages and the key observation
- Contrastive pre-training (from the original BERT-based E5): text pairs with weak supervision from filtered data. For example, cut a passage into two parts. One part is the query. The other part is the relevant passage.
- Contrastive fine-tuning: synthetic data from the LLM + labelled MS MARCO data (and, in one version, 18 more datasets).
PracticeGive the two different ways in which E5-Mistral uses LLMs. Which one is still necessary if you have much human-labelled training data?
PracticeWhy does E5-Mistral add [EOS] to queries and to documents?
13. LLM2Vec: turning any decoder into an encoder
Sources: slides 44–46 · Video 8 (LLM2Vec) · In-class slide 16
BehnamGhader et al., "LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders", 2024. LLM2Vec is a general method. It changes any decoder-only LLM into a strong text encoder in three steps. A fourth step is optional:
- Enable bidirectional attention. Replace the causal attention mask with a matrix of ones. Then each token can attend to all other tokens. The decoder becomes bidirectional, like an encoder. But this step alone decreases the embedding performance. The weights of the model never learned to attend to future tokens.
- Masked next-token prediction (MNTP). This step teaches the model to use its new bidirectional attention. Mask one token in the sequence. Train the model to predict it. The special rule: to predict the masked token at position , compute the loss from the logits at position . This agrees with the next-token habit of the decoder. Slide 45 gives the reason for this step. Decoder pre-training has no next-sentence-prediction objective. Thus, the model never learned to capture the context of the full sequence.
- Unsupervised contrastive learning (SimCSE). Put the same input sequence into the model two times, with two different random dropout masks. The dropout makes the two outputs a little different. The result is two "views" of one sentence. Train the model to make the similarity of the two views high. Train it to make the similarity to the other sequences in the batch low. This is an InfoNCE-type loss (Section 7.1). No labels are necessary.
- (Optional) Supervised contrastive learning on labelled retrieval data or sentence-similarity data. This gives more improvement.
Worked example 9What changes when the mask flips, and how MNTP works
Use the 4-token sequence the cat sat down.
the cat [MASK] down. The model must predict "sat". The loss comes from the logits at position (the representation of "cat"). To solve this task, the weights must use attention to future tokens ("down" is to the right of the mask). This makes the bidirectional attention useful.the cat sat down two times with different dropout. This gives vectors and . Train with a contrastive loss. The loss makes high. It makes low for each other sentence in the batch. The model learns sentence embeddings with no labels.PracticeWhy does bidirectional attention alone (LLM2Vec step 1) decrease performance? Which step corrects this?
PracticeA sequence has 6 tokens. How many ones are in the causal mask? How many are in the mask of ones?
14. RepLLaMA: supervised LLM retrieval with LoRA
Sources: slides 47–48 · Video 9 (RepLLaMA) · In-class slide 16
Ma et al., "Fine-Tuning LLaMA for Multi-Stage Text Retrieval", 2023. RepLLaMA uses the same architecture as E5-Mistral: a decoder-only backbone, a query with an instruction, an added [EOS], and the last hidden state of [EOS] as the embedding. Its backbone is Llama-2-7B.
| Design choice | RepLLaMA |
|---|---|
| Backbone | Llama-2-7B (decoder-only, open source) |
| Representation | [EOS] token embedding (last hidden state) |
| Similarity | Dot product |
| Loss | Negative log likelihood / InfoNCE (the same as DPR and ANCE) |
| Training | Only supervised labels (as DPR) + LoRA adapters |
Two practical results make it different:
- No synthetic data is necessary. E5-Mistral uses GPT-4 to make synthetic data. RepLLaMA shows that a strong LLM backbone does not need this step. It trains directly on supervised retrieval labels (as DPR does, for example on MS MARCO). This already works well.
- LoRA adapters for efficient training. Fine-tuning does not change all 7 billion parameters. LoRA (Low-Rank Adaptation) trains only small adapter modules on the projection layers. This decreases training time and computation very much. Retrieval effectiveness stays competitive. Thus, LLM-based retrievers become practical to train.
The lecture gives the name LoRA without the maths. For background, this is the standard LoRA update for one weight matrix:
- the pretrained weight matrix of one projection layer. It stays frozen.
- the two small adapter matrices that training changes
- the rank of the adapter (for example, 8 or 16). It is much smaller than and .
- their product: a low-rank correction that the model adds to
- the weight that the forward pass uses
Worked example 10How much smaller is a LoRA update?
Illustrative: one projection matrix (the hidden size of Llama-2-7B is 4096) and LoRA rank .
PracticeWhat do E5-Mistral and RepLLaMA share? On which design dimension are they most different?
PracticeDoes LoRA make the encoding of a RepLLaMA query faster at search time?
15. Model comparison summary
Sources: slides 18–48 · Videos 2–9 · In-class slides 10–12, 16
| Model | Backbone | Representation | Similarity | Loss | Negatives / training signal |
|---|---|---|---|---|---|
| DPR (2020) | Encoder | [CLS] embedding | Dot product | InfoNCE (negative log likelihood) | Random + BM25 hard + in-batch |
| RepBERT (2020) | Encoder | Mean of token embeddings | Inner product | Multi-label margin loss | In-batch |
| ANCE (2021) | Encoder | [CLS] embedding | Inner product | InfoNCE | Self-mined hard negatives with ANN over training checkpoints |
| Contriever (2021) | Encoder | Mean of token embeddings | Cosine similarity | InfoNCE | Unsupervised: two spans of the same document = positive pair |
| ColBERT (2020) | Encoder (multi-vector) | All token embeddings | MaxSim (late interaction) | InfoNCE | Standard triples (as in DPR-style training) |
| E5-Mistral (2023/24) | Decoder, Mistral-7B | [EOS] last hidden state, query with instruction | Cosine/dot (contrastive) | InfoNCE | GPT-4 synthetic ⟨q, p⁺, hard p⁻⟩ + MS MARCO; in-batch + hard |
| LLM2Vec (2024) | Decoder (any LLM) | Mean pooling over the bidirectional LLM | Cosine (contrastive) | SimCSE contrastive (+ optional supervised) | Unsupervised: dropout views of the same text (+ optional labels) |
| RepLLaMA (2023) | Decoder, Llama-2-7B | [EOS] last hidden state, query with instruction | Dot product | InfoNCE | Only supervised labels (MS MARCO), trained with LoRA adapters |
PracticeWhich two models in the table use cosine similarity on mean-pooled vectors? What else do they share?
PracticeWhich model is the only one of this week that trains with a margin loss instead of InfoNCE? Which model is the only multi-vector model?
16. Course context from the in-person lecture
Sources: In-class slides 2–7 · slides 4–6
The in-person session started with the "historical view of ranking models" timeline of the course. It shows how the weeks connect. It also shows which topics the exam covers:
The lecturer also told how the weeks build on each other. Weeks 1, 2 and 4 give preliminary knowledge. Weeks 3, 5, 6, 7 and 8 are the core models. This week is a core week. Weeks 9–10 cover new research. Weeks 11–13 are also important. In the cost table of relevance models, the last empty slot is for learned sparse retrievers, the topic of next week. The session also gave the course survey result (average 4.54 out of 5 from 24 responses). Students asked that lectures also include concepts. For this reason, the session included the transformer recap in Section 17.
16.1 The models at a glance: years and one-line innovations
The in-person slides show each model as a card: representation, similarity and loss. Each card also has a one-line "innovation" that makes the model important. Use this table as a revision checklist:
| Model (year) | Card (representation · similarity · loss) | Innovation in one line |
|---|---|---|
| DPR (2020) | [CLS] embedding · dot product · InfoNCE | The influential dual-encoder model. |
| RepBERT (2020) | Mean embedding · dot product · multi-label margin loss | Strong early BERT dual-encoder baseline. Mean pooling + margin loss is better than BM25 on MS MARCO. |
| ANCE (2021) | [CLS] embedding · dot product · InfoNCE | Asynchronous negative sampling from an index that changes during training. |
| Contriever (2021) | Mean embedding · cosine similarity · InfoNCE | Trains without labels on very large corpora with contrastive learning. |
| ColBERT (2020) | All token embeddings · MaxSim · InfoNCE | Efficient late interaction: token-level query and document embeddings, scored with MaxSim. |
| E5-Mistral (2024) | [EOS] embedding · dot product · InfoNCE | Supervised synthetic data from GPT-4 + MS MARCO labels. |
| LLM2Vec (2024) | Mean pooling · dot product · InfoNCE | Bidirectional attention + masked next-token prediction + unsupervised SimCSE. Supervised tuning is optional. |
| RepLLaMA (2023/24) | [EOS] embedding · dot product · InfoNCE | LLaMA bi-encoder trained on MS MARCO with hard and in-batch negatives. |
Remember these group remarks from the lecture. DPR and RepBERT are the early series (2020). The card of ANCE shows that negative sampling is important during training. The card of Contriever shows that these models can train without supervised labels.
PracticeWhich weeks did the lecturer call the "core models"? Which two of them cover neural first-stage retrievers?
17. Transformer recap from the in-person session
Sources: In-class slides 9, 14–15 · Week 4 lecture notes
The survey feedback said: "lecture should include concepts as well". Thus, the in-person session taught the transformer pipeline from Week 4 again before the exercises. All topics of this week use it. This is a short version of the recap:
- Tokenisation (WordPiece). The tokeniser cuts text into sub-word tokens. Each token has an integer ID. The vocabulary (approximately 30K for BERT; it changes with the model) comes from corpus statistics. The method joins character sequences that frequently occur together. Thus, "playing" can become
play+##ing. Different model families (BERT, Llama, GPT) train different tokenisers on different collections. Thus, the same text gives different tokens. - The model adds the input embeddings, element by element. The input vector of each token = token embedding + segment embedding + position embedding. This is vector addition. The dimension does not change. (This was an in-class exercise. The final input embedding of [CLS] is its token embedding + the segment-A embedding + the position-0 embedding.) Position embeddings are necessary because "I asked him" and "he asked me" have the same words in a different order. Segment embeddings (only in BERT, optional in other models) separate sentence A from sentence B.
- Special tokens. [CLS]: a slot for classification or summary. Retrievers use it very much. [SEP]: separates parts of the input. [PAD]: makes all sequences in a batch the same length for parallel GPU processing. [MASK]: BERT training uses it. Decoders use [BOS]/[EOS] to mark the start and end of a sequence. [EOS] tells generation when to stop. E5-Mistral and RepLLaMA use the [EOS] vector as the embedding.
- Self-attention (Q, K, V). Three learned weight matrices change the input vector of each token into three vectors. The query vector means "what do I look for". The key vector means "what do I give to others". The value vector means "how do I express myself". The output dimension is equal to the input dimension. Thus, you can stack the blocks (for example, 12 layers with more than one head in each layer). The attention formula is below.
- Encoder and decoder attention. The encoder is bidirectional. Each token attends to all other tokens, before and after it. The encoder processes all tokens in parallel. This parallel work is the reason why transformers operate well on GPUs. The decoder is causal. Each token attends only to earlier tokens, because future tokens do not exist yet during generation.
- How a decoder generates. Take the hidden vector of the last token from the last layer (dimension , for example 768 for BERT size or 2048+ for Llama). Multiply it by a learned linear layer ( = vocabulary size). This gives one logit for each vocabulary token. Softmax changes the logits into probabilities. The model selects a token (greedy selection or sampling). Sampling is the reason why the same ChatGPT prompt gives different answers. The model adds the selected token to the input. The loop continues until [EOS].
- Pre-training. BERT: masked language modelling. The method masks some random tokens and predicts them from both sides. GPT: next-token prediction over very large text collections. Both methods are unsupervised. For this reason, pre-training at web scale is possible.
- the query vector of token (the token that attends)
- the key vector and the value vector of token
- the dimension of the query and key vectors. keeps the dot products in a stable range.
- normalises over all tokens that token can see. The weights are positive and their sum is 1.
- the attention weight that token gives to token
- the output vector of token
PracticeBERT-base uses embeddings with 768 dimensions. What is the dimension of the input vector of the [CLS] token after the model adds its token, segment and position embeddings?
18. Q&A from the in-person lecture
Sources: In-class slides 21–22
The discussion was the most useful part of the live session. These are the questions (from students or from in-class exercises) and the answers, in a clear form.
18.1 Transformer mechanics
Q: Is the self-attention matrix symmetric?
A: No. Attention has a direction. The attention of token to token can be different from the attention of to . The lecturer said: "I care about that token; that token doesn't have to care about me." Each row comes from a different query vector. Thus, usually .
Q (in-class MCQ): Which statements about the transformer are correct?
A: "The encoder uses masked attention to prevent looking at future tokens": false. This describes the decoder. The encoder must see the full sentence to capture its meaning. "The decoder uses bidirectional attention": false. The two are the wrong way around again. "The encoder processes input tokens in parallel using bidirectional attention": true. This parallel work is the reason why transformers use GPUs well. "Both encoder and decoder use the same type of attention": false. One is bidirectional and one is causal.
Q: Does BERT have more layers than GPT? Do they use different tokenisation?
A: The number of layers is a free design choice. In practice, BERT (approximately 110M parameters) is much smaller than modern GPT-style models. But you can build an encoder with more layers. Generation is the harder task. For this reason, decoder models grew to billions or trillions of parameters (the "scaling law"). Both families use word-piece-style tokenisation. The difference is the trained vocabulary (size and contents). Each vocabulary comes from a different collection with different goals.
Q: In the exercise, how do the three embeddings combine? Into a matrix?
A: The model adds the vectors: first dimension + first dimension + first dimension, and so on. Each token gets one vector with the model dimension.
Q: What does the linear layer of the decoder do?
A: It is a learned projection from the model dimension to the vocabulary size . Multiply the hidden vector of the last token by it. This gives a score (logit) for each vocabulary token. Softmax changes these scores into probabilities between 0 and 1. The model selects the next token from the top of this distribution. Deployed systems use sampling from the distribution. They do not always take the argmax. Thus, the same prompt can give different generations.
18.2 Using encoders and decoders for ranking (in-class Exercises 1 and 2)
Q: How can you get a text representation from a transformer encoder? How do you compute similarity?
A: There are three methods. This is the answer to Exercise 1. Use the [CLS] vector (DPR, ANCE). Calculate the mean of the token vectors (RepBERT, Contriever). Or keep more than one token vector (ColBERT). For the single-vector methods, similarity is one dot product or cosine between the query vector and the document vector. For the multi-vector method, similarity is late interaction. Compare each query token embedding with each document token embedding. Take the maximum for each query token. Add the maximum values (MaxSim). A student gave this method in class. The lecturer said that ColBERT started from almost the same idea.
Q: How can a transformer decoder compute query–document similarity? (Exercise 2. The class gave many answers. All are correct.)
- As an embedder: remove the generation head. Use the internal vector of the last token as a text embedding. Then score with dot product or cosine. E5-Mistral, LLM2Vec and RepLLaMA use this method.
- As a generative ranker: give the LLM a direct prompt. "Here is a query and 10 documents, give me an order" (listwise). "Is this document relevant: yes/no or a score" (pointwise). "Which of these two documents is more relevant" (pairwise).
- For query or document expansion (doc2query): generate a pseudo-document from the query. Or generate possible query terms for each document at indexing time. Then a lexical ranker (BM25) matches in the larger term space. This method attacks vocabulary mismatch from the lexical side.
- With output probabilities: use the probability distribution of the model over the full vocabulary as a representation of the text. Terms that the document makes probable get a high weight. The lecturer said that this is a preview of learned sparse retrievers, the topic of next week.
Q: Why does expansion occur at indexing time and not at query time?
A: Latency. At query time, the user waits. Long generation at that time makes the experience bad. The system processes documents offline at indexing time. There, extra minutes cost nothing. The same reason applies to the offline document embeddings of a bi-encoder.
18.3 Beyond this week
Q: Can an encoder backbone generate text?
A: Yes. This is the idea of diffusion language models, a very new research area. The encoder learned masked-token prediction. Thus, add, for example, 256 [MASK] tokens to the prompt. Predict them all at the same time (parallel work, good for GPUs). Do not predict them one at a time. The problem: tokens that the model predicts separately can be incoherent. The solution comes from image diffusion. Run some denoising passes. This needs far fewer passes than 256 auto-regressive steps. The lecturer said that his own recent work applies diffusion language models to retrieval.
Q (assignment): What must a gain–loss plot look like?
A: One bar for each query. The bar is the evaluation score of System A minus the score of System B on that query. Usually, sort the bars from the largest gain to the largest loss. Then you can quickly see if A is always better than B. More importantly, you can see which queries A loses on. Analyse those queries. The BM25 baseline in the assignment must be your own implementation. Read the criteria. If the criteria are not clear, ask on the discussion board.
PracticeGive two ways to use a decoder LLM for ranking without running it at query time over each document.
19. In-class exercises and where the solutions live
Sources: In-class slides 21–25 · Practical solution pages for Exercises 3–5
The session had five exercises. Exercises 1–2 are conceptual. The Q&A section above answers them. Exercises 3–5 are calculation or challenge questions. Their full step-by-step solutions are in the Practical/ folder:
| Exercise | Question | Answer | Solution |
|---|---|---|---|
| 1: Encoder-based | Methods to get a text representation from an encoder; which tokens; how to compute similarity | [CLS] / mean / all tokens; dot or cosine, or MaxSim | Section 18.2; Sections 4.1 and 11 |
| 2: Decoder-based | How can a decoder compute query–document similarity? | Last-token embedding; LLM as ranker; doc2query; representations from output probabilities | Section 18.2; Section 4.2 |
| 3: InfoNCE loss | sim(q, p⁺) = 0.9; negatives 0.3, 0.1, 0.4 | worked solution | |
| 4: Margin loss | Rel(q, p⁺) = 1.5; negatives 0.3, 0.1, 0.9; margin 1 | worked solution | |
| 5: Challenge | Find negatives that are harder than the top BM25 documents | ANCE: self-mined negatives, refreshed from checkpoints | worked solution |
Worked example 11Exercise 3 in brief: InfoNCE with three negatives
From in-class slide 23: . Negatives: .
Worked example 12Exercise 4 in brief: margin loss with three negatives
From in-class slide 24: . Negatives: . Margin: 1.
PracticeIn Exercise 4, how low must the score of the hardest negative go to make the loss zero?
PracticeExercise 5: training with BM25 hard negatives improves slowly. Give a method to find harder negatives.
20. Summary and study checklist
- Dense retrieval: embed the two texts with the same embedder, then score with a cheap similarity function: . It solves vocabulary mismatch. It measures distance in a learned semantic space instead of term overlap.
- Bi-encoders are fast at query time because the system computes document embeddings offline. The transformer cost does not occur at query time. The scorer cost is .
- Pooling choices: [CLS] (DPR, ANCE), mean (RepBERT, Contriever), all tokens (ColBERT), last/[EOS] token (decoder models: with causal attention, only this position sees the full input).
- Negative sampling is an important design decision. Random = easy. In-batch = medium (and free). BM25 = lexically hard. ANCE checkpoint negatives = hard for the dense model itself. Harder and newer negatives give better retrievers, at a higher training cost.
- ColBERT gives up scoring and 1-vector storage to get token-level MaxSim matching. It is usually more effective. It costs much more storage and scoring.
- LLMs can be retrievers. Remove generation and use the last-token embedding (E5-Mistral, RepLLaMA). Or change the decoder into a true bidirectional encoder (LLM2Vec). Their auto-regressive pre-training already gives strong text representations. Thus, contrastive pre-training becomes unnecessary. Fine-tuning adds mainly task knowledge.
- Labels are optional. Contriever (spans of the same document) and LLM2Vec (dropout views) train with a contrastive loss and no human judgments. E5-Mistral uses synthetic triples from an LLM.
Study checklist. You must be able to:
- Complete the cost table of relevance models (scorer cost and encoder cost) for lexical, single-vector and multi-vector models. Explain each entry.
- Compute a dot product, a cosine similarity and a mean-pooled vector by hand. Explain when the dot product and the cosine give different results.
- Write the InfoNCE loss. Define each symbol. Compute it for given similarities (Worked examples 3 and 11).
- Write the multi-label margin loss. Set each pair to zero if it is below zero. Calculate the average over pairs (Worked examples 5 and 12).
- Describe the three negative types of DPR. Count the in-batch negatives for a batch (Worked example 4).
- Explain why ANCE checkpoint negatives are better than BM25 negatives for a dense model. Give their cost.
- Make Contriever training pairs from a raw document.
- Compute a ColBERT MaxSim score and its cost in dot products and storage (Worked examples 7 and 8).
- Explain why decoder embedders use the [EOS]/last token. Give the three steps of LLM2Vec. Explain why step 1 alone decreases quality.
- Compare E5-Mistral and RepLLaMA, with the function of synthetic data and LoRA.
- Put each model of this week on the five design dimensions (backbone, representation, similarity, loss, sampling).
21. Video index
YouTube playlist for Week 5: IR Week 5. The transcripts are in week_5_video_transcripts.md. The slides of the in-person lecture are in Lecture-Slides/. The file INFS7410_S2_2026_STL-transcript.txt in this folder is a copy of the Week 4 lecture transcript. These notes do not use it.
| Video | Topic | Length | Sections here |
|---|---|---|---|
| Video 1 | Overview: embedding-based retrievers | 18:34 | 1–6 |
| Video 2 | DPR (Dense Passage Retrieval) | 11:10 | 5, 7 |
| Video 3 | RepBERT | 04:26 | 8 |
| Video 4 | ANCE | 03:43 | 9 |
| Video 5 | Contriever | 03:06 | 10 |
| Video 6 | ColBERT | 07:11 | 5.2, 11 |
| Video 7 | E5-Mistral | 07:58 | 12 |
| Video 8 | LLM2Vec | 04:34 | 13 |
| Video 9 | RepLLaMA | 03:09 | 14 |
22. References
- Karpukhin et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020. (DPR)
- Zhan et al. (2020). RepBERT: Contextualized Text Embeddings for First-Stage Retrieval. arXiv 2020. (RepBERT)
- Xiong et al. (2021). Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. ICLR 2021. (ANCE)
- Izacard, Caron, Hosseini, Riedel, Bojanowski, Joulin and Grave (2021). Unsupervised Dense Information Retrieval with Contrastive Learning. arXiv 2112.09118. (Contriever)
- Khattab and Zaharia (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. SIGIR 2020. (ColBERT)
- Wang, Yang, Huang, Yang, Majumder and Wei (2023). Improving Text Embeddings with Large Language Models. arXiv 2401.00368. (E5-Mistral)
- BehnamGhader, Adlakha, Mosbach, Bahdanau, Chapados and Reddy (2024). LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders. arXiv 2404.05961. (LLM2Vec)
- Ma et al. (2023). Fine-Tuning LLaMA for Multi-Stage Text Retrieval. (RepLLaMA)