INFS7410 · Information Retrieval and Web Search · Week 6

Learned Sparse Retrievers: TILDE, TILDEv2, SPLADE and PromptReps

Lexical retrieval models with learned term weights. This week combines the efficiency of inverted-index lexical search with the semantic power of transformers. It covers query and document expansion (doc2query), the TILDE family, the SPLADE family, an effectiveness comparison across every retriever seen so far, the training-free LLM-based PromptReps, and the exercises from the in-class lecture.

1. Where Week 6 fits: the landscape of relevance models

Sources: slides 2–3 · lecture slides 2–5 · Video 1 (Overview) · Lecture transcript

1.1 The course so far

The in-class lecture opened with a recap of the ranking models covered so far. Week 1 defined information retrieval and text processing (tokenisation, stemming, stopwords). Week 2 covered evaluation (set-based and rank-based measures, pooling, statistical tests). Week 3 introduced the term-based lexical models (Boolean model, TF-IDF, the vector space model, BIM and BM25). Week 4 explained the transformer architecture (encoder and decoder, BERT and GPT). Week 5 turned transformers into retrievers (DPR, ANCE, RepBERT, ColBERT, E5-Mistral, LLM2Vec, RepLLaMA). Week 6 covers learned sparse models (TILDE, TILDEv2, SPLADE, SPLADEv2) and PromptReps.

On the lecture's historical timeline, learned sparse models sit beside dense retrievers in the "new inventions" era (2018–2022), after term-based lexical models (1950s–1990s) and the transformer (2017–2018). The lecturer's point: learned sparse models are a mix of the Week 4 transformer architecture and the ideas behind the Week 3 term-based lexical models.

Review the Week 3 lexical models, Week 4 transformer and MLM head, and Week 5 dense retrievers alongside this comparison.

1.2 The table of relevance models

Over the past weeks we have built up a table of relevance model families. Term-based lexical models such as BM25 (Week 3) match exact words through an inverted index (Week 1). Single-vector dense retrievers such as RepBERT and ANCE (Week 5) encode a whole query or document into one embedding. Multi-vector dense retrievers such as ColBERT keep one embedding per token. This week fills the remaining cell: learned sparse retrievers, also called lexical-based retrieval models with learned term weights. They sit between traditional lexical models and dense embedding models and try to get the best of both.

FamilyEncodingOnline scoringStored representation
BM25Tokenisation and term statisticsTraverse posting lists for query termsSparse term weights
TILDE-QLBERT on documents offline; tokeniser on queries onlineLook up query-token weights for each candidateDense vocabulary-sized document vector; sparse query
TILDEv2 / SPLADELearn document weights offline; standard SPLADE also encodes queries onlineUse shared active vocabulary termsSparse document weights
Single-vector denseTransformer on each text; document encoding offlineOne vector comparison per candidate; ANN can select candidatesOne dense vector per document
Multi-vector denseA similar transformer backbone; retain token vectorsToken-to-token comparisons and MaxSim aggregationSeveral dense vectors per document

Reading the lecture's cost diagram. Separate encoding a text, scoring one candidate, and searching a collection. The lecturer says that single-vector and multi-vector models can use the same encoder backbone. Keeping token vectors increases storage and scoring work. Full transformer self-attention considers token pairs in either family. The slide's “O(<1)” label is a qualitative efficiency cue. Actual search cost depends on posting-list lengths, active terms, candidate counts, and the index. Dense retrieval can use approximate nearest-neighbour (ANN) search to avoid an exhaustive scan.

query-time scoring cost (low to high) effectiveness (semantic matching) Term-based lexical BM25 · posting-list scoring Learned sparse (Week 6) TILDE · TILDEv2 · SPLADE Single-vector dense RepBERT · ANCE · one vector Multi-vector dense ColBERT · MaxSim scoring learn term weights, keep sparsity
Qualitative positions: learned sparse retrievers keep the low scoring cost of lexical models and move up towards the effectiveness of dense retrievers.
PracticeWhy is ColBERT's scoring more expensive than RepBERT's, and which column of the table do learned sparse retrievers improve on?
Count how many vectors each model keeps per query and per document.
AnswerRepBERT keeps one vector per query and per document, so scoring one document is a single dot product. ColBERT keeps one vector per token, so it must compare every query token vector with every document token vector (MaxSim), which multiplies the work. Learned sparse retrievers improve the scorer column: their sparse vectors let the engine score only documents that share active terms with the query, just like BM25.

2. Recap: term-based lexical vs dense retrievers

Sources: slides 4–6 · lecture slide 6 · Video 1 (Overview) · Lecture transcript

2.1 Term-based lexical models: high efficiency

For the query "machine learning algorithms", a traditional system like BM25 looks only at documents containing the exact words machine, learning, or algorithms. The corpus is organised in an inverted index: for each term, a posting list of the documents that contain it. At query time the engine only loops over the posting lists of the query terms, and this set of candidate documents is usually a tiny fraction of the whole collection. That is the source of the high efficiency of lexical models. The lecture added a practical reminder: BM25 runs on a CPU in a few milliseconds per query (the Assignment 1 BM25 runs should take minutes, and hours signal a bug).

Corpus docs with “machine” docs with “learning” docs with “algorithms” ∩ Query: machine learning algorithms • Only documents that share a query term are ever scored • The engine walks short posting lists, never the whole collection Benefit: high efficiency Limitation: exact term overlap only. “AI” or “neural networks” never match
Lexical retrieval only examines the small subsets of the corpus whose documents contain query terms.

2.2 Dense retrievers: high effectiveness

Dense embedding-based retrievers (Week 5) push both queries and documents through transformer encoders to produce dense vectors, and rank by vector similarity (cosine similarity, dot product, or even Euclidean distance). They are much more effective because they do semantic matching: they understand that machine learning relates to artificial intelligence, neural networks or transformers even when no word overlaps. The price is a computationally heavy encoder, and scoring that in principle involves every document vector in the collection.

Key idea Lexical models: highly efficient, limited in effectiveness (exact matching only). Dense models: highly effective (semantic matching), computationally expensive. The central question of Week 6: can we combine their advantages?
PracticeA user searches "AI tutoring". A relevant document talks only about "machine learning tutors". Which retriever family ranks it well, and what does the other family pay for that?
AnswerBM25 matches only "tutoring" (if stemming maps it to "tutor") and misses the link between "AI" and "machine learning", so it under-ranks the document. A dense retriever places both texts close together in embedding space and ranks it well. The dense retriever pays with a transformer encoding of the query at search time and vector similarity search (exhaustive or ANN).

3. The core idea of learned sparse retrievers

Sources: slides 7–11 · Video 1 (Overview) · Lecture transcript

Traditional term-based lexical retrievers have three limitations. First, their weighting is based solely on term overlap and collection statistics (term frequency, document frequency, document length). Second, they use hand-crafted formulas designed from intuition and some theoretical foundations. Third, they learn nothing from data: apart from a couple of tunable parameters, the same formula is applied to every query, domain, and document type.

Learned sparse models take a different approach:

This week presents two main routes to this goal, plus a recent LLM-based hybrid:

RouteIdea in one lineModels
Drop the query encoderTreat queries and documents asymmetrically: the query is only tokenised into a sparse term-existence vector (no neural network at query time), while documents are still encoded into dense vocabulary-sized vectors offline. Document indexing and encoding time stay the same; query encoding time almost disappears; the sparse query vector also cuts scoring time.TILDE, TILDEv2
Keep the encoder, output term weightsUse the transformer encoder for both queries and documents, and make it output weights over the whole vocabulary (an "expanded set of terms"). Encoding time stays the same as a dense model; scoring becomes cheap because the vectors are sparse.SPLADE, SPLADEv2
Prompt an LLM, no training at allPrompt a generative LLM to summarise a text in one word, and harvest a dense vector (hidden state) plus a sparse vector (next-token logits) from that single inference.PromptReps
Route 1: drop the query encoder TILDE, TILDEv2 Route 2: encode both, output weights SPLADE, SPLADEv2 Query Document Query Document BERT tokeniserlookup, no GPU BERT + linearoffline, at indexing BERT + MLM headonline, at query time BERT + MLM headoffline, at indexing binary sparse1 if the term is in q dense |V| vectora weight for every term sparse |V| vectorlearned weights sparse |V| vectorlearned weights Score: inner productsaves query encoding and scoring time Score: sparse dot productsaves scoring time only
The two main routes: TILDE removes the query encoder (asymmetric), SPLADE keeps both encoders and makes their outputs sparse.

3.1 Why sparsity is the efficiency

Both routes end with a dot product in the ∣V∣ |V| -dimensional vocabulary space. Written out, it shows exactly where the saving comes from:

s(q,d)=∑i=1∣V∣qi di=∑i : qi≠0qi di s(q,d) = \sum_{i=1}^{|V|} q_i\, d_i = \sum_{i \,:\, q_i \neq 0} q_i\, d_i
Where
s(q,d)s(q,d)
relevance score of document dd for query qq
∣V∣|V|
vocabulary size (30,522 for the BERT vocabulary)
ii
index of one vocabulary term (one dimension)
qiq_i
weight of term ii in the query vector
did_i
weight of term ii in the document vector
i:qi≠0i : q_i \neq 0
only the dimensions where the query weight is non-zero
Read it aloud The score of document d for query q is the sum, over every vocabulary term i, of the query weight times the document weight, which equals the same sum taken only over the terms where the query weight is non-zero.
Formula in plain words Every dimension where the query weight is zero contributes zero, whatever the document holds there, so the engine skips it. The work per document therefore grows with the number of active query terms, a handful, instead of the 30,522 vocabulary dimensions. If most weights are exactly zero, the engine can also precompute posting lists over the non-zero terms and score only documents that share active terms with the query, exactly like BM25. A query that activates more terms (for example through expansion) costs proportionally more to score.
Key idea Sparsity is the efficiency. A learned sparse model is only as fast as its vectors are sparse: if every weight were non-zero, the dot product would cost as much as a 30,522-dimensional dense one.
PracticeA sparse query vector has 4 non-zero entries out of 30,522. How many multiplications does scoring one document need, and how many would a fully dense 30,522-dimensional vector need?
Answer4 multiplications (one per non-zero query entry, plus 3 additions), against 30,522 for the dense case. The zero entries contribute nothing, so the engine never computes them.
PracticeName the three limitations of BM25-style retrievers that learned sparse models address, and the property they keep from BM25.
Answer(1) Weights come only from term overlap and collection statistics; (2) the formulas are hand-crafted; (3) nothing is learned from data. Learned sparse models learn term weights from training data with a transformer, and they keep BM25's sparsity, which preserves inverted-index efficiency.

4. Query and document expansion, and doc2query

Sources: slides 11–18 · lecture slide 7 · Video 1 (Overview) · Video 2 (doc2query) · Lecture transcript

4.1 What is expansion?

Query/document expansion means augmenting the user query, or a document in the corpus, with additional relevant terms. The goal is to improve effectiveness by matching related terms that are absent from the original text. This attacks the vocabulary mismatch problem directly (a query says "lower blood sugar", the document says "reduces blood sugar levels"). A variety of automatic and semi-automatic expansion techniques exist; semi-automatic ones require user interaction to select the best expansion terms, automatic ones need no user input. Query/document rewriting is a closely related technique. Many learned sparse retrievers rely on expansion for better effectiveness, which is why it comes first. The lecturer noted that expansion returns in more depth in Week 12.

The in-class lecture used this example. The original query "who wrote the book of the lord of the rings" is expanded with terms that capture what the question means: write, created, authored, novel, LOTR, Tolkien. The expanded terms are new words (repeating the query's own words adds nothing), and the expanded query is the original query plus these terms. The same idea applies on the document side: a document about "AI" expanded with "machine" and "learning" now matches a query that uses those words.

Original query: who wrote the book of the lord of the rings expansion adds related terms write created authored novel LOTR Tolkien Expanded query = original words + expanded terms documents that say “Tolkien” or “novel” now share terms with the query
Expansion adds related terms so that relevant documents with different wording still share terms with the query (lecture slide 7).

4.2 doc2query: document expansion with a seq2seq transformer

With document expansion we add terms (or modify term weights) in the index to improve performance. doc2query does this with a sequence-to-sequence transformer: a transformer with both an encoder and a decoder that maps input text to output text (T5 is a common example). BERT, by contrast, maps input text to embeddings. The model's task: given a document, generate a query that this document would answer. Training is supervised, using pairs of ⟨query, relevant document⟩ (e.g. from MS MARCO).

Input: document “Researchers are finding that cinnamon reduces blood sugar levels naturally when taken daily …” doc2query seq2seq, e.g. T5 Output: queries “does cinnamon lower blood sugar?” … 5–40 queries per document top-k or nucleus sampling + concatenate Expanded document = original text + predicted queries original document text appended queries Inverted index + BM25 user queries match the expanded text indexing runs offline: no extra query-time cost
The doc2query pipeline: generate 5–40 queries per document, concatenate them onto the document, and index the expanded documents as usual.

In practice, between 5 and 40 queries are sampled per document (with top-k or nucleus sampling). Each document is concatenated with its predicted queries to form an expanded document, which is indexed into a traditional inverted index. When a user searches, the engine (e.g. BM25) matches against the expanded index, which bridges much of the vocabulary gap between query and document.

4.3 Does it work?

Yes. The expansion strategy consistently improves results over the same ranking model on non-expanded documents, regardless of the ranker (a bag-of-words model, BM25 with RM3 pseudo-relevance feedback, or BM25 followed by a monoBERT/T5 reranker):

RankerMS MARCO Passage (MRR@10)TREC-DL 19 (nDCG@10)
BM250.1840.506
BM25 + doc2query0.2770.642
BM25 + RM3 (pseudo-relevance feedback)0.1560.518
BM25 + RM3 + doc2query0.2140.655
BM25 + monoBERT/T5 reranking0.3650.738
BM25 + monoBERT/T5 + doc2query0.3790.754

Inspecting the generated queries shows doc2query does two useful things at once: it injects new words into the document (directly addressing keyword mismatch), and it copies/repeats keywords already in the document. Excluding stop words, about 69% of generated words are copied and 31% are new. Repetition matters too: it increases the importance (term frequency) of those keywords in the expanded document. The slide's three examples:

Input document (shortened)Predicted queryTarget queryNew words
"July is the hottest month in Washington DC with an average temperature of 27C …"weather in washington dcwhat is the temperature in washingtonweather
"The Delaware River flows through Philadelphia into the Delaware Bay …"what river flows through delawarewhere does the delaware river start and endnone (all copied)
"sex chromosome … a threadlike strand of DNA in the cell nucleus that carries the genes …"what is the relationship between genes and chromosomeswhich chromosome controls sex characteristicsrelationship
Ablation (MS MARCO passage)MRR@10R@1000
Original document0.1840.853
+ expansion with new words only0.1950.907
+ expansion with copied words only0.2210.893
+ expansion copied + new0.2770.944
Only expansions (original document removed)0.2630.927
Takeaways Both kinds of expansion help, and copied words contribute more than new words (0.221 against 0.195 MRR@10). Strikingly, indexing only the generated queries, with the original document removed (0.263), beats indexing the original documents alone (0.184).

Worked example 1How doc2query expansion changes a BM25 score

Setup (illustrative numbers). A user query q= q = "lower blood sugar" and a passage d d : "Researchers are finding that cinnamon reduces blood sugar levels naturally when taken daily" (13 terms, so ∣d∣=13 |d| = 13 ). Assume the collection has N=1,000,000 N = 1{,}000{,}000 passages, average length avgdl=15 avgdl = 15 , BM25 parameters k1=1.2 k_1 = 1.2 , b=0.75 b = 0.75 , and document frequencies df(lower)=40,000 df(\text{lower}) = 40{,}000 , df(blood)=25,000 df(\text{blood}) = 25{,}000 , df(sugar)=20,000 df(\text{sugar}) = 20{,}000 . We use the BM25 form from Week 3:

BM25(q,d)=∑t∈qln⁡ ⁣(N−df(t)+0.5df(t)+0.5+1)⋅tf(t,d) (k1+1)tf(t,d)+k1(1−b+b ∣d∣avgdl) \mathrm{BM25}(q,d) = \sum_{t \in q} \ln\!\Big(\frac{N - df(t) + 0.5}{df(t) + 0.5} + 1\Big) \cdot \frac{tf(t,d)\,(k_1+1)}{tf(t,d) + k_1\big(1 - b + b\,\frac{|d|}{avgdl}\big)}
Formula in plain words The score is the sum over every query term of two factors: an IDF factor (the log of roughly "how rare the term is across the collection", so rarer terms count more) multiplied by a saturating term-frequency factor (the more often the term appears in the document the higher the factor, with diminishing returns, adjusted for whether the document is longer or shorter than average). Here tf(t,d) tf(t,d) is the count of t t in d d , df(t) df(t) the number of documents containing t t , N N the collection size, k1 k_1 the saturation parameter and b b the length-normalisation parameter.
1
Score the original document. Term frequencies in d d : tf(lower)=0 tf(\text{lower}) = 0 , tf(blood)=1 tf(\text{blood}) = 1 , tf(sugar)=1 tf(\text{sugar}) = 1 . The passage says "reduces", so "lower" never appears. Its contribution is 0: this is vocabulary mismatch.
2
Compute the shared pieces. Length normalisation: 1−b+b∣d∣avgdl=1−0.75+0.75×1315=0.25+0.65=0.90 \begin{aligned} 1 - b + b\frac{|d|}{avgdl} &= 1 - 0.75 + 0.75 \times \frac{13}{15} \\ &= 0.25 + 0.65 = 0.90 \end{aligned} So the TF factor for a term with tf=1 tf = 1 is 1×2.21+1.2×0.90=2.22.08=1.058 \frac{1 \times 2.2}{1 + 1.2 \times 0.90} = \frac{2.2}{2.08} = 1.058
3
Compute IDFs. idf(blood)=ln⁡(1,000,000−25,000+0.525,000+0.5+1)=ln⁡(40.0)=3.689idf(sugar)=ln⁡(980,00020,000.5+1)=ln⁡(50.0)=3.912idf(lower)=ln⁡(960,00040,000.5+1)=ln⁡(25.0)=3.219 \begin{aligned} \mathrm{idf}(\text{blood}) &= \ln\Big(\frac{1{,}000{,}000 - 25{,}000 + 0.5}{25{,}000 + 0.5} + 1\Big) \\ &= \ln(40.0) = 3.689 \\ \mathrm{idf}(\text{sugar}) &= \ln\Big(\frac{980{,}000}{20{,}000.5} + 1\Big) = \ln(50.0) = 3.912 \\ \mathrm{idf}(\text{lower}) &= \ln\Big(\frac{960{,}000}{40{,}000.5} + 1\Big) = \ln(25.0) = 3.219 \end{aligned} The value for "lower" is unused for now, since tf=0 tf = 0 .
4
Original score. BM25=3.689×1.058+3.912×1.058=3.903+4.139=8.04 \mathrm{BM25} = 3.689 \times 1.058 + 3.912 \times 1.058 = 3.903 + 4.139 = \mathbf{8.04} . The term "lower" contributed nothing.
5
Expand the document. doc2query predicts the query "does cinnamon lower blood sugar" (5 terms), which we append. The expanded document d′ d' has ∣d′∣=18 |d'| = 18 terms and new counts: tf(lower)=1 tf(\text{lower}) = 1 , tf(blood)=2 tf(\text{blood}) = 2 , tf(sugar)=2 tf(\text{sugar}) = 2 . Both effects from the lecture appear: a new word ("lower") was injected, and existing words ("blood", "sugar") were copied, raising their term frequency.
6
Recompute the pieces for d′ d' . length normalisation:0.25+0.75×1815=0.25+0.90=1.15denominator constant:k1×1.15=1.38TF factor, tf=1:2.21+1.38=0.924TF factor, tf=2:2×2.22+1.38=4.43.38=1.302 \begin{aligned} \text{length normalisation:}\quad 0.25 + 0.75 \times \frac{18}{15} &= 0.25 + 0.90 = 1.15 \\ \text{denominator constant:}\quad k_1 \times 1.15 &= 1.38 \\ \text{TF factor, }tf = 1\text{:}\quad \frac{2.2}{1 + 1.38} &= 0.924 \\ \text{TF factor, }tf = 2\text{:}\quad \frac{2 \times 2.2}{2 + 1.38} &= \frac{4.4}{3.38} = 1.302 \end{aligned} That is larger than for tf=1 tf=1 and less than double: BM25's term frequency saturates.
7
Expanded score. BM25=3.219×0.924⏟lower+3.689×1.302⏟blood+3.912×1.302⏟sugar=2.974+4.803+5.093=12.87 \mathrm{BM25} = \underbrace{3.219 \times 0.924}_{\text{lower}} + \underbrace{3.689 \times 1.302}_{\text{blood}} + \underbrace{3.912 \times 1.302}_{\text{sugar}} = 2.974 + 4.803 + 5.093 = \mathbf{12.87} .
Result Expansion lifted the document's score for this query from 8.04 to 12.87, and, more importantly, the previously unmatched query term "lower" now matches. Query time is unchanged: the expansion work happened once, offline, at indexing. This is the mechanism behind the 0.184 → 0.277 MRR@10 jump for BM25 on MS MARCO.
PracticeThe ablation shows that expansion with copied words alone improves MRR@10. How can adding words that are already in the document help BM25?
Think about which BM25 input the copies change.
AnswerCopies raise the term frequency of the words that the generated queries consider important, so BM25 gives those words a larger TF factor. The document's representation then emphasises the terms users actually search for (0.184 → 0.221 MRR@10 with copied words only).
PracticeWhy does doc2query add no cost at query time, and where does its cost go instead?
AnswerThe seq2seq model runs once per document at indexing time; the expanded text sits in an ordinary inverted index, and queries are processed by plain BM25. The cost moves offline (generation for every document, plus a larger index). Section 6 shows this offline cost can still be very large.

5. TILDE: Term Independent Likelihood moDEl

Sources: slides 8–9, slides 19–26 · lecture slides 8–14, 21 · Video 3 (TILDE) · Lecture transcript

5.1 The idea: replace the query encoder with a tokeniser

TILDE (Zhuang & Zuccon, SIGIR 2021) takes the first route from Section 3: remove query encoding entirely. In a dense bi-encoder (Week 5), both towers run BERT. TILDE keeps the document tower and replaces the query tower with the plain BERT tokeniser, the first step of the Week 4 input pipeline, before any embedding or encoder layer:

s(q,d)=q⊤d=∑i=1∣V∣qi di=∑ti∈qdi s(q, d) = \mathbf{q}^{\top} \mathbf{d} = \sum_{i=1}^{|V|} q_i \, d_i = \sum_{t_i \in q} d_i
Where
q\mathbf{q}
binary query vector of length ∣V∣|V|, produced by the BERT tokeniser
d\mathbf{d}
dense document vector of length ∣V∣|V|: [CLS] embedding passed through the linear layer
qiq_i
1 if vocabulary token tit_i occurs in the query, else 0
did_i
learned weight (probability) of token tit_i for the document
ti∈qt_i \in q
the vocabulary tokens that occur in the query
∣V∣|V|
BERT vocabulary size, 30,522
Read it aloud The score s of document d for query q is the transpose of q times d, which is the sum over all vocabulary positions i of q-i times d-i, which equals the sum of d-i over the tokens t-i that occur in the query.
Formula in plain words Because query entries are only 0 or 1, the inner product collapses to one operation: add up the document's learned weights at the positions of the query's tokens. A 5-token query costs 5 look-ups and additions per document. A document scores higher when BERT assigned high weights to the query's tokens, even tokens that never appear in its text. Adding a query token adds one more term to the sum; the query side carries no weights at all, so all the learning lives in the document vector.
s inner product 0100…0100 Sparse vector · 1 × 30,522 0.110.330.050.15…0.220.10 Dense vector · 1 × 30,522 BERT tokeniser a lookup table: no GPU needed ⟨q⟩ query processed online, at query time Linear 768 → |V| BERT encoder: [CLS] output dense vector 1 × 768 ⟨d⟩ document precomputed offline, at indexing time
TILDE: the query side is just a tokeniser producing a binary sparse vector; the document side is BERT's [CLS] embedding projected to vocabulary size.
Takeaways (1) Semantic matching survives. Even if a query keyword never appears in the document, the document's dense vector (produced by BERT) can still assign that keyword a high weight: the model has learned which vocabulary terms the document is about. (2) No GPU at query time. The tokeniser is a small lookup table, so query "encoding" is nearly free and runs fine on a CPU.

Worked example 2Scoring with TILDE's sparse × dense inner product

Setup (illustrative numbers). To keep the numbers readable, use a toy vocabulary of 8 BERT tokens in place of 30,522: V={apple, store, account, fruit, buy, banana, login, the} V = \{\text{apple},\ \text{store},\ \text{account},\ \text{fruit},\ \text{buy},\ \text{banana},\ \text{login},\ \text{the}\} . The user query is "apple account". Two candidate passages were encoded offline; their [CLS] embeddings, after the linear projection, give these dense vocabulary-space weights (each row is Pθ(ti∣d) P_\theta(t_i \mid d) ):

Passageapplestoreaccountfruitbuybananaloginthe
d1 d_1 : "how to sign in to your Apple ID"0.620.280.510.040.190.020.440.30
d2 d_2 : "apples are a healthy fruit to snack on"0.660.070.030.580.210.350.010.31
1
Tokenise the query (no GPU). "apple account" activates dimensions apple and account: q=[1,0,1,0,0,0,0,0] \mathbf{q} = [1, 0, 1, 0, 0, 0, 0, 0] . Every other dimension is 0.
2
Inner product with d1 d_1 . Only the two active dimensions matter: s(q,d1)=1×0.62+1×0.51=1.13 s(q, d_1) = 1 \times 0.62 + 1 \times 0.51 = \mathbf{1.13} .
3
Inner product with d2 d_2 . s(q,d2)=1×0.66+1×0.03=0.69 s(q, d_2) = 1 \times 0.66 + 1 \times 0.03 = \mathbf{0.69} .
4
Rank. d1≻d2 d_1 \succ d_2 . Semantic matching is at work: d1 d_1 never contains the literal word "account", yet BERT learned that a passage about "signing in to your Apple ID" is about accounts, so its dense vector carries a high weight (0.51) on the account dimension. A purely lexical matcher would have missed this. d1 d_1 also has a weight of 0.44 on "login", another term absent from the passage: this is implicit expansion in the document representation.
5
TILDE-QL with log weights. Using log-likelihoods in place of raw probabilities: TILDE-QL(q∣d1)=ln⁡0.62+ln⁡0.51=−0.478−0.673=−1.151TILDE-QL(q∣d2)=ln⁡0.66+ln⁡0.03=−0.416−3.507=−3.923 \begin{aligned} \text{TILDE-QL}(q \mid d_1) &= \ln 0.62 + \ln 0.51 = -0.478 - 0.673 = -1.151 \\ \text{TILDE-QL}(q \mid d_2) &= \ln 0.66 + \ln 0.03 = -0.416 - 3.507 = -3.923 \end{aligned} Since −1.151>−3.923 -1.151 > -3.923 , the ranking is the same: d1 d_1 first.
Result Scoring each document cost two weight lookups and one addition after a tokeniser lookup. This is why TILDE's query latency is tiny and GPU-free, while it still rewards a document that matches the query only semantically.
Common trapThe lecture architecture illustrates a sum of term weights. For the TILDE-QL ranking rule in Section 5.4, store log-probabilities as those weights. Adding raw probabilities and adding log-probabilities can produce different rankings. Worked example 2 happens to give the same order with both. Each TILDE output is an independent token probability, so its vocabulary entries can sum to more than one.

5.2 Why TILDE is fast at query time

In the lecture's Exercise 1 (lecture slide 21) the class listed three separate reasons for TILDE's efficiency compared with dense retrievers:

  1. No query encoding. At query time the query is only tokenised; it never enters the BERT model. Tokenising is the first step before the input embeddings, so it is extremely fast.
  2. Document encoding happens offline. Documents still pass through BERT, but once, at indexing time. If the collection is static, the index never changes, however many queries arrive.
  3. The query vector is sparse. Only the query's own tokens are 1; all other vocabulary entries are 0. A query almost always has far fewer tokens than the vocabulary, so the dot product touches only a few dimensions. For each candidate, score only the active query-token dimensions.
Cross-encoder reranker (e.g. monoBERT) K BERT inferences Query + passagejoint input pair BERTruns once per pair Relevance scoreone candidate at a time Dense bi-encoder (Week 5) 1 BERT inference Queryonline input BERT query encoderone dense vector Dense searchANN or dot products TILDE-QL 0 BERT inferences Queryonline input BERT tokenisera few token IDs Look up and addprecomputed passage values
Query-time neural work for K candidate passages: TILDE-QL needs no BERT inference at all. Adding the TILDE-DL signal (Section 5.4) brings back one query inference.

The "term independent" in TILDE's name is what makes the offline work possible. TILDE assumes query tokens are independent of each other, so the likelihood of each vocabulary term depends only on the passage, Pθ(t∣d) P_\theta(t \mid d) . The model can therefore predict every term's likelihood for a passage before any query exists, store the results, and answer a query with look-ups. A term-dependent model, where each query term's probability also depends on the earlier query terms, has to run its neural inference after the query arrives.

Term-dependent query likelihood P(qᵢ | earlier q terms, d)needs the actual query Neural inferenceafter the query arrives Online scoringrepeated for every query TILDE: term-independent likelihood P(t | d) for every t in Vdepends on the passage only Offline BERT inferencepredict all |V| terms once Stored passage weightsready before any query Query time: look up and add
Term independence turns online neural inference into offline precomputation followed by query-time look-ups.

Worked example 3Query-time operation count for 1,000 candidates

Setup (illustrative numbers). A three-token query must be scored against 1,000 candidate passages. TILDE-QL has the passage weights stored in the index. A dense retriever scores the same candidates exactly with 768-dimensional vectors.

1
TILDE-QL. Tokenise the query (no BERT). For each candidate, look up and add 3 stored values: 1,000×3=3,000 1{,}000 \times 3 = 3{,}000 look-ups and additions.
2
Dense scoring. Run BERT once on the query, then compute one 768-dimensional dot product per candidate: 1,000×768=768,000 1{,}000 \times 768 = 768{,}000 multiplications.
3
Compare. 768,0003,000=256 \frac{768{,}000}{3{,}000} = 256 The dense scorer processes 256 times more values, and it also needs a BERT query encoding that TILDE-QL skips.
Result TILDE-QL: 3,000 cheap operations and no neural inference. Dense: 768,000 multiplications plus one BERT inference. (This counts exact candidate scoring; an ANN index changes the dense cost profile.)
TILDE-QL 3,000 look-ups and additions Dense scoring 768,000 multiplications 1,000 10,000 100,000 1,000,000 values processed per query for 1,000 candidates (log scale)
Worked example 3 on a log scale: the dense scorer processes 256 times more values than TILDE-QL.
Watch out TILDE-QL has a sparse query and a dense document vector. Each candidate can have a weight for every query token, including tokens absent from its text. For 1,000 candidates and 3 distinct query tokens, scoring reads 3,000 stored weights. Scanning all 30,522 dimensions would read 30,522,000 weights. The saving is in dimensions per candidate. A first-stage retriever can supply the candidate set. Document precomputation also helps dense bi-encoders; TILDE-QL additionally removes online query inference.
Query: 3 active token IDs For each of 1,000 candidates:read only these 3 stored weights 3,000 weight lookups, then rank
Query sparsity reduces work within each candidate. Candidate selection is a separate step.

5.3 The training loss: BiQDL

Another key difference from other bi-encoders is the loss. In place of the margin or contrastive loss of Week 5, TILDE is trained with the traditional query likelihood paradigm from statistical language models, assuming query tokens are independent of each other (the same assumption query likelihood models make). The bi-directional query-document likelihood loss (BiQDL) has two directions. Training uses relevant ⟨query, document⟩ pairs.

Direction 1: query likelihood (QL). The model reads the document text and must predict which vocabulary tokens appear in the query:

LQL(D)=−∑(q,d)∈D1∣V∣∑i∣V∣[ylog⁡(Pθ(ti∣d))+(1−y)log⁡(1−Pθ(ti∣d))] \mathcal{L}_{QL}(D) = - \sum_{(q,d) \in D} \frac{1}{|V|} \sum_{i}^{|V|} \Big[ y \log\big(P_\theta(t_i \mid d)\big) + (1-y)\log\big(1 - P_\theta(t_i \mid d)\big) \Big] y={1,if vocabulary term ti appears in query q,0,otherwise. y = \begin{cases} 1, & \text{if vocabulary term } t_i \text{ appears in query } q, \\ 0, & \text{otherwise.} \end{cases}
Where
LQL(D)\mathcal{L}_{QL}(D)
query-likelihood loss over the training set DD
DD
training set of relevant ⟨query, document⟩ pairs
(q,d)(q,d)
one relevant query–document pair
∣V∣|V|
vocabulary size; 1∣V∣\frac{1}{|V|} averages over all vocabulary tokens
tit_i
the ii-th vocabulary token
yy
label: 1 if tit_i is in the query, 0 otherwise (yiy_i on the lecture slides)
Pθ(ti∣d)P_\theta(t_i \mid d)
the model's probability (importance) of token tit_i given the document: entry ii of the document vector
θ\theta
the model parameters (BERT and the linear layer) being trained
Read it aloud The query-likelihood loss over D is minus the sum over every query–document pair in D of one over the vocabulary size times the sum over every vocabulary token t-i of: y times the log of the probability of t-i given d, plus one minus y times the log of one minus that probability. Here y is 1 if t-i appears in the query and 0 otherwise.
Formula in plain words For each training pair, take the average over all ∣V∣ |V| vocabulary tokens of a binary cross-entropy. If the token occurs in the query (y=1 y = 1 ), only the log⁡Pθ(ti∣d) \log P_\theta(t_i \mid d) part is active: a higher probability makes that log closer to zero and lowers the loss. If the token is absent from the query (y=0 y = 0 ), only the log⁡(1−Pθ) \log(1 - P_\theta) part is active: a higher probability now raises the loss. The bracket is a quantity we want to maximise; the leading minus sign turns it into a loss to minimise. In short: push up the probability of query tokens, push down the probability of every other token. The lecturer's advice: understand why each part is there; the exam is more likely to ask you to explain the function than to write it down.
y (in query?) P(t | d) 0 1 0 0 … 0 1 0 0.11 0.33 0.05 0.15 … 0.03 0.01 0.22 lower ↓ raise ↑ lower ↓ lower ↓ lower ↓ raise ↑ lower ↓ y = 1: a higher P(t | d) lowers the loss y = 0: a higher P(t | d) raises the loss
The QL direction of BiQDL (lecture slide 10): training raises the document's weights at the query's tokens and lowers them everywhere else.

Direction 2: document likelihood (DL). Simply swap the roles: the query is encoded by BERT into a dense representation, and the document becomes the sparse tokenised side. The model learns, from the query text, which vocabulary terms are important for the relevant document:

LDL(D)=−∑(q,d)∈D1∣V∣∑i∣V∣[ylog⁡(Pθ(ti∣q))+(1−y)log⁡(1−Pθ(ti∣q))] \mathcal{L}_{DL}(D) = - \sum_{(q,d) \in D} \frac{1}{|V|} \sum_{i}^{|V|} \Big[ y \log\big(P_\theta(t_i \mid q)\big) + (1-y)\log\big(1 - P_\theta(t_i \mid q)\big) \Big] y={1,if vocabulary term ti appears in document d,0,otherwise. y = \begin{cases} 1, & \text{if vocabulary term } t_i \text{ appears in document } d, \\ 0, & \text{otherwise.} \end{cases}
Where
LDL(D)\mathcal{L}_{DL}(D)
document-likelihood loss over the training set DD
Pθ(ti∣q)P_\theta(t_i \mid q)
the model's probability of token tit_i given the query (BERT now encodes the query)
yy
label: 1 if tit_i is in the document, 0 otherwise
D, (q,d), ∣V∣, ti, θD,\ (q,d),\ |V|,\ t_i,\ \theta
as in LQL\mathcal{L}_{QL}
Read it aloud The document-likelihood loss over D is minus the sum over every pair of one over the vocabulary size times the sum over every vocabulary token of: y times the log of the probability of t-i given q, plus one minus y times the log of one minus that probability, where y is 1 if t-i appears in the document.
Formula in plain words Identical structure, mirrored: given the query text, push up the probability of tokens that appear in the document and push down the probability of tokens that do not. If you understand the QL direction, you understand this one. It trains the query side to predict document terms, which enables the TILDE-DL ranking mode below.

Combined. The bi-directional loss is the average of both directions:

LBiQDL(D)=LQL(D)+LDL(D)2 \mathcal{L}_{BiQDL}(D) = \frac{\mathcal{L}_{QL}(D) + \mathcal{L}_{DL}(D)}{2}
Where
LBiQDL(D)\mathcal{L}_{BiQDL}(D)
the bi-directional query-document likelihood loss used to train TILDE
LQL(D), LDL(D)\mathcal{L}_{QL}(D),\ \mathcal{L}_{DL}(D)
the query-likelihood and document-likelihood losses above
Read it aloud The BiQDL loss over D equals the query-likelihood loss plus the document-likelihood loss, all divided by two.
Formula in plain words The final training loss is an equal-weight average of the two directions. Lowering either direction's loss lowers the total, so training improves both abilities at once. After training with BiQDL, TILDE can estimate the query likelihood given a document text and the document likelihood given a query text.

Worked example 4Computing the BiQDL loss on a tiny vocabulary

Setup (illustrative numbers). One training pair: query q= q = "apple account", relevant document d= d = "apple store account help". Toy vocabulary V={apple,store,account,banana} V = \{\text{apple}, \text{store}, \text{account}, \text{banana}\} , so ∣V∣=4 |V| = 4 . The current model, reading d d , predicts these probabilities of each token appearing in the query: Pθ(apple∣d)=0.7 P_\theta(\text{apple} \mid d) = 0.7 , Pθ(store∣d)=0.5 P_\theta(\text{store} \mid d) = 0.5 , Pθ(account∣d)=0.3 P_\theta(\text{account} \mid d) = 0.3 , Pθ(banana∣d)=0.2 P_\theta(\text{banana} \mid d) = 0.2 . Reading q q , it predicts these probabilities of each token appearing in the document: Pθ(apple∣q)=0.8 P_\theta(\text{apple} \mid q) = 0.8 , Pθ(store∣q)=0.4 P_\theta(\text{store} \mid q) = 0.4 , Pθ(account∣q)=0.6 P_\theta(\text{account} \mid q) = 0.6 , Pθ(banana∣q)=0.1 P_\theta(\text{banana} \mid q) = 0.1 . (All logs are natural logs.)

1
Build the label vector for the QL direction. yi=1 y_i = 1 if token ti t_i is in the query. Query tokens: {apple, account}. So y=[1,0,1,0] y = [1, 0, 1, 0] for (apple, store, account, banana).
2
Per-token QL terms. For tokens in the query use log⁡P \log P ; for tokens outside the query use log⁡(1−P) \log(1-P) : apple (y=1):log⁡0.7=−0.357store (y=0):log⁡(1−0.5)=log⁡0.5=−0.693account (y=1):log⁡0.3=−1.204banana (y=0):log⁡(1−0.2)=log⁡0.8=−0.223 \begin{aligned} \text{apple } (y=1)\text{:}\quad \log 0.7 &= -0.357 \\ \text{store } (y=0)\text{:}\quad \log(1 - 0.5) &= \log 0.5 = -0.693 \\ \text{account } (y=1)\text{:}\quad \log 0.3 &= -1.204 \\ \text{banana } (y=0)\text{:}\quad \log(1 - 0.2) &= \log 0.8 = -0.223 \end{aligned}
3
Average and negate. sum=−0.357−0.693−1.204−0.223=−2.477average=sum∣V∣=sum4=−0.619LQL=−(average)=0.619 \begin{aligned} \text{sum} &= -0.357 - 0.693 - 1.204 - 0.223 = -2.477 \\ \text{average} &= \frac{\text{sum}}{|V|} = \frac{\text{sum}}{4} = -0.619 \\ \mathcal{L}_{QL} &= -(\text{average}) = \mathbf{0.619} \end{aligned} The biggest contributor is "account": the model gave a true query token only probability 0.3, and gave the non-query token "store" a too-high 0.5. Training will push P(account∣d) P(\text{account} \mid d) up and P(store∣d) P(\text{store} \mid d) down.
4
Label vector for the DL direction. Now yi=1 y_i = 1 if ti t_i is in the document "apple store account help". Document tokens within our toy V V : {apple, store, account}, so y=[1,1,1,0] y = [1, 1, 1, 0] .
5
Per-token DL terms. apple:log⁡0.8=−0.223store:log⁡0.4=−0.916account:log⁡0.6=−0.511banana:log⁡(1−0.1)=−0.105 \begin{aligned} \text{apple:}\quad \log 0.8 &= -0.223 \\ \text{store:}\quad \log 0.4 &= -0.916 \\ \text{account:}\quad \log 0.6 &= -0.511 \\ \text{banana:}\quad \log(1 - 0.1) &= -0.105 \end{aligned}
6
Average and negate. sum=−1.755average=sum4=−0.439LDL=−(average)=0.439 \begin{aligned} \text{sum} &= -1.755 \\ \text{average} &= \frac{\text{sum}}{4} = -0.439 \\ \mathcal{L}_{DL} &= -(\text{average}) = \mathbf{0.439} \end{aligned}
7
Combine. LBiQDL=0.619+0.4392=0.529 \mathcal{L}_{BiQDL} = \frac{0.619 + 0.439}{2} = \mathbf{0.529} .
Result The BiQDL loss for this pair is 0.529. Minimising it simultaneously teaches the model to predict query tokens from documents (QL direction) and document tokens from queries (DL direction), which is precisely what lets a trained TILDE rank with query likelihood, document likelihood, or both.

5.4 Three ways of ranking with TILDE

These two abilities give three ranking settings that trade efficiency for effectiveness. The in-class lecture annotated each one (lecture slide 13).

TILDE-QL(q∣dk)=∑i∣q∣log⁡(Pθ(qi∣dk)) \text{TILDE-QL}(q \mid d^k) = \sum_i^{|q|} \log\big(P_\theta(q_i \mid d^k)\big)
Where
dkd^k
the kk-th candidate document being ranked
∣q∣|q|
number of tokens in the query
qiq_i
the ii-th query token
Pθ(qi∣dk)P_\theta(q_i \mid d^k)
stored probability of query token qiq_i given the document (from the offline document vector)
Read it aloud TILDE-QL of q given d-k is the sum, over each token q-i of the query, of the log of the probability of q-i given d-k.
Formula in plain words Rank each document by how likely it makes the query's tokens. Because document representations are precomputed, this is just a few look-ups and additions: in effect a dot product, cheap and effective, with no GPU at query time. This is TILDE's default setting. Logs of probabilities are negative, so scores closer to zero are better; one query token with a tiny probability drags the whole score down, because its log is a large negative number.
TILDE-DL(dk∣q)=1∣dk∣∑i∣dk∣log⁡(Pθ(dik∣q)) \text{TILDE-DL}(d^k \mid q) = \frac{1}{|d^k|} \sum_i^{|d^k|} \log\big(P_\theta(d_i^k \mid q)\big)
Where
∣dk∣|d^k|
number of tokens in document dkd^k
dikd_i^k
the ii-th token of document dkd^k
Pθ(dik∣q)P_\theta(d_i^k \mid q)
probability of that document token given the query (BERT encodes the query at search time)
Read it aloud TILDE-DL of d-k given q is one over the length of d-k times the sum, over each token of d-k, of the log of the probability of that token given q.
Formula in plain words Rank by the average log-probability of the document's tokens given the query. The division by document length ∣dk∣ |d^k| normalises the score: longer documents contain more tokens and would otherwise accumulate lower (more negative) likelihoods. This direction needs a BERT inference on the query at search time, so online inference adds latency; a GPU can accelerate it. The lecture calls it an ablation in the paper; TILDE itself does not need it.
TILDE-QDL(q,dk)=α⋅TILDE-QL(q∣dk)+(1−α)⋅TILDE-DL(dk∣q) \text{TILDE-QDL}(q, d^k) = \alpha \cdot \text{TILDE-QL}(q \mid d^k) + (1 - \alpha) \cdot \text{TILDE-DL}(d^k \mid q)
Where
α\alpha
interpolation weight in [0,1][0,1]: the share given to query likelihood
TILDE-QL, TILDE-DL\text{TILDE-QL},\ \text{TILDE-DL}
the two scores above
Read it aloud TILDE-QDL of q and d-k equals alpha times TILDE-QL of q given d-k, plus one minus alpha times TILDE-DL of d-k given q.
Formula in plain words The final score is a weighted mix of the two directions. α=1 \alpha = 1 gives pure query likelihood (fastest, tokeniser only); α=0 \alpha = 0 gives pure document likelihood; values in between blend the evidence. Any α<1 \alpha \lt 1 still needs the online query inference, so the mix gives a minor effectiveness gain at the cost of query latency.
ModeOnline workLecture commentWhen a system designer picks it
TILDE-QL (default)Tokenise the query; look up stored document weights"Just a dot product (cheap and effective)"Usual case: the corpus is a manageable size, so encode every document once offline and give users the fastest response
TILDE-DLBERT inference on the query at search timeAn ablation in the paper; slow at query timeThe document side is huge and you prefer to skip BERT document preprocessing, while queries are relatively few
TILDE-QDLBoth of the aboveA minor gain; still slow, because it involves online computationEffectiveness matters most and extra latency is acceptable (still faster than a dense model)

5.5 Impact of the document likelihood signal

Plotting nDCG@10 and MAP against α \alpha (on TREC DL 2019 and 2020) shows:

0.2 0.3 0.4 0.5 0.6 0 0.2 0.4 0.6 0.8 1.0 α (weight on query likelihood) nDCG@10 DL2020 DL2019 best: α ≈ 0.4–0.5 α = 0: document likelihood only α = 1: query likelihood only, no GPU TREC DL 2019 TREC DL 2020
nDCG@10 of TILDE-QDL as α varies (approximate values read from slide 26). The MAP plot on the same slide has the same shape.

Worked example 5TILDE-QDL interpolation with α

Setup (illustrative numbers). For a query q q and two candidate passages, a trained TILDE gives: TILDE-QL(q∣d1)=−1.9 \text{TILDE-QL}(q\mid d_1) = -1.9 , TILDE-DL(d1∣q)=−0.6 \text{TILDE-DL}(d_1\mid q) = -0.6 ; and TILDE-QL(q∣d2)=−1.6 \text{TILDE-QL}(q\mid d_2) = -1.6 , TILDE-DL(d2∣q)=−1.4 \text{TILDE-DL}(d_2\mid q) = -1.4 . All are log-likelihoods, so higher (closer to zero) is better. The DL scores are already length-normalised.

1
Pure query likelihood, α=1 \alpha = 1 . Scores are just QL: d1:−1.9 d_1 : -1.9 , d2:−1.6 d_2 : -1.6 → ranking d2≻d1 d_2 \succ d_1 . Fastest setting: no GPU needed.
2
Pure document likelihood, α=0 \alpha = 0 . Scores are just DL: d1:−0.6 d_1 : -0.6 , d2:−1.4 d_2 : -1.4 → ranking d1≻d2 d_1 \succ d_2 . The lecture showed this setting alone performs badly overall.
3
Interpolate at the sweet spot α=0.4 \alpha = 0.4 . d1:0.4×(−1.9)+0.6×(−0.6)=−0.76−0.36=−1.12d2:0.4×(−1.6)+0.6×(−1.4)=−0.64−0.84=−1.48 \begin{aligned} d_1\text{:}\quad 0.4 \times (-1.9) + 0.6 \times (-0.6) &= -0.76 - 0.36 = \mathbf{-1.12} \\ d_2\text{:}\quad 0.4 \times (-1.6) + 0.6 \times (-1.4) &= -0.64 - 0.84 = \mathbf{-1.48} \end{aligned}
4
Rank. −1.12>−1.48 -1.12 > -1.48 , so d1≻d2 d_1 \succ d_2 . The document-likelihood evidence (which strongly favoured d1 d_1 ) flipped the ranking relative to pure QL.
Result α controls a real trade-off: α=1 \alpha = 1 gives minimum latency (tokeniser only), while α≈0.4–0.5 \alpha \approx 0.4\text{–}0.5 gives the best effectiveness at the cost of one BERT inference on the query at search time (GPU needed). The empirical plots show the effectiveness gap between these two settings is small.
PracticeLecture Exercise 1: how does TILDE achieve high efficiency compared to dense retrievers?
The lecture expected three separate aspects.
Answer(1) At query time the query is only tokenised, so no BERT encoding runs online. (2) Documents do pass through BERT, but offline at indexing time, once; the index stays fixed while queries arrive. (3) The query vector is sparse (1 only at the query's own tokens), so scoring touches only those few dimensions of each document vector.
PracticeIn LQL \mathcal{L}_{QL} , what happens to the loss if training raises Pθ(ti∣d) P_\theta(t_i \mid d) for a token that is absent from the query? Why is there a minus sign in front of the sum?
AnswerFor that token y=0 y = 0 , so only log⁡(1−Pθ) \log(1 - P_\theta) is active. Raising Pθ P_\theta shrinks 1−Pθ 1 - P_\theta , makes the log more negative and so raises the loss. The bracketed sum is a log-likelihood we want to maximise; the minus sign turns it into a loss that training minimises.
PracticeYour search service has an enormous, frequently changing document collection and a small number of queries per day. Which TILDE ranking mode suits it, and what do you give up?
AnswerTILDE-DL: it avoids running BERT over every document offline, because the neural work happens on the (few) queries. You give up query latency (a BERT inference per query, GPU needed) and effectiveness, since DL alone ranks poorly (nDCG@10 ≈ 0.23 at α=0 \alpha = 0 ).

6. TILDEv2: contextualised exact matching + cheap expansion

Sources: slides 27–31 · Video 4 (TILDEv2)

6.1 What changes from TILDE

TILDEv2 keeps the BERT-tokeniser sparse query encoding and changes the document side in two ways:

s sum of matched token weights 00…1…100 Sparse query vector · 1 × |V| BERT tokeniser ⟨q⟩ “apple account” apple4.7 store0.2 account (added)3.1 Contextualised token weights BERT encoder (token outputs) ⟨d⟩ “apple store” + “account” expanded offline by doc2query or TILDE
TILDEv2: exact term matching between the sparse query vector and contextualised token weights of the expanded document. Here "account" exists in the document only because expansion added it.

Worked example 6Scoring a query with TILDEv2

Setup (illustrative numbers). Query: "apple account". Original passage d d : "apple store". At indexing time, TILDE expansion proposed the extra tokens {account, iphone} (likely query terms for this passage); after filtering (see 6.2), both were appended. BERT then encoded the expanded passage and produced one learned weight per token: apple → 4.7, store → 0.2, account → 3.1, iphone → 2.4.

1
Tokenise the query. Active terms: {apple, account}. This is the sparse query vector; only these terms will be scored.
2
Exact-match lookup. For each query term, find matching tokens in the expanded document and take their contextualised weights: "apple" matches with weight 4.7; "account" matches with weight 3.1, only because expansion added it. (If a term appeared several times, we would take its stored match weight from the index.)
3
Sum the matched weights. s(q,d)=4.7+3.1=7.8 s(q,d) = 4.7 + 3.1 = \mathbf{7.8} .
4
Contrast with the unexpanded passage. Without expansion, "account" matches nothing and s=4.7 s = 4.7 : the passage would rank as if it had nothing to say about accounts. Contrast also with plain BM25: BM25 would give "apple" a weight from counts and rarity alone, while TILDEv2's 4.7 is contextualised. The same word "apple" in a fruit passage would get a different weight than in this tech-support passage.
Result TILDEv2 = exact term matching (fast, inverted-index friendly) + learned contextualised term weights (semantics) + offline expansion (fixes vocabulary mismatch). Scoring cost is again a handful of additions per document.

6.2 How important is the expansion component?

Empirical results (MRR@10, MS MARCO dev) as the expansion method varies. When TILDE generates m m candidate terms, only terms that are not stop words and not already in the document are actually added (the same filtering is applied to doc2query). More added terms also means a larger index:

MeasureNo expansiondoc2queryTILDE m=128TILDE m=150TILDE m=200
MRR@100.2990.3330.3260.3270.330
Avg added tokens–19.013.025.261.6
Index size4.3 GB5.2 GB5.2 GB5.6 GB6.9 GB
Expansion cost–320 hours, $768 (TPU)7.22 hours, $5.34 (GPU)7.25 hours, $5.377.33 hours, $5.42

Reading the table: expansion always beats no expansion (0.299 → 0.32+). doc2query gives the best effectiveness (0.333); TILDE needs many more added terms to get close (m = 200 adds 61.6 tokens on average against doc2query's 19), which also grows the index (6.9 GB against 5.2 GB). This suggests TILDE's individual generated terms are somewhat weaker than doc2query's. The cost column tells the rest of the story:

Collectiondoc2query (TPU)TILDE expansion (GPU)
MS MARCO320 hours (about 13 days), $768about 7¼ hours, about $5
MS MARCO v2 (15× larger)about 200 days, more than $11,000about 4.5 days, about $80
Key idea Offline does not mean free. doc2query ran on Google TPUs (about 15× more powerful than GPUs, and expensive); TILDE expansion runs on cheap GPUs. The gap keeps growing at the scale of commercial search engines (Google, Bing, Baidu), where waiting 13 days, or 200, for an index is infeasible.
PracticeWhy does TILDEv2 need document expansion, while the original TILDE works without it?
AnswerTILDE's document vector is dense over the whole vocabulary, so every query token receives some learned weight (implicit expansion). TILDEv2 scores only exact matches between query tokens and the document's own tokens, so a query term absent from the document contributes nothing. Expansion puts likely query terms into the document before encoding, which restores matching for those terms.
Practicedoc2query gives the best MRR@10 in the TILDEv2 expansion table. Give two reasons to choose TILDE expansion anyway.
Answer(1) Cost: about 7 hours and $5 on GPUs against 320 hours and $768 on TPUs for MS MARCO, a gap that grows to about 4.5 days against 200 days on MS MARCO v2. (2) The effectiveness gap is small (0.330 with m = 200 against 0.333), so the saving costs very little quality. The price is a larger index when m is large.

7. SPLADE: expansion and weighting from the MLM head

Sources: slide 10, slides 32–39 · lecture slides 15–16, 18, 22 · Video 5 (SPLADE and SPLADEv2) · Lecture transcript

7.1 Intuition

SPLADE (Formal, Piwowarski & Clinchant, SIGIR 2021) takes the second route: keep the transformer encoder for both queries and documents, and use it to predict term-importance weights over the entire vocabulary. For the query "Who wrote the book of the lord of the rings", a traditional lexical model can only match those exact words. SPLADE activates related vocabulary terms with learned importance weights, for example:

Activated termwritecreatedauthorednovelLOTRTolkien
Contribution3.51.23.33.354

These terms can come from outside the query (Tolkien gets weight 4 purely from semantic association), and yet the vector is still sparse: nearly all of the 30,522 vocabulary dimensions stay at zero. If we can predict the importance of highly related vocabulary terms for their contribution to ranking, we get a much better lexical term-based retriever. The transformer supplies these weights; collection statistics play no part.

7.2 The mechanism: reuse BERT's MLM head

BERT pre-training uses a masked language modelling (MLM) head to predict vocabulary tokens. SPLADE reuses this head at every input position. It obtains vocabulary logits, converts them to non-negative term activations, then pools each vocabulary column into one weight. The lecture uses “probability” as a simple explanation of term importance. Here pijp_{ij} denotes the supplied non-negative activation at input position ii for vocabulary term jj.

wj=∑i=1npijj=1,…,∣V∣ w_j = \sum_{i=1}^{n} p_{ij} \qquad j = 1, \dots, |V|
Where
wjw_j
importance weight of vocabulary term jj for this query or document
nn
number of token positions in the input text
ii
index of one input token position
pijp_{ij}
weight that the MLM head at position ii assigns to vocabulary term jj
j, ∣V∣j,\ |V|
index of a vocabulary term, and the vocabulary size (30,522)
Read it aloud The weight w-j equals the sum, over every input position i from 1 to n, of p-i-j, computed for every vocabulary term j from 1 to the vocabulary size.
Formula in plain words A vocabulary term becomes important when many positions of the text "point at it", even if the term itself never occurs in the text. Each position adds its weight for the term, so a term supported weakly by many positions or strongly by one position both end up with weight. Longer inputs have more positions and so tend to accumulate larger sums. The result is one vocabulary-sized vector per query or document.
one vocabulary vector per input position MLM head (from BERT pre-training) Transformer encoder (BERT) tok₁ “who” tok₂ “wrote” tok₃ “the” tokₙ “rings” pool sum v1 max v2 Importance vector one sparse |V| vector after ReLU + FLOPS/L1
SPLADE: each token position yields an MLM activation vector over the vocabulary; pooling across positions gives the term-importance vector.
Technical detail beyond the lecture shorthandThe original SPLADE paper, Section 3.2 defines the activation before pooling as follows.
pij=log⁡(1+max⁡(0,zij)) p_{ij}=\log(1+\max(0,z_{ij}))
Where
zijz_{ij}
raw MLM logit at input position ii for vocabulary term jj
pijp_{ij}
non-negative activation used in the pooling formulas
max⁡(0,zij)\max(0,z_{ij})
ReLU: keep a positive logit, otherwise return zero
log⁡\log
natural logarithm
Read it aloudp-i-j equals the natural log of one plus the maximum of zero and z-i-j.
Formula in plain wordsReLU removes non-positive logits. The logarithm reduces the effect of large positive logits. Zero remains zero. Pool these activations to obtain vocabulary weights; the weights need no normalisation to a sum of one.

7.3 How SPLADE keeps weights sparse

A softmax gives positive probability to every vocabulary term. SPLADE instead applies ReLU to raw logits, then log saturation, before pooling. ReLU makes non-positive logits exactly zero. Sparsity regularisation encourages the model to activate fewer terms; the ranking loss keeps useful matching terms. FLOPS regularisation discourages frequently active dimensions that make posting lists costly. L1 penalises total activation magnitude. The lecture mentions both approaches; their detailed equations are optional reading.

Exam angleExplain the roles separately: the ranking objective learns relevance, ReLU creates zeros, and regularisation encourages efficient sparse representations. Sum or max pooling combines positions. Max pooling alone does not create zeros when every input value is positive.

7.4 SPLADE core ideas, summarised

Worked example 7SPLADE pooling (sum vs max) and scoring

Setup (illustrative numbers). Document: "apple reduces sugar" (3 token positions). Toy vocabulary of 6 terms. Use the following illustrative non-negative activations (these example rows happen to sum to 1):

Positionapplefruitreduceslowerssugarcar
pos 1 · "apple"0.500.300.020.020.100.06
pos 2 · "reduces"0.050.050.450.300.100.05
pos 3 · "sugar"0.100.150.050.050.600.05
1
Sum pooling (SPLADEv1). Add each column: apple=0.50+0.05+0.10=0.65fruit=0.30+0.05+0.15=0.50reduces=0.02+0.45+0.05=0.52lowers=0.02+0.30+0.05=0.37sugar=0.10+0.10+0.60=0.80car=0.06+0.05+0.05=0.16 \begin{aligned} \text{apple} &= 0.50 + 0.05 + 0.10 = 0.65 \\ \text{fruit} &= 0.30 + 0.05 + 0.15 = 0.50 \\ \text{reduces} &= 0.02 + 0.45 + 0.05 = 0.52 \\ \text{lowers} &= 0.02 + 0.30 + 0.05 = 0.37 \\ \text{sugar} &= 0.10 + 0.10 + 0.60 = 0.80 \\ \text{car} &= 0.06 + 0.05 + 0.05 = 0.16 \end{aligned} Document vector (sum): [0.65,0.50,0.52,0.37,0.80,0.16] [0.65, 0.50, 0.52, 0.37, 0.80, 0.16] . fruit (0.50) and lowers (0.37) got real weight although neither word is in the document: implicit expansion.
2
Max pooling (SPLADEv2). Take each column's largest value across positions: apple=max⁡(0.50,0.05,0.10)=0.50fruit=0.30reduces=0.45lowers=0.30sugar=0.60car=0.06 \begin{aligned} \text{apple} &= \max(0.50, 0.05, 0.10) = 0.50 \\ \text{fruit} &= 0.30 \\ \text{reduces} &= 0.45 \\ \text{lowers} &= 0.30 \\ \text{sugar} &= 0.60 \\ \text{car} &= 0.06 \end{aligned} Document vector (max): [0.50,0.30,0.45,0.30,0.60,0.06] [0.50, 0.30, 0.45, 0.30, 0.60, 0.06] . Max pooling keeps the strongest single signal per term and ignores the accumulation of many weak ones: "car", which no position really supports, drops from 0.16 to 0.06.
3
Illustrate optional pruning. For this arithmetic example only, set pooled weights below 0.2 to zero. This threshold is an extra illustration; SPLADE learns sparsity through its activation and regularisation. The sum-pooled vector → [0.65,0.50,0.52,0.37,0.80,0] [0.65, 0.50, 0.52, 0.37, 0.80, 0] . "car" is now inactive, so this document never enters the posting list for "car".
4
Encode a query the same way. Query "lowers sugar" passes through the same SPLADE encoder; suppose its sparse vector comes out as [0,0,0.3,0.9,1.1,0] [0, 0, 0.3, 0.9, 1.1, 0] (activating reduces by expansion, plus lowers and sugar).
5
Score with a dot product over shared non-zero terms. Using the sparsified sum-pooled document vector: s(q,d)=0.3×0.52+0.9×0.37+1.1×0.80=0.156+0.333+0.880=1.369 s(q,d) = 0.3 \times 0.52 + 0.9 \times 0.37 + 1.1 \times 0.80 = 0.156 + 0.333 + 0.880 = \mathbf{1.369} . Only 3 of 6 dimensions were touched; with a real vocabulary, only the few dozen active terms matter, which is what makes SPLADE scoring inverted-index fast.
Result The query word "lowers" is absent from the document and the document word "reduces" is absent from the query, yet the two matched through their expanded, weighted representations: semantic matching carried out entirely inside a sparse, lexical-style dot product.

7.5 TILDE versus SPLADE

The in-class lecture compared the two families side by side (lecture slide 18), and recommended the comparison figure for revision:

TILDE SPLADE “deep learning” Document text “deep learning” Document text BERT tokeniser BERT BERT BERT no encoder [CLS] vector (768-d) token vectors h₁…hₙ token vectors h₁…hₘ Linear layer MLM head + pooling MLM head + pooling 0 1 0 … 1 0.12 1.35 0.00 … 0.67 0.18 1.08 0.00 … 0.74 0.09 1.21 0.00 … 0.63 binary query vector learned doc vector learned query vector learned doc vector Score = sum of doc weights at query terms no query encoding: fastest Score = dot product of two sparse vectors the query runs through BERT at search time
TILDE against SPLADE for the query "deep learning" (redrawn from lecture slide 18): TILDE weights come from [CLS] + linear layer, SPLADE weights from every token's MLM activation vector, pooled.
PracticeLecture Exercise 1.1: how does SPLADE get its term weights?
Say which representation it takes from BERT, what head it uses, and how it combines them.
AnswerSPLADE runs the query or document through BERT and keeps every token's output vector. Each vector goes through the pre-trained MLM head, which gives logits over the whole vocabulary for that position. ReLU and log saturation turn these into an activation vector. SPLADE then pools these activation vectors into one vocabulary-sized vector: sum pooling in SPLADEv1, max pooling in SPLADEv2. Each dimension's value is that term's importance. ReLU and the FLOPS/L1 sparsity losses applied during training make most of the weights exactly zero.
PracticeWhy is TILDE usually faster than SPLADE at query time, even though both score with a sparse dot product?
AnswerTILDE's query side is only the BERT tokeniser, a lookup table with no neural inference. SPLADE must run the query through the full BERT encoder and MLM head before it can score anything, so every query pays one transformer inference.

8. SPLADEv2: what changed

Sources: slide 40 · lecture slides 17, 24–25 · Video 5 (SPLADE and SPLADEv2) · Lecture transcript

SPLADEv2 (same research group) made three main improvements:

AspectSPLADEv1SPLADEv2
PoolingSum pooling to aggregate token activation vectorsMax pooling: for each vocabulary dimension, take the highest weight across all input token positions; empirically this captures the most important signal better
Training lossBasic contrastive trainingAdds a distillation loss: the model also learns from a teacher model (beyond the relevance labels), improving weighting and expansion
Weighting & expansionBoth query and document are weighted and expanded, so full SPLADE inference runs at query timeNew variant with document-only weighting and expansion (similar in spirit to TILDE), so queries stay simple and close to their original form

Max pooling replaces the sum in the SPLADE weight formula:

wj=max⁡i=1,…,n  pijj=1,…,∣V∣ w_j = \max_{i = 1, \dots, n} \; p_{ij} \qquad j = 1, \dots, |V|
Where
wjw_j
importance weight of vocabulary term jj
max⁡i\max_{i}
the largest value over all input positions i=1,…,ni = 1, \dots, n
pijp_{ij}
MLM weight of term jj at position ii
n, ∣V∣n,\ |V|
number of input positions, and the vocabulary size
Read it aloud The weight w-j equals the maximum, over all input positions i from 1 to n, of p-i-j, for every vocabulary term j.
Formula in plain words For each vocabulary term, keep only the strongest vote any single position gave it. A term backed strongly by one position keeps that full weight; a term backed weakly by many positions stays small, because weak votes no longer add up. Weights can exceed 1, and adding more text can only raise a weight when a new position votes more strongly than every earlier one.
Sum pooling (SPLADEv1) Max pooling (SPLADEv2) 1.0 0.8 0.5 0.5 0.9 0.5 0.5 0.3 deeplearningneuralmachine deeplearningneuralmachine from position “deep” from position “learning”
Worked example 8 for four of the ten terms: sum pooling stacks both positions' weights; max pooling keeps the larger one.
Reading slides 24–25The two supplied rows sum to 2.2 and 1.9. Treat them as term weights for this pooling exercise. Keep their numbers unchanged. Slide 25 repeats “SPLADEv1” in its opening line, but its title and question ask for SPLADEv2, so use max pooling. Pool each vocabulary column across input positions.

Worked example 8Lecture Exercises 2 and 2.1: SPLADEv1 and SPLADEv2 pooling for “deep learning”

Setup (lecture slides 24–25; a deliberately tiny vocabulary). The whole vocabulary has 10 terms: [deep, apple, learning, neural, networks, fruit, like, machine, map, shallow]. The query is "deep learning". The MLM head predicts one activation vector per query position:

Positiondeepapplelearningneuralnetworksfruitlikemachinemapshallow
“deep”0.900.30.50.2000.200.1
“learning”0.10.40.500.200.40.300
1
SPLADEv1: sum pooling. Add the two rows term by term: deep=0.9+0.1=1.0apple=0+0.4=0.4learning=0.3+0.5=0.8neural=0.5+0=0.5networks=0.2+0.2=0.4fruit=0+0=0like=0+0.4=0.4machine=0.2+0.3=0.5map=0+0=0shallow=0.1+0=0.1 \begin{aligned} \text{deep} &= 0.9 + 0.1 = 1.0 \\ \text{apple} &= 0 + 0.4 = 0.4 \\ \text{learning} &= 0.3 + 0.5 = 0.8 \\ \text{neural} &= 0.5 + 0 = 0.5 \\ \text{networks} &= 0.2 + 0.2 = 0.4 \\ \text{fruit} &= 0 + 0 = 0 \\ \text{like} &= 0 + 0.4 = 0.4 \\ \text{machine} &= 0.2 + 0.3 = 0.5 \\ \text{map} &= 0 + 0 = 0 \\ \text{shallow} &= 0.1 + 0 = 0.1 \end{aligned}
wv1=[1.0, 0.4, 0.8, 0.5, 0.4, 0, 0.4, 0.5, 0, 0.1] \mathbf{w}_{v1} = [1.0,\ 0.4,\ 0.8,\ 0.5,\ 0.4,\ 0,\ 0.4,\ 0.5,\ 0,\ 0.1]
2
SPLADEv2: max pooling. Take the larger value in each column: deep=max⁡(0.9,0.1)=0.9apple=max⁡(0,0.4)=0.4learning=max⁡(0.3,0.5)=0.5neural=max⁡(0.5,0)=0.5networks=0.2fruit=0like=0.4machine=max⁡(0.2,0.3)=0.3map=0shallow=0.1 \begin{aligned} \text{deep} &= \max(0.9, 0.1) = 0.9 \\ \text{apple} &= \max(0, 0.4) = 0.4 \\ \text{learning} &= \max(0.3, 0.5) = 0.5 \\ \text{neural} &= \max(0.5, 0) = 0.5 \\ \text{networks} &= 0.2 \\ \text{fruit} &= 0 \\ \text{like} &= 0.4 \\ \text{machine} &= \max(0.2, 0.3) = 0.3 \\ \text{map} &= 0 \\ \text{shallow} &= 0.1 \end{aligned}
wv2=[0.9, 0.4, 0.5, 0.5, 0.2, 0, 0.4, 0.3, 0, 0.1] \mathbf{w}_{v2} = [0.9,\ 0.4,\ 0.5,\ 0.5,\ 0.2,\ 0,\ 0.4,\ 0.3,\ 0,\ 0.1]
3
Read the vectors. Both keep neural, networks and machine active although the query never says them: implicit expansion. Terms supported by both positions (learning, networks, machine) gain the most under sum pooling. fruit and map stay at 0 in both, so the query never touches their posting lists.
Result SPLADEv1 (sum): [1.0, 0.4, 0.8, 0.5, 0.4, 0, 0.4, 0.5, 0, 0.1]. SPLADEv2 (max): [0.9, 0.4, 0.5, 0.5, 0.2, 0, 0.4, 0.3, 0, 0.1]. The lecturer's summary: many vectors come out of the MLM head, and you aggregate them; aggregate with sum and it is v1, aggregate with max and it is v2.

The document-only variant matters because the asymmetric argument from TILDE applies here too. Document expansion happens offline at indexing time, with plenty of time to process thoroughly; query processing happens online while a user waits, so keeping the query side simple means much faster search response times. If most of the semantic-matching benefit comes from the document side anyway, dropping query-side expansion buys efficiency at little cost.

PracticeTwo positions give MLM activation vectors over four terms: p1=[0.6,0.2,0.0,0.4] p_1 = [0.6, 0.2, 0.0, 0.4] and p2=[0.1,0.5,0.3,0.4] p_2 = [0.1, 0.5, 0.3, 0.4] . Compute the SPLADEv1 and SPLADEv2 vectors.
AnswerSum (v1): [0.7,0.7,0.3,0.8] [0.7, 0.7, 0.3, 0.8] . Max (v2): [0.6,0.5,0.3,0.4] [0.6, 0.5, 0.3, 0.4] . The fourth term, supported equally by both positions, is the largest weight under sum pooling and only mid-sized under max pooling.
PracticeWhy does SPLADEv2's document-only variant search faster than SPLADEv1?
AnswerSPLADEv1 weights and expands the query with a full SPLADE inference at query time. The document-only variant does all weighting and expansion offline on documents, so the online query stays close to its original tokens: no expensive query inference, and fewer active query terms to score.

9. Effectiveness comparison of all retrievers so far

Sources: slides 41–46 · Video 6 (Performance of retrievers)

All numbers below are on the MS MARCO dev set with MRR@10 (mean reciprocal rank over the top 10, from Week 2). MRR@10 is standard for MS MARCO dev because its relevance judgements are sparse, usually one judged relevant passage per query. Values are the approximate figures narrated in Video 6, with the chart bars used where the narration gives no number.

BM25≈ 0.18 RepBERT≈ 0.33 ANCE≈ 0.33 ColBERT≈ 0.39 RepLLaMA≈ 0.41 LLM2Vec≈ 0.42 BM25 + TILDE≈ 0.27 BM25 + TILDEv2≈ 0.33 SPLADEv1≈ 0.32 SPLADEv2≈ 0.34 0 0.125 0.25 0.375 0.5 MRR@10 on MS MARCO dev Term-based lexical Encoder dense Decoder dense (LLM) Learned sparse
MRR@10 on MS MARCO dev (approximate). Grey: term-based lexical. Blue: encoder-based dense (single- and multi-vector). Orange: decoder-based dense. Green: learned sparse.
Watch outThe bar chart on slides 41–46 draws RepBERT near 0.30 and ColBERT near 0.36, while Video 6 narrates about 0.33 and 0.39. The ordering is the same either way; quote approximate values and focus on the ranking of the families.

Walking through the chart the way the lecture does:

Takeaways Learned sparse models fall a little short of the very best dense retrievers on effectiveness, and they are generally much more efficient: like traditional lexical models they only score documents that share (expanded) terms with the query, giving fast search and low computational requirements. They occupy a sweet spot: significantly better effectiveness than BM25 while keeping much of the efficiency that makes lexical models practical at large scale.
PracticeLLM2Vec scores about 0.42 MRR@10 and SPLADEv2 about 0.34. Give two reasons a production search team might still choose SPLADEv2.
Answer(1) Efficiency: SPLADEv2 scores through an inverted index and only touches documents sharing active terms, while LLM2Vec compares high-dimensional dense vectors. (2) Model size: SPLADE uses a BERT-sized encoder (about 110M parameters) against a 7B-parameter decoder, so encoding the collection and every query costs far less GPU time and memory. With the document-only variant, query cost drops further.

10. PromptReps: training-free dense + sparse representations from LLMs

Sources: slides 47–51 · lecture slides 19–20, 23 · Video 7 (PromptReps) · Lecture transcript

10.1 Why another approach?

The LLM-based dense retrievers from Week 5 all need training to become retrievers:

Contrastive fine-tuning is expensive twice over: you need labels (or synthetic data), and you need serious GPU infrastructure; the larger the decoder backbone, the longer the training and the more memory required. The question PromptReps (Zhuang et al., EMNLP 2024) asks: can we engineer prompts so the LLM produces an effective retrieval representation with no contrastive fine-tuning at all? This is "zero-shot" generation of representations: no training beyond the LLM's own pre-training, and no in-context examples either.

10.2 How it works

Prompt the LLM to represent the given text with a single word (queries use a slightly different prompt, "represent the query"):

<System> You are an AI assistant that can understand human language.
<User> Passage: "[text]". Use one word to represent the passage in a retrieval task. Make sure your word is in lowercase.
<Assistant> The word is: "

Run one inference and harvest two representations from the position of that final quote character, the position where the model is about to generate its one summarising word:

  1. Dense: the last hidden state at that position (it encodes the word the model is about to say), used for dense retrieval via an ANN index. The lecture noted this is the same kind of embedding RepLLaMA uses.
  2. Sparse: continue through the LM head to get the next-token logits, raw prediction scores over the LLM's entire vocabulary, and keep the top-k predicted terms as a sparse vector, used for classic inverted-index matching.
one inference per text Prompted LLM (decoder) “Use one word to represent the passage … The word is: “ Last hidden state at the final quote token LM head next-token logits Sparse vector top-k logits → inverted index Dense vector → ANN index (as in RepLLaMA) Hybrid score linear interpolation
One inference, two representations: the hidden state gives a dense vector; the next-token logits give a sparse vector over the LLM vocabulary. Their scores are combined linearly.

At retrieval time, the dense representation performs dense retrieval and the sparse representation performs typical inverted-index matching; each gives a score, and the final relevance score combines them linearly (an interpolation). The result is a hybrid retriever built from one encoder. The lecture stressed that PromptReps can run as a pure sparse model, a pure dense model, or the hybrid; it works zero-shot, and it becomes much better still if you do train it with a standard contrastive loss. A later model in Week 9, built on a text-diffusion model, follows the same idea with multi-vector representations.

The two scores live on different scales (cosine similarity lies in [−1, 1]; a sparse dot product is unbounded), so a common recipe first min–max normalises each score list over the candidates and then interpolates:

s~=s−smin⁡smax⁡−smin⁡,shybrid=λ s~dense+(1−λ) s~sparse \tilde{s} = \frac{s - s_{\min}}{s_{\max} - s_{\min}}, \qquad s_{hybrid} = \lambda\, \tilde{s}_{dense} + (1 - \lambda)\, \tilde{s}_{sparse}
Where
ss
a raw score (dense or sparse) of one candidate
smin⁡, smax⁡s_{\min},\ s_{\max}
the smallest and largest raw score of that kind in the candidate list
s~\tilde{s}
the normalised score, between 0 and 1
s~dense, s~sparse\tilde{s}_{dense},\ \tilde{s}_{sparse}
normalised dense (hidden-state) and sparse (logits) scores
λ\lambda
interpolation weight in [0,1][0,1] given to the dense score
shybrids_{hybrid}
final hybrid relevance score
Read it aloud s-tilde equals s minus s-min, divided by s-max minus s-min. The hybrid score equals lambda times the normalised dense score plus one minus lambda times the normalised sparse score.
Formula in plain words First rescale each score list so its smallest value becomes 0 and its largest becomes 1 (subtract the minimum, then divide by the range); this puts the two signals on the same footing. Then take a weighted sum: a fraction λ \lambda of the dense score plus the remaining fraction of the sparse score. λ=1 \lambda = 1 is pure dense retrieval, λ=0 \lambda = 0 pure sparse; values in between reward documents that do well on both signals. The slides specify only "combine linearly"; min–max normalisation is an illustrative choice. If all scores in a list are equal, set their normalised contribution to zero to avoid dividing by zero.
Worked-example conventionThe lecture specifies linear combination. The min–max scaling here is an illustrative choice. When a score list has equal minimum and maximum, give that list a zero normalised contribution to avoid division by zero. Check effectiveness with relevance judgements after choosing a fusion rule.

10.3 Does zero-shot retrieval work?

On MS MARCO (MRR@10, shown ×100 on the slide), with everything zero-shot (no in-context examples, no contrastive training of the representations):

SystemMRR@10 (approx.)Notes
LLM2Vec (unsupervised, SimCSE-trained baseline)≈ 13.5still needs unsupervised contrastive training
PromptReps (Dense only)≈ 17.5below BM25
BM25 (dotted reference line)≈ 18.5also training-free
PromptReps (Sparse only)≈ 21above BM25
PromptReps (Hybrid)≈ 24.5combining dense + sparse is the key
Key idea Combining the dense and sparse representations allows very effective zero-shot retrieval. The hybrid clearly beats BM25 and the unsupervised LLM2Vec baseline, with no training of the representations at all.

Extension: more than one word. The prompt can ask for two words, three words, a sentence, a paragraph… which requires multiple autoregressive inferences and yields multiple token embeddings/distributions. These can be aggregated (max pooling over the sparse distributions, mean pooling over the dense vectors) or even used ColBERT-style with MaxSim between query-side and passage-side multi-token representations. The lecture stops there, because these multi-token paragraph/passage representations turn out to be weak.

Worked example 9Combining dense and sparse scores in a hybrid retriever

Setup (illustrative numbers). A query is issued to a PromptReps system. Three candidate passages receive a dense score (cosine similarity from the ANN index) and a sparse score (inverted-index match on the logits vectors, illustrative scale). Use the normalisation and interpolation above with λ=0.5 \lambda = 0.5 :

PassageDense score sdense s_{dense} Sparse score ssparse s_{sparse}
d1 d_1 0.826.0
d2 d_2 0.7812.0
d3 d_3 0.709.0
1
Normalise dense scores. Min = 0.70, max = 0.82, range = 0.12. d1:0.82−0.700.12=1.00d2:0.78−0.700.12=0.667d3:0.70−0.700.12=0.00 \begin{aligned} d_1\text{:}\quad \frac{0.82-0.70}{0.12} &= 1.00 \\ d_2\text{:}\quad \frac{0.78-0.70}{0.12} &= 0.667 \\ d_3\text{:}\quad \frac{0.70-0.70}{0.12} &= 0.00 \end{aligned}
2
Normalise sparse scores. Min = 6.0, max = 12.0, range = 6.0. d1:6−66=0.00d2:12−66=1.00d3:9−66=0.50 \begin{aligned} d_1\text{:}\quad \frac{6-6}{6} &= 0.00 \\ d_2\text{:}\quad \frac{12-6}{6} &= 1.00 \\ d_3\text{:}\quad \frac{9-6}{6} &= 0.50 \end{aligned}
3
Interpolate with λ=0.5 \lambda = 0.5 . d1:0.5×1.00+0.5×0.00=0.50d2:0.5×0.667+0.5×1.00=0.83d3:0.5×0.00+0.5×0.50=0.25 \begin{aligned} d_1\text{:}\quad 0.5 \times 1.00 + 0.5 \times 0.00 &= \mathbf{0.50} \\ d_2\text{:}\quad 0.5 \times 0.667 + 0.5 \times 1.00 &= \mathbf{0.83} \\ d_3\text{:}\quad 0.5 \times 0.00 + 0.5 \times 0.50 &= \mathbf{0.25} \end{aligned}
4
Rank. d2≻d1≻d3 d_2 \succ d_1 \succ d_3 . Neither signal alone gives this ranking: dense alone would put d1 d_1 first, sparse alone would put d1 d_1 last. The hybrid rewards d2 d_2 for being strong on both signals, illustrating how fusion can change the order. The measured effectiveness comparison is the separate MS MARCO result above.
Result Interpolation produces a combined ranking. Relevance judgements are needed to measure whether this ranking improves effectiveness; these illustrative scores demonstrate the calculation only.
PracticeLecture Exercise 1.2: what makes PromptReps a "hybrid" approach?
Say where the dense part and the sparse part come from.
AnswerOne prompted LLM inference gives two representations. The dense part is the last-layer hidden state at the final prompt token, used like a RepLLaMA embedding in an ANN index. The sparse (lexical) part is the next-token scores over the vocabulary: because the prompt asks for one word that represents the text, the top-k predicted terms capture its meaning, and they go into an inverted index. The dense and sparse scores are then combined, so a single encoder yields a hybrid retriever.
PracticeRepeat Worked example 9 with λ=0.8 \lambda = 0.8 . Does the ranking change?
Answer d1:0.8×1.00+0.2×0.00=0.80d2:0.8×0.667+0.2×1.00=0.733d3:0.8×0.00+0.2×0.50=0.10 \begin{aligned} d_1\text{:}\quad 0.8 \times 1.00 + 0.2 \times 0.00 &= 0.80 \\ d_2\text{:}\quad 0.8 \times 0.667 + 0.2 \times 1.00 &= 0.733 \\ d_3\text{:}\quad 0.8 \times 0.00 + 0.2 \times 0.50 &= 0.10 \end{aligned} The ranking becomes d1≻d2≻d3 d_1 \succ d_2 \succ d_3 : with most weight on the dense signal, d1 d_1 's top dense score wins.

11. Summary and study checklist

ModelQuery side (online)Document side (offline)ExpansionTraining signalOnline neural inference?
BM25 + doc2queryPlain termsText + generated queries in a normal inverted indexExplicit, generated by seq2seq (T5)Supervised ⟨query, doc⟩ pairs (for doc2query)No
TILDEBERT tokeniser → binary sparse vector[CLS] → linear → dense |V| vectorImplicit (dense doc vector covers whole vocabulary)BiQDL (bi-directional query/document likelihood)No for QL; yes for DL or QDL
TILDEv2BERT tokeniser → binary sparse vectorContextualised token weights of the expanded documentExplicit, via doc2query or TILDE (cheap)Contextualised exact-match trainingNo
SPLADEFull encoder → sparse weighted vectorFull encoder → sparse weighted vector (sum pooling)Implicit, via MLM head over vocabularyContrastive + ReLU/FLOPS/L1 for sparsityYes (query encoder)
SPLADEv2 (doc-only variant)Simple, near-original queryFull encoder → sparse weighted vector (max pooling)Document-only+ distillation from a teacherNo for the document-only variant
PromptRepsOne LLM inference each → dense (hidden state) + sparse (top-k logits)Via the LLM's summarising wordNone (zero-shot); optional contrastive training helps furtherYes (LLM inference)
Exam angleThe lecturer emphasised explaining each model from its architecture. Practise these four tasks: explain TILDE's three efficiency gains; trace SPLADE from token vectors to vocabulary weights; identify both PromptReps outputs; calculate sum and max pooling column by column. For BiQDL, explain which token probabilities each direction raises or lowers and how ranking choices affect online work (lecture slides 10–14, 21–25).

You should now be able to:

Looking ahead. Week 7 moves from retrieval models to reranking with learning to rank: traditional methods, often applied on top of term-based lexical models, that underpin modern reranking and still sit on top of neural pipelines at search companies. The lecture said the week after turns to reranking with large language models. (At the time of the lecture, Assignment 1 was due on 4 September at 4pm.)

12. Video index and slide decks

VideoTopicLengthNotes sections
Video 1Overview: landscape, intuition, the two routes, expansion14:171–4
Video 2doc2query3:484
Video 3TILDE6:285
Video 4TILDEv26:126
Video 5SPLADE and SPLADEv210:557–8
Video 6Performance of retrievers7:409
Video 7PromptReps8:4510

Slide decks: pre-class deck (51 slides, cited as "slides n") and in-class lecture deck (27 slides, cited as "lecture slides n"). The in-class lecture transcript (w6-lec-INFS7410_S2_2026_STL-transcript.txt) is cited as "Lecture transcript"; the pre-class video transcripts are in Pre-class youtube Transcripts/.

13. References