INFS7410 · Information Retrieval and Web Search · Week 6
Learned Sparse Retrievers: TILDE, TILDEv2, SPLADE and PromptReps
Lexical retrieval models with learned term weights. This week combines the efficiency of inverted-index lexical search with the semantic power of transformers. It covers query and document expansion (doc2query), the TILDE family, the SPLADE family, an effectiveness comparison across every retriever seen so far, the training-free LLM-based PromptReps, and the exercises from the in-class lecture.
The in-class lecture opened with a recap of the ranking models covered so far. Week 1 defined information retrieval and text processing (tokenisation, stemming, stopwords). Week 2 covered evaluation (set-based and rank-based measures, pooling, statistical tests). Week 3 introduced the term-based lexical models (Boolean model, TF-IDF, the vector space model, BIM and BM25). Week 4 explained the transformer architecture (encoder and decoder, BERT and GPT). Week 5 turned transformers into retrievers (DPR, ANCE, RepBERT, ColBERT, E5-Mistral, LLM2Vec, RepLLaMA). Week 6 covers learned sparse models (TILDE, TILDEv2, SPLADE, SPLADEv2) and PromptReps.
On the lecture's historical timeline, learned sparse models sit beside dense retrievers in the "new inventions" era (2018–2022), after term-based lexical models (1950s–1990s) and the transformer (2017–2018). The lecturer's point: learned sparse models are a mix of the Week 4 transformer architecture and the ideas behind the Week 3 term-based lexical models.
Over the past weeks we have built up a table of relevance model families. Term-based lexical models such as BM25 (Week 3) match exact words through an inverted index (Week 1). Single-vector dense retrievers such as RepBERT and ANCE (Week 5) encode a whole query or document into one embedding. Multi-vector dense retrievers such as ColBERT keep one embedding per token. This week fills the remaining cell: learned sparse retrievers, also called lexical-based retrieval models with learned term weights. They sit between traditional lexical models and dense embedding models and try to get the best of both.
Family
Encoding
Online scoring
Stored representation
BM25
Tokenisation and term statistics
Traverse posting lists for query terms
Sparse term weights
TILDE-QL
BERT on documents offline; tokeniser on queries online
Learn document weights offline; standard SPLADE also encodes queries online
Use shared active vocabulary terms
Sparse document weights
Single-vector dense
Transformer on each text; document encoding offline
One vector comparison per candidate; ANN can select candidates
One dense vector per document
Multi-vector dense
A similar transformer backbone; retain token vectors
Token-to-token comparisons and MaxSim aggregation
Several dense vectors per document
Reading the lecture's cost diagram. Separate encoding a text, scoring one candidate, and searching a collection. The lecturer says that single-vector and multi-vector models can use the same encoder backbone. Keeping token vectors increases storage and scoring work. Full transformer self-attention considers token pairs in either family. The slide's “O(<1)” label is a qualitative efficiency cue. Actual search cost depends on posting-list lengths, active terms, candidate counts, and the index. Dense retrieval can use approximate nearest-neighbour (ANN) search to avoid an exhaustive scan.
Qualitative positions: learned sparse retrievers keep the low scoring cost of lexical models and move up towards the effectiveness of dense retrievers.PracticeWhy is ColBERT's scoring more expensive than RepBERT's, and which column of the table do learned sparse retrievers improve on?
Count how many vectors each model keeps per query and per document.
AnswerRepBERT keeps one vector per query and per document, so scoring one document is a single dot product. ColBERT keeps one vector per token, so it must compare every query token vector with every document token vector (MaxSim), which multiplies the work. Learned sparse retrievers improve the scorer column: their sparse vectors let the engine score only documents that share active terms with the query, just like BM25.
For the query "machine learning algorithms", a traditional system like BM25 looks only at documents containing the exact words machine, learning, or algorithms. The corpus is organised in an inverted index: for each term, a posting list of the documents that contain it. At query time the engine only loops over the posting lists of the query terms, and this set of candidate documents is usually a tiny fraction of the whole collection. That is the source of the high efficiency of lexical models. The lecture added a practical reminder: BM25 runs on a CPU in a few milliseconds per query (the Assignment 1 BM25 runs should take minutes, and hours signal a bug).
Lexical retrieval only examines the small subsets of the corpus whose documents contain query terms.
2.2 Dense retrievers: high effectiveness
Dense embedding-based retrievers (Week 5) push both queries and documents through transformer encoders to produce dense vectors, and rank by vector similarity (cosine similarity, dot product, or even Euclidean distance). They are much more effective because they do semantic matching: they understand that machine learning relates to artificial intelligence, neural networks or transformers even when no word overlaps. The price is a computationally heavy encoder, and scoring that in principle involves every document vector in the collection.
Key idea
Lexical models: highly efficient, limited in effectiveness (exact matching only). Dense models: highly effective (semantic matching), computationally expensive. The central question of Week 6: can we combine their advantages?
PracticeA user searches "AI tutoring". A relevant document talks only about "machine learning tutors". Which retriever family ranks it well, and what does the other family pay for that?
AnswerBM25 matches only "tutoring" (if stemming maps it to "tutor") and misses the link between "AI" and "machine learning", so it under-ranks the document. A dense retriever places both texts close together in embedding space and ranks it well. The dense retriever pays with a transformer encoding of the query at search time and vector similarity search (exhaustive or ANN).
Traditional term-based lexical retrievers have three limitations. First, their weighting is based solely on term overlap and collection statistics (term frequency, document frequency, document length). Second, they use hand-crafted formulas designed from intuition and some theoretical foundations. Third, they learn nothing from data: apart from a couple of tunable parameters, the same formula is applied to every query, domain, and document type.
Learned sparse models take a different approach:
Learn optimal term weights from training data, so the model adapts to the content, queries, and domain it is trained on.
Use a powerful transformer to predict term importance, which gives far higher effectiveness than simple statistical measures.
Maintain sparsity for efficiency, so scoring can still use the inverted-index trick of only touching documents that share (possibly expanded) terms with the query.
This week presents two main routes to this goal, plus a recent LLM-based hybrid:
Route
Idea in one line
Models
Drop the query encoder
Treat queries and documents asymmetrically: the query is only tokenised into a sparse term-existence vector (no neural network at query time), while documents are still encoded into dense vocabulary-sized vectors offline. Document indexing and encoding time stay the same; query encoding time almost disappears; the sparse query vector also cuts scoring time.
TILDE, TILDEv2
Keep the encoder, output term weights
Use the transformer encoder for both queries and documents, and make it output weights over the whole vocabulary (an "expanded set of terms"). Encoding time stays the same as a dense model; scoring becomes cheap because the vectors are sparse.
SPLADE, SPLADEv2
Prompt an LLM, no training at all
Prompt a generative LLM to summarise a text in one word, and harvest a dense vector (hidden state) plus a sparse vector (next-token logits) from that single inference.
PromptReps
The two main routes: TILDE removes the query encoder (asymmetric), SPLADE keeps both encoders and makes their outputs sparse.
3.1 Why sparsity is the efficiency
Both routes end with a dot product in the ∣V∣-dimensional vocabulary space. Written out, it shows exactly where the saving comes from:
s(q,d)=i=1∑∣V∣qidi=i:qi=0∑qidi
Where
s(q,d)
relevance score of document d for query q
∣V∣
vocabulary size (30,522 for the BERT vocabulary)
i
index of one vocabulary term (one dimension)
qi
weight of term i in the query vector
di
weight of term i in the document vector
i:qi=0
only the dimensions where the query weight is non-zero
Read it aloud
The score of document d for query q is the sum, over every vocabulary term i, of the query weight times the document weight, which equals the same sum taken only over the terms where the query weight is non-zero.
Formula in plain words
Every dimension where the query weight is zero contributes zero, whatever the document holds there, so the engine skips it. The work per document therefore grows with the number of active query terms, a handful, instead of the 30,522 vocabulary dimensions. If most weights are exactly zero, the engine can also precompute posting lists over the non-zero terms and score only documents that share active terms with the query, exactly like BM25. A query that activates more terms (for example through expansion) costs proportionally more to score.
Key idea
Sparsity is the efficiency. A learned sparse model is only as fast as its vectors are sparse: if every weight were non-zero, the dot product would cost as much as a 30,522-dimensional dense one.
PracticeA sparse query vector has 4 non-zero entries out of 30,522. How many multiplications does scoring one document need, and how many would a fully dense 30,522-dimensional vector need?
Answer4 multiplications (one per non-zero query entry, plus 3 additions), against 30,522 for the dense case. The zero entries contribute nothing, so the engine never computes them.
PracticeName the three limitations of BM25-style retrievers that learned sparse models address, and the property they keep from BM25.
Answer(1) Weights come only from term overlap and collection statistics; (2) the formulas are hand-crafted; (3) nothing is learned from data. Learned sparse models learn term weights from training data with a transformer, and they keep BM25's sparsity, which preserves inverted-index efficiency.
Query/document expansion means augmenting the user query, or a document in the corpus, with additional relevant terms. The goal is to improve effectiveness by matching related terms that are absent from the original text. This attacks the vocabulary mismatch problem directly (a query says "lower blood sugar", the document says "reduces blood sugar levels"). A variety of automatic and semi-automatic expansion techniques exist; semi-automatic ones require user interaction to select the best expansion terms, automatic ones need no user input. Query/document rewriting is a closely related technique. Many learned sparse retrievers rely on expansion for better effectiveness, which is why it comes first. The lecturer noted that expansion returns in more depth in Week 12.
The in-class lecture used this example. The original query "who wrote the book of the lord of the rings" is expanded with terms that capture what the question means: write, created, authored, novel, LOTR, Tolkien. The expanded terms are new words (repeating the query's own words adds nothing), and the expanded query is the original query plus these terms. The same idea applies on the document side: a document about "AI" expanded with "machine" and "learning" now matches a query that uses those words.
Expansion adds related terms so that relevant documents with different wording still share terms with the query (lecture slide 7).
4.2 doc2query: document expansion with a seq2seq transformer
With document expansion we add terms (or modify term weights) in the index to improve performance. doc2query does this with a sequence-to-sequence transformer: a transformer with both an encoder and a decoder that maps input text to output text (T5 is a common example). BERT, by contrast, maps input text to embeddings. The model's task: given a document, generate a query that this document would answer. Training is supervised, using pairs of ⟨query, relevant document⟩ (e.g. from MS MARCO).
The doc2query pipeline: generate 5–40 queries per document, concatenate them onto the document, and index the expanded documents as usual.
In practice, between 5 and 40 queries are sampled per document (with top-k or nucleus sampling). Each document is concatenated with its predicted queries to form an expanded document, which is indexed into a traditional inverted index. When a user searches, the engine (e.g. BM25) matches against the expanded index, which bridges much of the vocabulary gap between query and document.
4.3 Does it work?
Yes. The expansion strategy consistently improves results over the same ranking model on non-expanded documents, regardless of the ranker (a bag-of-words model, BM25 with RM3 pseudo-relevance feedback, or BM25 followed by a monoBERT/T5 reranker):
Ranker
MS MARCO Passage (MRR@10)
TREC-DL 19 (nDCG@10)
BM25
0.184
0.506
BM25 + doc2query
0.277
0.642
BM25 + RM3 (pseudo-relevance feedback)
0.156
0.518
BM25 + RM3 + doc2query
0.214
0.655
BM25 + monoBERT/T5 reranking
0.365
0.738
BM25 + monoBERT/T5 + doc2query
0.379
0.754
Inspecting the generated queries shows doc2query does two useful things at once: it injects new words into the document (directly addressing keyword mismatch), and it copies/repeats keywords already in the document. Excluding stop words, about 69% of generated words are copied and 31% are new. Repetition matters too: it increases the importance (term frequency) of those keywords in the expanded document. The slide's three examples:
Input document (shortened)
Predicted query
Target query
New words
"July is the hottest month in Washington DC with an average temperature of 27C …"
weather in washington dc
what is the temperature in washington
weather
"The Delaware River flows through Philadelphia into the Delaware Bay …"
what river flows through delaware
where does the delaware river start and end
none (all copied)
"sex chromosome … a threadlike strand of DNA in the cell nucleus that carries the genes …"
what is the relationship between genes and chromosomes
which chromosome controls sex characteristics
relationship
Ablation (MS MARCO passage)
MRR@10
R@1000
Original document
0.184
0.853
+ expansion with new words only
0.195
0.907
+ expansion with copied words only
0.221
0.893
+ expansion copied + new
0.277
0.944
Only expansions (original document removed)
0.263
0.927
Takeaways
Both kinds of expansion help, and copied words contribute more than new words (0.221 against 0.195 MRR@10). Strikingly, indexing only the generated queries, with the original document removed (0.263), beats indexing the original documents alone (0.184).
Worked example 1How doc2query expansion changes a BM25 score
Setup (illustrative numbers). A user query q="lower blood sugar" and a passage d: "Researchers are finding that cinnamon reduces blood sugar levels naturally when taken daily" (13 terms, so ∣d∣=13). Assume the collection has N=1,000,000 passages, average length avgdl=15, BM25 parameters k1=1.2, b=0.75, and document frequencies df(lower)=40,000, df(blood)=25,000, df(sugar)=20,000. We use the BM25 form from Week 3:
Formula in plain words
The score is the sum over every query term of two factors: an IDF factor (the log of roughly "how rare the term is across the collection", so rarer terms count more) multiplied by a saturating term-frequency factor (the more often the term appears in the document the higher the factor, with diminishing returns, adjusted for whether the document is longer or shorter than average). Here tf(t,d) is the count of t in d, df(t) the number of documents containing t, N the collection size, k1 the saturation parameter and b the length-normalisation parameter.
1
Score the original document. Term frequencies in d: tf(lower)=0, tf(blood)=1, tf(sugar)=1. The passage says "reduces", so "lower" never appears. Its contribution is 0: this is vocabulary mismatch.
2
Compute the shared pieces. Length normalisation:
1−b+bavgdl∣d∣=1−0.75+0.75×1513=0.25+0.65=0.90
So the TF factor for a term with tf=1 is
1+1.2×0.901×2.2=2.082.2=1.058
3
Compute IDFs.idf(blood)idf(sugar)idf(lower)=ln(25,000+0.51,000,000−25,000+0.5+1)=ln(40.0)=3.689=ln(20,000.5980,000+1)=ln(50.0)=3.912=ln(40,000.5960,000+1)=ln(25.0)=3.219
The value for "lower" is unused for now, since tf=0.
4
Original score.BM25=3.689×1.058+3.912×1.058=3.903+4.139=8.04. The term "lower" contributed nothing.
5
Expand the document. doc2query predicts the query "does cinnamon lower blood sugar" (5 terms), which we append. The expanded document d′ has ∣d′∣=18 terms and new counts: tf(lower)=1, tf(blood)=2, tf(sugar)=2. Both effects from the lecture appear: a new word ("lower") was injected, and existing words ("blood", "sugar") were copied, raising their term frequency.
6
Recompute the pieces for d′.length normalisation:0.25+0.75×1518denominator constant:k1×1.15TF factor, tf=1:1+1.382.2TF factor, tf=2:2+1.382×2.2=0.25+0.90=1.15=1.38=0.924=3.384.4=1.302
That is larger than for tf=1 and less than double: BM25's term frequency saturates.
Result
Expansion lifted the document's score for this query from 8.04 to 12.87, and, more importantly, the previously unmatched query term "lower" now matches. Query time is unchanged: the expansion work happened once, offline, at indexing. This is the mechanism behind the 0.184 → 0.277 MRR@10 jump for BM25 on MS MARCO.
PracticeThe ablation shows that expansion with copied words alone improves MRR@10. How can adding words that are already in the document help BM25?
Think about which BM25 input the copies change.
AnswerCopies raise the term frequency of the words that the generated queries consider important, so BM25 gives those words a larger TF factor. The document's representation then emphasises the terms users actually search for (0.184 → 0.221 MRR@10 with copied words only).
PracticeWhy does doc2query add no cost at query time, and where does its cost go instead?
AnswerThe seq2seq model runs once per document at indexing time; the expanded text sits in an ordinary inverted index, and queries are processed by plain BM25. The cost moves offline (generation for every document, plus a larger index). Section 6 shows this offline cost can still be very large.
5.1 The idea: replace the query encoder with a tokeniser
TILDE (Zhuang & Zuccon, SIGIR 2021) takes the first route from Section 3: remove query encoding entirely. In a dense bi-encoder (Week 5), both towers run BERT. TILDE keeps the document tower and replaces the query tower with the plain BERT tokeniser, the first step of the Week 4 input pipeline, before any embedding or encoder layer:
Query: tokenised into a sparse vector of size ∣V∣=30,522 (the BERT vocabulary). Element i is 1 if token ti appears in the query and 0 otherwise: a term-existence vector, like exact matching over the BERT vocabulary in place of the collection vocabulary.
Document: encoded by BERT; the [CLS] token embedding (768 dims) is projected by a trained linear layer up to a dense vector of size ∣V∣=30,522. Each dimension holds the learned importance (probability) of that vocabulary term for the document. This is fully precomputed offline at indexing time.
Score: the inner product between the sparse query vector and the dense document vector, in the ∣V∣-dimensional space.
s(q,d)=q⊤d=i=1∑∣V∣qidi=ti∈q∑di
Where
q
binary query vector of length ∣V∣, produced by the BERT tokeniser
d
dense document vector of length ∣V∣: [CLS] embedding passed through the linear layer
qi
1 if vocabulary token ti occurs in the query, else 0
di
learned weight (probability) of token ti for the document
ti∈q
the vocabulary tokens that occur in the query
∣V∣
BERT vocabulary size, 30,522
Read it aloud
The score s of document d for query q is the transpose of q times d, which is the sum over all vocabulary positions i of q-i times d-i, which equals the sum of d-i over the tokens t-i that occur in the query.
Formula in plain words
Because query entries are only 0 or 1, the inner product collapses to one operation: add up the document's learned weights at the positions of the query's tokens. A 5-token query costs 5 look-ups and additions per document. A document scores higher when BERT assigned high weights to the query's tokens, even tokens that never appear in its text. Adding a query token adds one more term to the sum; the query side carries no weights at all, so all the learning lives in the document vector.
TILDE: the query side is just a tokeniser producing a binary sparse vector; the document side is BERT's [CLS] embedding projected to vocabulary size.
Takeaways(1) Semantic matching survives. Even if a query keyword never appears in the document, the document's dense vector (produced by BERT) can still assign that keyword a high weight: the model has learned which vocabulary terms the document is about. (2) No GPU at query time. The tokeniser is a small lookup table, so query "encoding" is nearly free and runs fine on a CPU.
Worked example 2Scoring with TILDE's sparse × dense inner product
Setup (illustrative numbers). To keep the numbers readable, use a toy vocabulary of 8 BERT tokens in place of 30,522: V={apple,store,account,fruit,buy,banana,login,the}. The user query is "apple account". Two candidate passages were encoded offline; their [CLS] embeddings, after the linear projection, give these dense vocabulary-space weights (each row is Pθ(ti∣d)):
Passage
apple
store
account
fruit
buy
banana
login
the
d1: "how to sign in to your Apple ID"
0.62
0.28
0.51
0.04
0.19
0.02
0.44
0.30
d2: "apples are a healthy fruit to snack on"
0.66
0.07
0.03
0.58
0.21
0.35
0.01
0.31
1
Tokenise the query (no GPU). "apple account" activates dimensions apple and account: q=[1,0,1,0,0,0,0,0]. Every other dimension is 0.
2
Inner product with d1. Only the two active dimensions matter: s(q,d1)=1×0.62+1×0.51=1.13.
3
Inner product with d2.s(q,d2)=1×0.66+1×0.03=0.69.
4
Rank.d1≻d2. Semantic matching is at work: d1 never contains the literal word "account", yet BERT learned that a passage about "signing in to your Apple ID" is about accounts, so its dense vector carries a high weight (0.51) on the account dimension. A purely lexical matcher would have missed this. d1 also has a weight of 0.44 on "login", another term absent from the passage: this is implicit expansion in the document representation.
5
TILDE-QL with log weights. Using log-likelihoods in place of raw probabilities:
TILDE-QL(q∣d1)TILDE-QL(q∣d2)=ln0.62+ln0.51=−0.478−0.673=−1.151=ln0.66+ln0.03=−0.416−3.507=−3.923
Since −1.151>−3.923, the ranking is the same: d1 first.
Result
Scoring each document cost two weight lookups and one addition after a tokeniser lookup. This is why TILDE's query latency is tiny and GPU-free, while it still rewards a document that matches the query only semantically.
Common trapThe lecture architecture illustrates a sum of term weights. For the TILDE-QL ranking rule in Section 5.4, store log-probabilities as those weights. Adding raw probabilities and adding log-probabilities can produce different rankings. Worked example 2 happens to give the same order with both. Each TILDE output is an independent token probability, so its vocabulary entries can sum to more than one.
5.2 Why TILDE is fast at query time
In the lecture's Exercise 1 (lecture slide 21) the class listed three separate reasons for TILDE's efficiency compared with dense retrievers:
No query encoding. At query time the query is only tokenised; it never enters the BERT model. Tokenising is the first step before the input embeddings, so it is extremely fast.
Document encoding happens offline. Documents still pass through BERT, but once, at indexing time. If the collection is static, the index never changes, however many queries arrive.
The query vector is sparse. Only the query's own tokens are 1; all other vocabulary entries are 0. A query almost always has far fewer tokens than the vocabulary, so the dot product touches only a few dimensions. For each candidate, score only the active query-token dimensions.
Query-time neural work for K candidate passages: TILDE-QL needs no BERT inference at all. Adding the TILDE-DL signal (Section 5.4) brings back one query inference.
The "term independent" in TILDE's name is what makes the offline work possible. TILDE assumes query tokens are independent of each other, so the likelihood of each vocabulary term depends only on the passage, Pθ(t∣d). The model can therefore predict every term's likelihood for a passage before any query exists, store the results, and answer a query with look-ups. A term-dependent model, where each query term's probability also depends on the earlier query terms, has to run its neural inference after the query arrives.
Term independence turns online neural inference into offline precomputation followed by query-time look-ups.
Worked example 3Query-time operation count for 1,000 candidates
Setup (illustrative numbers). A three-token query must be scored against 1,000 candidate passages. TILDE-QL has the passage weights stored in the index. A dense retriever scores the same candidates exactly with 768-dimensional vectors.
1
TILDE-QL. Tokenise the query (no BERT). For each candidate, look up and add 3 stored values: 1,000×3=3,000 look-ups and additions.
2
Dense scoring. Run BERT once on the query, then compute one 768-dimensional dot product per candidate: 1,000×768=768,000 multiplications.
3
Compare.3,000768,000=256
The dense scorer processes 256 times more values, and it also needs a BERT query encoding that TILDE-QL skips.
Result
TILDE-QL: 3,000 cheap operations and no neural inference. Dense: 768,000 multiplications plus one BERT inference. (This counts exact candidate scoring; an ANN index changes the dense cost profile.)
Worked example 3 on a log scale: the dense scorer processes 256 times more values than TILDE-QL.
Watch out
TILDE-QL has a sparse query and a dense document vector. Each candidate can have a weight for every query token, including tokens absent from its text. For 1,000 candidates and 3 distinct query tokens, scoring reads 3,000 stored weights. Scanning all 30,522 dimensions would read 30,522,000 weights. The saving is in dimensions per candidate. A first-stage retriever can supply the candidate set. Document precomputation also helps dense bi-encoders; TILDE-QL additionally removes online query inference.
Query sparsity reduces work within each candidate. Candidate selection is a separate step.
5.3 The training loss: BiQDL
Another key difference from other bi-encoders is the loss. In place of the margin or contrastive loss of Week 5, TILDE is trained with the traditional query likelihood paradigm from statistical language models, assuming query tokens are independent of each other (the same assumption query likelihood models make). The bi-directional query-document likelihood loss (BiQDL) has two directions. Training uses relevant ⟨query, document⟩ pairs.
Direction 1: query likelihood (QL). The model reads the document text and must predict which vocabulary tokens appear in the query:
LQL(D)=−(q,d)∈D∑∣V∣1i∑∣V∣[ylog(Pθ(ti∣d))+(1−y)log(1−Pθ(ti∣d))]y={1,0,if vocabulary term ti appears in query q,otherwise.
Where
LQL(D)
query-likelihood loss over the training set D
D
training set of relevant ⟨query, document⟩ pairs
(q,d)
one relevant query–document pair
∣V∣
vocabulary size; ∣V∣1 averages over all vocabulary tokens
ti
the i-th vocabulary token
y
label: 1 if ti is in the query, 0 otherwise (yi on the lecture slides)
Pθ(ti∣d)
the model's probability (importance) of token ti given the document: entry i of the document vector
θ
the model parameters (BERT and the linear layer) being trained
Read it aloud
The query-likelihood loss over D is minus the sum over every query–document pair in D of one over the vocabulary size times the sum over every vocabulary token t-i of: y times the log of the probability of t-i given d, plus one minus y times the log of one minus that probability. Here y is 1 if t-i appears in the query and 0 otherwise.
Formula in plain words
For each training pair, take the average over all ∣V∣ vocabulary tokens of a binary cross-entropy. If the token occurs in the query (y=1), only the logPθ(ti∣d) part is active: a higher probability makes that log closer to zero and lowers the loss. If the token is absent from the query (y=0), only the log(1−Pθ) part is active: a higher probability now raises the loss. The bracket is a quantity we want to maximise; the leading minus sign turns it into a loss to minimise. In short: push up the probability of query tokens, push down the probability of every other token. The lecturer's advice: understand why each part is there; the exam is more likely to ask you to explain the function than to write it down.
The QL direction of BiQDL (lecture slide 10): training raises the document's weights at the query's tokens and lowers them everywhere else.
Direction 2: document likelihood (DL). Simply swap the roles: the query is encoded by BERT into a dense representation, and the document becomes the sparse tokenised side. The model learns, from the query text, which vocabulary terms are important for the relevant document:
LDL(D)=−(q,d)∈D∑∣V∣1i∑∣V∣[ylog(Pθ(ti∣q))+(1−y)log(1−Pθ(ti∣q))]y={1,0,if vocabulary term ti appears in document d,otherwise.
Where
LDL(D)
document-likelihood loss over the training set D
Pθ(ti∣q)
the model's probability of token ti given the query (BERT now encodes the query)
y
label: 1 if ti is in the document, 0 otherwise
D,(q,d),∣V∣,ti,θ
as in LQL
Read it aloud
The document-likelihood loss over D is minus the sum over every pair of one over the vocabulary size times the sum over every vocabulary token of: y times the log of the probability of t-i given q, plus one minus y times the log of one minus that probability, where y is 1 if t-i appears in the document.
Formula in plain words
Identical structure, mirrored: given the query text, push up the probability of tokens that appear in the document and push down the probability of tokens that do not. If you understand the QL direction, you understand this one. It trains the query side to predict document terms, which enables the TILDE-DL ranking mode below.
Combined. The bi-directional loss is the average of both directions:
LBiQDL(D)=2LQL(D)+LDL(D)
Where
LBiQDL(D)
the bi-directional query-document likelihood loss used to train TILDE
LQL(D),LDL(D)
the query-likelihood and document-likelihood losses above
Read it aloud
The BiQDL loss over D equals the query-likelihood loss plus the document-likelihood loss, all divided by two.
Formula in plain words
The final training loss is an equal-weight average of the two directions. Lowering either direction's loss lowers the total, so training improves both abilities at once. After training with BiQDL, TILDE can estimate the query likelihood given a document text and the document likelihood given a query text.
Worked example 4Computing the BiQDL loss on a tiny vocabulary
Setup (illustrative numbers). One training pair: query q= "apple account", relevant document d= "apple store account help". Toy vocabulary V={apple,store,account,banana}, so ∣V∣=4. The current model, reading d, predicts these probabilities of each token appearing in the query: Pθ(apple∣d)=0.7, Pθ(store∣d)=0.5, Pθ(account∣d)=0.3, Pθ(banana∣d)=0.2. Reading q, it predicts these probabilities of each token appearing in the document: Pθ(apple∣q)=0.8, Pθ(store∣q)=0.4, Pθ(account∣q)=0.6, Pθ(banana∣q)=0.1. (All logs are natural logs.)
1
Build the label vector for the QL direction.yi=1 if token ti is in the query. Query tokens: {apple, account}. So y=[1,0,1,0] for (apple, store, account, banana).
2
Per-token QL terms. For tokens in the query use logP; for tokens outside the query use log(1−P):
apple (y=1):log0.7store (y=0):log(1−0.5)account (y=1):log0.3banana (y=0):log(1−0.2)=−0.357=log0.5=−0.693=−1.204=log0.8=−0.223
3
Average and negate.sumaverageLQL=−0.357−0.693−1.204−0.223=−2.477=∣V∣sum=4sum=−0.619=−(average)=0.619
The biggest contributor is "account": the model gave a true query token only probability 0.3, and gave the non-query token "store" a too-high 0.5. Training will push P(account∣d) up and P(store∣d) down.
4
Label vector for the DL direction. Now yi=1 if ti is in the document "apple store account help". Document tokens within our toy V: {apple, store, account}, so y=[1,1,1,0].
Average and negate.sumaverageLDL=−1.755=4sum=−0.439=−(average)=0.439
7
Combine.LBiQDL=20.619+0.439=0.529.
Result
The BiQDL loss for this pair is 0.529. Minimising it simultaneously teaches the model to predict query tokens from documents (QL direction) and document tokens from queries (DL direction), which is precisely what lets a trained TILDE rank with query likelihood, document likelihood, or both.
5.4 Three ways of ranking with TILDE
These two abilities give three ranking settings that trade efficiency for effectiveness. The in-class lecture annotated each one (lecture slide 13).
TILDE-QL(q∣dk)=i∑∣q∣log(Pθ(qi∣dk))
Where
dk
the k-th candidate document being ranked
∣q∣
number of tokens in the query
qi
the i-th query token
Pθ(qi∣dk)
stored probability of query token qi given the document (from the offline document vector)
Read it aloud
TILDE-QL of q given d-k is the sum, over each token q-i of the query, of the log of the probability of q-i given d-k.
Formula in plain words
Rank each document by how likely it makes the query's tokens. Because document representations are precomputed, this is just a few look-ups and additions: in effect a dot product, cheap and effective, with no GPU at query time. This is TILDE's default setting. Logs of probabilities are negative, so scores closer to zero are better; one query token with a tiny probability drags the whole score down, because its log is a large negative number.
TILDE-DL(dk∣q)=∣dk∣1i∑∣dk∣log(Pθ(dik∣q))
Where
∣dk∣
number of tokens in document dk
dik
the i-th token of document dk
Pθ(dik∣q)
probability of that document token given the query (BERT encodes the query at search time)
Read it aloud
TILDE-DL of d-k given q is one over the length of d-k times the sum, over each token of d-k, of the log of the probability of that token given q.
Formula in plain words
Rank by the average log-probability of the document's tokens given the query. The division by document length ∣dk∣ normalises the score: longer documents contain more tokens and would otherwise accumulate lower (more negative) likelihoods. This direction needs a BERT inference on the query at search time, so online inference adds latency; a GPU can accelerate it. The lecture calls it an ablation in the paper; TILDE itself does not need it.
interpolation weight in [0,1]: the share given to query likelihood
TILDE-QL,TILDE-DL
the two scores above
Read it aloud
TILDE-QDL of q and d-k equals alpha times TILDE-QL of q given d-k, plus one minus alpha times TILDE-DL of d-k given q.
Formula in plain words
The final score is a weighted mix of the two directions. α=1 gives pure query likelihood (fastest, tokeniser only); α=0 gives pure document likelihood; values in between blend the evidence. Any α<1 still needs the online query inference, so the mix gives a minor effectiveness gain at the cost of query latency.
Mode
Online work
Lecture comment
When a system designer picks it
TILDE-QL (default)
Tokenise the query; look up stored document weights
"Just a dot product (cheap and effective)"
Usual case: the corpus is a manageable size, so encode every document once offline and give users the fastest response
TILDE-DL
BERT inference on the query at search time
An ablation in the paper; slow at query time
The document side is huge and you prefer to skip BERT document preprocessing, while queries are relatively few
TILDE-QDL
Both of the above
A minor gain; still slow, because it involves online computation
Effectiveness matters most and extra latency is acceptable (still faster than a dense model)
5.5 Impact of the document likelihood signal
Plotting nDCG@10 and MAP against α (on TREC DL 2019 and 2020) shows:
α=0 (document likelihood only): performance is very poor (nDCG@10 ≈ 0.23–0.25, MAP ≈ 0.13–0.20). Documents cannot be ranked with document likelihood alone.
α=0.4–0.5: best performance (nDCG@10 ≈ 0.61–0.62). Document likelihood helps when mixed in.
α=1 (pure query likelihood): a small drop from the best (≈ 0.58–0.62), and this setting has the minimum query latency because no GPU inference is needed at query time.
nDCG@10 of TILDE-QDL as α varies (approximate values read from slide 26). The MAP plot on the same slide has the same shape.
Worked example 5TILDE-QDL interpolation with α
Setup (illustrative numbers). For a query q and two candidate passages, a trained TILDE gives: TILDE-QL(q∣d1)=−1.9, TILDE-DL(d1∣q)=−0.6; and TILDE-QL(q∣d2)=−1.6, TILDE-DL(d2∣q)=−1.4. All are log-likelihoods, so higher (closer to zero) is better. The DL scores are already length-normalised.
1
Pure query likelihood, α=1. Scores are just QL: d1:−1.9, d2:−1.6 → ranking d2≻d1. Fastest setting: no GPU needed.
2
Pure document likelihood, α=0. Scores are just DL: d1:−0.6, d2:−1.4 → ranking d1≻d2. The lecture showed this setting alone performs badly overall.
3
Interpolate at the sweet spot α=0.4.d1:0.4×(−1.9)+0.6×(−0.6)d2:0.4×(−1.6)+0.6×(−1.4)=−0.76−0.36=−1.12=−0.64−0.84=−1.48
4
Rank.−1.12>−1.48, so d1≻d2. The document-likelihood evidence (which strongly favoured d1) flipped the ranking relative to pure QL.
Result
α controls a real trade-off: α=1 gives minimum latency (tokeniser only), while α≈0.4–0.5 gives the best effectiveness at the cost of one BERT inference on the query at search time (GPU needed). The empirical plots show the effectiveness gap between these two settings is small.
PracticeLecture Exercise 1: how does TILDE achieve high efficiency compared to dense retrievers?
The lecture expected three separate aspects.
Answer(1) At query time the query is only tokenised, so no BERT encoding runs online. (2) Documents do pass through BERT, but offline at indexing time, once; the index stays fixed while queries arrive. (3) The query vector is sparse (1 only at the query's own tokens), so scoring touches only those few dimensions of each document vector.
PracticeIn LQL, what happens to the loss if training raises Pθ(ti∣d) for a token that is absent from the query? Why is there a minus sign in front of the sum?
AnswerFor that token y=0, so only log(1−Pθ) is active. Raising Pθ shrinks 1−Pθ, makes the log more negative and so raises the loss. The bracketed sum is a log-likelihood we want to maximise; the minus sign turns it into a loss that training minimises.
PracticeYour search service has an enormous, frequently changing document collection and a small number of queries per day. Which TILDE ranking mode suits it, and what do you give up?
AnswerTILDE-DL: it avoids running BERT over every document offline, because the neural work happens on the (few) queries. You give up query latency (a BERT inference per query, GPU needed) and effectiveness, since DL alone ranks poorly (nDCG@10 ≈ 0.23 at α=0).
TILDEv2 keeps the BERT-tokeniser sparse query encoding and changes the document side in two ways:
Token embeddings in place of [CLS]. The document is represented by its contextualised token embeddings, one weight per token. Scoring computes the inner product between the sparse query vector and these token embeddings; in practice this only scores exact matches between query terms and tokens of the (expanded) document. Each matched token carries a contextualised learned weight in place of a count, which handles the semantic-gap problem.
Document expansion before encoding. Pure exact matching would suffer from query–document vocabulary mismatch, so the document is expanded before encoding, using doc2query or TILDE itself: a trained TILDE can generate the most likely query terms for a document, just as doc2query does, far more cheaply.
TILDEv2: exact term matching between the sparse query vector and contextualised token weights of the expanded document. Here "account" exists in the document only because expansion added it.
Worked example 6Scoring a query with TILDEv2
Setup (illustrative numbers). Query: "apple account". Original passage d: "apple store". At indexing time, TILDE expansion proposed the extra tokens {account, iphone} (likely query terms for this passage); after filtering (see 6.2), both were appended. BERT then encoded the expanded passage and produced one learned weight per token: apple → 4.7, store → 0.2, account → 3.1, iphone → 2.4.
1
Tokenise the query. Active terms: {apple, account}. This is the sparse query vector; only these terms will be scored.
2
Exact-match lookup. For each query term, find matching tokens in the expanded document and take their contextualised weights: "apple" matches with weight 4.7; "account" matches with weight 3.1, only because expansion added it. (If a term appeared several times, we would take its stored match weight from the index.)
3
Sum the matched weights.s(q,d)=4.7+3.1=7.8.
4
Contrast with the unexpanded passage. Without expansion, "account" matches nothing and s=4.7: the passage would rank as if it had nothing to say about accounts. Contrast also with plain BM25: BM25 would give "apple" a weight from counts and rarity alone, while TILDEv2's 4.7 is contextualised. The same word "apple" in a fruit passage would get a different weight than in this tech-support passage.
Result
TILDEv2 = exact term matching (fast, inverted-index friendly) + learned contextualised term weights (semantics) + offline expansion (fixes vocabulary mismatch). Scoring cost is again a handful of additions per document.
6.2 How important is the expansion component?
Empirical results (MRR@10, MS MARCO dev) as the expansion method varies. When TILDE generates m candidate terms, only terms that are not stop words and not already in the document are actually added (the same filtering is applied to doc2query). More added terms also means a larger index:
Measure
No expansion
doc2query
TILDE m=128
TILDE m=150
TILDE m=200
MRR@10
0.299
0.333
0.326
0.327
0.330
Avg added tokens
–
19.0
13.0
25.2
61.6
Index size
4.3 GB
5.2 GB
5.2 GB
5.6 GB
6.9 GB
Expansion cost
–
320 hours, $768 (TPU)
7.22 hours, $5.34 (GPU)
7.25 hours, $5.37
7.33 hours, $5.42
Reading the table: expansion always beats no expansion (0.299 → 0.32+). doc2query gives the best effectiveness (0.333); TILDE needs many more added terms to get close (m = 200 adds 61.6 tokens on average against doc2query's 19), which also grows the index (6.9 GB against 5.2 GB). This suggests TILDE's individual generated terms are somewhat weaker than doc2query's. The cost column tells the rest of the story:
Collection
doc2query (TPU)
TILDE expansion (GPU)
MS MARCO
320 hours (about 13 days), $768
about 7¼ hours, about $5
MS MARCO v2 (15× larger)
about 200 days, more than $11,000
about 4.5 days, about $80
Key idea
Offline does not mean free. doc2query ran on Google TPUs (about 15× more powerful than GPUs, and expensive); TILDE expansion runs on cheap GPUs. The gap keeps growing at the scale of commercial search engines (Google, Bing, Baidu), where waiting 13 days, or 200, for an index is infeasible.
PracticeWhy does TILDEv2 need document expansion, while the original TILDE works without it?
AnswerTILDE's document vector is dense over the whole vocabulary, so every query token receives some learned weight (implicit expansion). TILDEv2 scores only exact matches between query tokens and the document's own tokens, so a query term absent from the document contributes nothing. Expansion puts likely query terms into the document before encoding, which restores matching for those terms.
Practicedoc2query gives the best MRR@10 in the TILDEv2 expansion table. Give two reasons to choose TILDE expansion anyway.
Answer(1) Cost: about 7 hours and $5 on GPUs against 320 hours and $768 on TPUs for MS MARCO, a gap that grows to about 4.5 days against 200 days on MS MARCO v2. (2) The effectiveness gap is small (0.330 with m = 200 against 0.333), so the saving costs very little quality. The price is a larger index when m is large.
7. SPLADE: expansion and weighting from the MLM head
SPLADE (Formal, Piwowarski & Clinchant, SIGIR 2021) takes the second route: keep the transformer encoder for both queries and documents, and use it to predict term-importance weights over the entire vocabulary. For the query "Who wrote the book of the lord of the rings", a traditional lexical model can only match those exact words. SPLADE activates related vocabulary terms with learned importance weights, for example:
Activated term
write
created
authored
novel
LOTR
Tolkien
Contribution
3.5
1.2
3.3
3.3
5
4
These terms can come from outside the query (Tolkien gets weight 4 purely from semantic association), and yet the vector is still sparse: nearly all of the 30,522 vocabulary dimensions stay at zero. If we can predict the importance of highly related vocabulary terms for their contribution to ranking, we get a much better lexical term-based retriever. The transformer supplies these weights; collection statistics play no part.
7.2 The mechanism: reuse BERT's MLM head
BERT pre-training uses a masked language modelling (MLM) head to predict vocabulary tokens. SPLADE reuses this head at every input position. It obtains vocabulary logits, converts them to non-negative term activations, then pools each vocabulary column into one weight. The lecture uses “probability” as a simple explanation of term importance. Here pij denotes the supplied non-negative activation at input position i for vocabulary term j.
wj=i=1∑npijj=1,…,∣V∣
Where
wj
importance weight of vocabulary term j for this query or document
n
number of token positions in the input text
i
index of one input token position
pij
weight that the MLM head at position i assigns to vocabulary term j
j,∣V∣
index of a vocabulary term, and the vocabulary size (30,522)
Read it aloud
The weight w-j equals the sum, over every input position i from 1 to n, of p-i-j, computed for every vocabulary term j from 1 to the vocabulary size.
Formula in plain words
A vocabulary term becomes important when many positions of the text "point at it", even if the term itself never occurs in the text. Each position adds its weight for the term, so a term supported weakly by many positions or strongly by one position both end up with weight. Longer inputs have more positions and so tend to accumulate larger sums. The result is one vocabulary-sized vector per query or document.
SPLADE: each token position yields an MLM activation vector over the vocabulary; pooling across positions gives the term-importance vector.
Technical detail beyond the lecture shorthandThe original SPLADE paper, Section 3.2 defines the activation before pooling as follows.
pij=log(1+max(0,zij))
Where
zij
raw MLM logit at input position i for vocabulary term j
pij
non-negative activation used in the pooling formulas
max(0,zij)
ReLU: keep a positive logit, otherwise return zero
log
natural logarithm
Read it aloudp-i-j equals the natural log of one plus the maximum of zero and z-i-j.
Formula in plain wordsReLU removes non-positive logits. The logarithm reduces the effect of large positive logits. Zero remains zero. Pool these activations to obtain vocabulary weights; the weights need no normalisation to a sum of one.
7.3 How SPLADE keeps weights sparse
A softmax gives positive probability to every vocabulary term. SPLADE instead applies ReLU to raw logits, then log saturation, before pooling. ReLU makes non-positive logits exactly zero. Sparsity regularisation encourages the model to activate fewer terms; the ranking loss keeps useful matching terms. FLOPS regularisation discourages frequently active dimensions that make posting lists costly. L1 penalises total activation magnitude. The lecture mentions both approaches; their detailed equations are optional reading.
Exam angleExplain the roles separately: the ranking objective learns relevance, ReLU creates zeros, and regularisation encourages efficient sparse representations. Sum or max pooling combines positions. Max pooling alone does not create zeros when every input value is positive.
7.4 SPLADE core ideas, summarised
Every vocabulary term can get a weight, while traditional models weight only terms present in the text.
A BERT-based model predicts term importance given the query or document, reusing all the semantic knowledge from pre-training.
Implicit expansion: weights are free of the original text's terms, so related terms get activated and help ranking.
Sparsity is maintained through ReLU + regularisation, preserving the efficiency of sparse representations.
Worked example 7SPLADE pooling (sum vs max) and scoring
Setup (illustrative numbers). Document: "apple reduces sugar" (3 token positions). Toy vocabulary of 6 terms. Use the following illustrative non-negative activations (these example rows happen to sum to 1):
Position
apple
fruit
reduces
lowers
sugar
car
pos 1 · "apple"
0.50
0.30
0.02
0.02
0.10
0.06
pos 2 · "reduces"
0.05
0.05
0.45
0.30
0.10
0.05
pos 3 · "sugar"
0.10
0.15
0.05
0.05
0.60
0.05
1
Sum pooling (SPLADEv1). Add each column:
applefruitreduceslowerssugarcar=0.50+0.05+0.10=0.65=0.30+0.05+0.15=0.50=0.02+0.45+0.05=0.52=0.02+0.30+0.05=0.37=0.10+0.10+0.60=0.80=0.06+0.05+0.05=0.16
Document vector (sum): [0.65,0.50,0.52,0.37,0.80,0.16]. fruit (0.50) and lowers (0.37) got real weight although neither word is in the document: implicit expansion.
2
Max pooling (SPLADEv2). Take each column's largest value across positions:
applefruitreduceslowerssugarcar=max(0.50,0.05,0.10)=0.50=0.30=0.45=0.30=0.60=0.06
Document vector (max): [0.50,0.30,0.45,0.30,0.60,0.06]. Max pooling keeps the strongest single signal per term and ignores the accumulation of many weak ones: "car", which no position really supports, drops from 0.16 to 0.06.
3
Illustrate optional pruning. For this arithmetic example only, set pooled weights below 0.2 to zero. This threshold is an extra illustration; SPLADE learns sparsity through its activation and regularisation. The sum-pooled vector → [0.65,0.50,0.52,0.37,0.80,0]. "car" is now inactive, so this document never enters the posting list for "car".
4
Encode a query the same way. Query "lowers sugar" passes through the same SPLADE encoder; suppose its sparse vector comes out as [0,0,0.3,0.9,1.1,0] (activating reduces by expansion, plus lowers and sugar).
5
Score with a dot product over shared non-zero terms. Using the sparsified sum-pooled document vector: s(q,d)=0.3×0.52+0.9×0.37+1.1×0.80=0.156+0.333+0.880=1.369. Only 3 of 6 dimensions were touched; with a real vocabulary, only the few dozen active terms matter, which is what makes SPLADE scoring inverted-index fast.
Result
The query word "lowers" is absent from the document and the document word "reduces" is absent from the query, yet the two matched through their expanded, weighted representations: semantic matching carried out entirely inside a sparse, lexical-style dot product.
7.5 TILDE versus SPLADE
The in-class lecture compared the two families side by side (lecture slide 18), and recommended the comparison figure for revision:
Where the weights come from. TILDE takes only the [CLS] vector and projects it with a trained linear layer to vocabulary size. SPLADE keeps all token vectors (as ColBERT did in Week 5), sends each through the pre-trained MLM head, and pools the resulting activation vectors (sum in v1, max in v2). Reusing the MLM head gives SPLADE a better starting point.
Why the vectors are sparse. TILDE is sparse by architecture: the tokenised query vector is sparse by construction, and it has no sparsity loss. SPLADE is sparse by training: both vectors come from the encoder, and the sparsity loss makes them sparse.
Speed. TILDE is often much faster than SPLADE, because its query side needs only the tokeniser. Standard two-sided SPLADE runs a query encoder; the document-only variant changes this.
What they share. Both give every vocabulary term a chance at a weight, both use a BERT-based model to predict term importance, and both score with a dot product that skips the zeros.
TILDE against SPLADE for the query "deep learning" (redrawn from lecture slide 18): TILDE weights come from [CLS] + linear layer, SPLADE weights from every token's MLM activation vector, pooled.PracticeLecture Exercise 1.1: how does SPLADE get its term weights?
Say which representation it takes from BERT, what head it uses, and how it combines them.
AnswerSPLADE runs the query or document through BERT and keeps every token's output vector. Each vector goes through the pre-trained MLM head, which gives logits over the whole vocabulary for that position. ReLU and log saturation turn these into an activation vector. SPLADE then pools these activation vectors into one vocabulary-sized vector: sum pooling in SPLADEv1, max pooling in SPLADEv2. Each dimension's value is that term's importance. ReLU and the FLOPS/L1 sparsity losses applied during training make most of the weights exactly zero.
PracticeWhy is TILDE usually faster than SPLADE at query time, even though both score with a sparse dot product?
AnswerTILDE's query side is only the BERT tokeniser, a lookup table with no neural inference. SPLADE must run the query through the full BERT encoder and MLM head before it can score anything, so every query pays one transformer inference.
SPLADEv2 (same research group) made three main improvements:
Aspect
SPLADEv1
SPLADEv2
Pooling
Sum pooling to aggregate token activation vectors
Max pooling: for each vocabulary dimension, take the highest weight across all input token positions; empirically this captures the most important signal better
Training loss
Basic contrastive training
Adds a distillation loss: the model also learns from a teacher model (beyond the relevance labels), improving weighting and expansion
Weighting & expansion
Both query and document are weighted and expanded, so full SPLADE inference runs at query time
New variant with document-only weighting and expansion (similar in spirit to TILDE), so queries stay simple and close to their original form
Max pooling replaces the sum in the SPLADE weight formula:
wj=i=1,…,nmaxpijj=1,…,∣V∣
Where
wj
importance weight of vocabulary term j
maxi
the largest value over all input positions i=1,…,n
pij
MLM weight of term j at position i
n,∣V∣
number of input positions, and the vocabulary size
Read it aloud
The weight w-j equals the maximum, over all input positions i from 1 to n, of p-i-j, for every vocabulary term j.
Formula in plain words
For each vocabulary term, keep only the strongest vote any single position gave it. A term backed strongly by one position keeps that full weight; a term backed weakly by many positions stays small, because weak votes no longer add up. Weights can exceed 1, and adding more text can only raise a weight when a new position votes more strongly than every earlier one.
Worked example 8 for four of the ten terms: sum pooling stacks both positions' weights; max pooling keeps the larger one.
Reading slides 24–25The two supplied rows sum to 2.2 and 1.9. Treat them as term weights for this pooling exercise. Keep their numbers unchanged. Slide 25 repeats “SPLADEv1” in its opening line, but its title and question ask for SPLADEv2, so use max pooling. Pool each vocabulary column across input positions.
Worked example 8Lecture Exercises 2 and 2.1: SPLADEv1 and SPLADEv2 pooling for “deep learning”
Setup (lecture slides 24–25; a deliberately tiny vocabulary). The whole vocabulary has 10 terms: [deep, apple, learning, neural, networks, fruit, like, machine, map, shallow]. The query is "deep learning". The MLM head predicts one activation vector per query position:
Position
deep
apple
learning
neural
networks
fruit
like
machine
map
shallow
“deep”
0.9
0
0.3
0.5
0.2
0
0
0.2
0
0.1
“learning”
0.1
0.4
0.5
0
0.2
0
0.4
0.3
0
0
1
SPLADEv1: sum pooling. Add the two rows term by term:
deepapplelearningneuralnetworksfruitlikemachinemapshallow=0.9+0.1=1.0=0+0.4=0.4=0.3+0.5=0.8=0.5+0=0.5=0.2+0.2=0.4=0+0=0=0+0.4=0.4=0.2+0.3=0.5=0+0=0=0.1+0=0.1
wv1=[1.0,0.4,0.8,0.5,0.4,0,0.4,0.5,0,0.1]
2
SPLADEv2: max pooling. Take the larger value in each column:
deepapplelearningneuralnetworksfruitlikemachinemapshallow=max(0.9,0.1)=0.9=max(0,0.4)=0.4=max(0.3,0.5)=0.5=max(0.5,0)=0.5=0.2=0=0.4=max(0.2,0.3)=0.3=0=0.1
wv2=[0.9,0.4,0.5,0.5,0.2,0,0.4,0.3,0,0.1]
3
Read the vectors. Both keep neural, networks and machine active although the query never says them: implicit expansion. Terms supported by both positions (learning, networks, machine) gain the most under sum pooling. fruit and map stay at 0 in both, so the query never touches their posting lists.
Result
SPLADEv1 (sum): [1.0, 0.4, 0.8, 0.5, 0.4, 0, 0.4, 0.5, 0, 0.1]. SPLADEv2 (max): [0.9, 0.4, 0.5, 0.5, 0.2, 0, 0.4, 0.3, 0, 0.1]. The lecturer's summary: many vectors come out of the MLM head, and you aggregate them; aggregate with sum and it is v1, aggregate with max and it is v2.
The document-only variant matters because the asymmetric argument from TILDE applies here too. Document expansion happens offline at indexing time, with plenty of time to process thoroughly; query processing happens online while a user waits, so keeping the query side simple means much faster search response times. If most of the semantic-matching benefit comes from the document side anyway, dropping query-side expansion buys efficiency at little cost.
PracticeTwo positions give MLM activation vectors over four terms: p1=[0.6,0.2,0.0,0.4] and p2=[0.1,0.5,0.3,0.4]. Compute the SPLADEv1 and SPLADEv2 vectors.
AnswerSum (v1): [0.7,0.7,0.3,0.8]. Max (v2): [0.6,0.5,0.3,0.4]. The fourth term, supported equally by both positions, is the largest weight under sum pooling and only mid-sized under max pooling.
PracticeWhy does SPLADEv2's document-only variant search faster than SPLADEv1?
AnswerSPLADEv1 weights and expands the query with a full SPLADE inference at query time. The document-only variant does all weighting and expansion offline on documents, so the online query stays close to its original tokens: no expensive query inference, and fewer active query terms to score.
9. Effectiveness comparison of all retrievers so far
All numbers below are on the MS MARCO dev set with MRR@10 (mean reciprocal rank over the top 10, from Week 2). MRR@10 is standard for MS MARCO dev because its relevance judgements are sparse, usually one judged relevant passage per query. Values are the approximate figures narrated in Video 6, with the chart bars used where the narration gives no number.
MRR@10 on MS MARCO dev (approximate). Grey: term-based lexical. Blue: encoder-based dense (single- and multi-vector). Orange: decoder-based dense. Green: learned sparse.
Watch outThe bar chart on slides 41–46 draws RepBERT near 0.30 and ColBERT near 0.36, while Video 6 narrates about 0.33 and 0.39. The ordering is the same either way; quote approximate values and focus on the ranking of the families.
Walking through the chart the way the lecture does:
BM25 (≈ 0.18): the traditional baseline, fast and efficient, limited by exact matching.
Single-vector dense: RepBERT, ANCE (≈ 0.33): nearly double BM25; the improvement that made the field take notice of transformer-based retrieval. Semantic matching does the work.
Multi-vector dense: ColBERT (≈ 0.39): better again, thanks to one vector per token and finer-grained matching; much slower to encode and score than single-vector models.
Decoder-based: RepLLaMA, LLM2Vec (≈ 0.41–0.42): the most recent advances, with backbones of about 7B parameters (against BERT's 110M) and higher-dimensional vectors, so both encoding and scoring cost far more.
Learned sparse: BM25+TILDE ≈ 0.27, BM25+TILDEv2 ≈ 0.33, SPLADEv1 ≈ 0.32, SPLADEv2 ≈ 0.34: a large jump over BM25 (TILDEv2 improves substantially over TILDE), approaching single-vector dense effectiveness. SPLADE lands a little above the TILDE family (it is also a later model), and is less efficient than TILDE because it still runs a query encoder at search time.
Takeaways
Learned sparse models fall a little short of the very best dense retrievers on effectiveness, and they are generally much more efficient: like traditional lexical models they only score documents that share (expanded) terms with the query, giving fast search and low computational requirements. They occupy a sweet spot: significantly better effectiveness than BM25 while keeping much of the efficiency that makes lexical models practical at large scale.
PracticeLLM2Vec scores about 0.42 MRR@10 and SPLADEv2 about 0.34. Give two reasons a production search team might still choose SPLADEv2.
Answer(1) Efficiency: SPLADEv2 scores through an inverted index and only touches documents sharing active terms, while LLM2Vec compares high-dimensional dense vectors. (2) Model size: SPLADE uses a BERT-sized encoder (about 110M parameters) against a 7B-parameter decoder, so encoding the collection and every query costs far less GPU time and memory. With the document-only variant, query cost drops further.
10. PromptReps: training-free dense + sparse representations from LLMs
The LLM-based dense retrievers from Week 5 all need training to become retrievers:
E5(-Mistral): takes the embedding from the last hidden layer of an [EOS] token appended to the query/document; requires (supervised) contrastive fine-tuning (it can also use LLM-generated synthetic data).
LLM2Vec: takes the embedding from mean pooling over all sequence tokens' last hidden states; requires (unsupervised or supervised) contrastive fine-tuning.
Contrastive fine-tuning is expensive twice over: you need labels (or synthetic data), and you need serious GPU infrastructure; the larger the decoder backbone, the longer the training and the more memory required. The question PromptReps (Zhuang et al., EMNLP 2024) asks: can we engineer prompts so the LLM produces an effective retrieval representation with no contrastive fine-tuning at all? This is "zero-shot" generation of representations: no training beyond the LLM's own pre-training, and no in-context examples either.
10.2 How it works
Prompt the LLM to represent the given text with a single word (queries use a slightly different prompt, "represent the query"):
<System> You are an AI assistant that can understand human language. <User> Passage: "[text]". Use one word to represent the passage in a retrieval task. Make sure your word is in lowercase. <Assistant> The word is: "
Run one inference and harvest two representations from the position of that final quote character, the position where the model is about to generate its one summarising word:
Dense: the last hidden state at that position (it encodes the word the model is about to say), used for dense retrieval via an ANN index. The lecture noted this is the same kind of embedding RepLLaMA uses.
Sparse: continue through the LM head to get the next-token logits, raw prediction scores over the LLM's entire vocabulary, and keep the top-k predicted terms as a sparse vector, used for classic inverted-index matching.
One inference, two representations: the hidden state gives a dense vector; the next-token logits give a sparse vector over the LLM vocabulary. Their scores are combined linearly.
At retrieval time, the dense representation performs dense retrieval and the sparse representation performs typical inverted-index matching; each gives a score, and the final relevance score combines them linearly (an interpolation). The result is a hybrid retriever built from one encoder. The lecture stressed that PromptReps can run as a pure sparse model, a pure dense model, or the hybrid; it works zero-shot, and it becomes much better still if you do train it with a standard contrastive loss. A later model in Week 9, built on a text-diffusion model, follows the same idea with multi-vector representations.
The two scores live on different scales (cosine similarity lies in [−1, 1]; a sparse dot product is unbounded), so a common recipe first min–max normalises each score list over the candidates and then interpolates:
the smallest and largest raw score of that kind in the candidate list
s~
the normalised score, between 0 and 1
s~dense,s~sparse
normalised dense (hidden-state) and sparse (logits) scores
λ
interpolation weight in [0,1] given to the dense score
shybrid
final hybrid relevance score
Read it aloud
s-tilde equals s minus s-min, divided by s-max minus s-min. The hybrid score equals lambda times the normalised dense score plus one minus lambda times the normalised sparse score.
Formula in plain words
First rescale each score list so its smallest value becomes 0 and its largest becomes 1 (subtract the minimum, then divide by the range); this puts the two signals on the same footing. Then take a weighted sum: a fraction λ of the dense score plus the remaining fraction of the sparse score. λ=1 is pure dense retrieval, λ=0 pure sparse; values in between reward documents that do well on both signals. The slides specify only "combine linearly"; min–max normalisation is an illustrative choice. If all scores in a list are equal, set their normalised contribution to zero to avoid dividing by zero.
Worked-example conventionThe lecture specifies linear combination. The min–max scaling here is an illustrative choice. When a score list has equal minimum and maximum, give that list a zero normalised contribution to avoid division by zero. Check effectiveness with relevance judgements after choosing a fusion rule.
10.3 Does zero-shot retrieval work?
On MS MARCO (MRR@10, shown ×100 on the slide), with everything zero-shot (no in-context examples, no contrastive training of the representations):
System
MRR@10 (approx.)
Notes
LLM2Vec (unsupervised, SimCSE-trained baseline)
≈ 13.5
still needs unsupervised contrastive training
PromptReps (Dense only)
≈ 17.5
below BM25
BM25 (dotted reference line)
≈ 18.5
also training-free
PromptReps (Sparse only)
≈ 21
above BM25
PromptReps (Hybrid)
≈ 24.5
combining dense + sparse is the key
Key ideaCombining the dense and sparse representations allows very effective zero-shot retrieval. The hybrid clearly beats BM25 and the unsupervised LLM2Vec baseline, with no training of the representations at all.
Extension: more than one word. The prompt can ask for two words, three words, a sentence, a paragraph… which requires multiple autoregressive inferences and yields multiple token embeddings/distributions. These can be aggregated (max pooling over the sparse distributions, mean pooling over the dense vectors) or even used ColBERT-style with MaxSim between query-side and passage-side multi-token representations. The lecture stops there, because these multi-token paragraph/passage representations turn out to be weak.
Worked example 9Combining dense and sparse scores in a hybrid retriever
Setup (illustrative numbers). A query is issued to a PromptReps system. Three candidate passages receive a dense score (cosine similarity from the ANN index) and a sparse score (inverted-index match on the logits vectors, illustrative scale). Use the normalisation and interpolation above with λ=0.5:
Passage
Dense score sdense
Sparse score ssparse
d1
0.82
6.0
d2
0.78
12.0
d3
0.70
9.0
1
Normalise dense scores. Min = 0.70, max = 0.82, range = 0.12.
d1:0.120.82−0.70d2:0.120.78−0.70d3:0.120.70−0.70=1.00=0.667=0.00
2
Normalise sparse scores. Min = 6.0, max = 12.0, range = 6.0.
d1:66−6d2:612−6d3:69−6=0.00=1.00=0.50
3
Interpolate with λ=0.5.d1:0.5×1.00+0.5×0.00d2:0.5×0.667+0.5×1.00d3:0.5×0.00+0.5×0.50=0.50=0.83=0.25
4
Rank.d2≻d1≻d3. Neither signal alone gives this ranking: dense alone would put d1 first, sparse alone would put d1 last. The hybrid rewards d2 for being strong on both signals, illustrating how fusion can change the order. The measured effectiveness comparison is the separate MS MARCO result above.
Result
Interpolation produces a combined ranking. Relevance judgements are needed to measure whether this ranking improves effectiveness; these illustrative scores demonstrate the calculation only.
PracticeLecture Exercise 1.2: what makes PromptReps a "hybrid" approach?
Say where the dense part and the sparse part come from.
AnswerOne prompted LLM inference gives two representations. The dense part is the last-layer hidden state at the final prompt token, used like a RepLLaMA embedding in an ANN index. The sparse (lexical) part is the next-token scores over the vocabulary: because the prompt asks for one word that represents the text, the top-k predicted terms capture its meaning, and they go into an inverted index. The dense and sparse scores are then combined, so a single encoder yields a hybrid retriever.
PracticeRepeat Worked example 9 with λ=0.8. Does the ranking change?
Answerd1:0.8×1.00+0.2×0.00d2:0.8×0.667+0.2×1.00d3:0.8×0.00+0.2×0.50=0.80=0.733=0.10
The ranking becomes d1≻d2≻d3: with most weight on the dense signal, d1's top dense score wins.
11. Summary and study checklist
Model
Query side (online)
Document side (offline)
Expansion
Training signal
Online neural inference?
BM25 + doc2query
Plain terms
Text + generated queries in a normal inverted index
Contextualised token weights of the expanded document
Explicit, via doc2query or TILDE (cheap)
Contextualised exact-match training
No
SPLADE
Full encoder → sparse weighted vector
Full encoder → sparse weighted vector (sum pooling)
Implicit, via MLM head over vocabulary
Contrastive + ReLU/FLOPS/L1 for sparsity
Yes (query encoder)
SPLADEv2 (doc-only variant)
Simple, near-original query
Full encoder → sparse weighted vector (max pooling)
Document-only
+ distillation from a teacher
No for the document-only variant
PromptReps
One LLM inference each → dense (hidden state) + sparse (top-k logits)
Via the LLM's summarising word
None (zero-shot); optional contrastive training helps further
Yes (LLM inference)
Exam angleThe lecturer emphasised explaining each model from its architecture. Practise these four tasks: explain TILDE's three efficiency gains; trace SPLADE from token vectors to vocabulary weights; identify both PromptReps outputs; calculate sum and max pooling column by column. For BiQDL, explain which token probabilities each direction raises or lowers and how ranking choices affect online work (lecture slides 10–14, 21–25).
You should now be able to:
Place learned sparse retrievers in the relevance-model landscape and explain the efficiency/effectiveness trade-off they target.
Explain why sparsity gives efficiency, using the sparse dot product.
Explain query/document expansion (including the Lord of the Rings example) and describe the doc2query pipeline, including why copied words help as well as new words.
Draw the TILDE architecture, give the three reasons for its query-time efficiency, explain (in words) the BiQDL loss, and describe the three ranking modes, their costs and the role of α.
Explain what TILDEv2 changes (token embeddings, exact matching, expansion) and argue the cost case for TILDE expansion over doc2query.
Explain how SPLADE turns MLM-head predictions into term weights, why sparsity is at risk, and how ReLU + FLOPS/L1 regularisation restore it.
Compute SPLADEv1 (sum) and SPLADEv2 (max) pooled vectors by hand, and list the other SPLADEv1 → v2 changes.
Contrast TILDE and SPLADE: [CLS] + linear against all tokens + MLM head; sparse by architecture against sparse by training; relative speed.
Compare all retriever families on MS MARCO and state where learned sparse models sit.
Describe PromptReps: the prompt, where the dense and sparse representations come from, and why the hybrid works zero-shot.
Reproduce every worked example with different numbers.
Looking ahead. Week 7 moves from retrieval models to reranking with learning to rank: traditional methods, often applied on top of term-based lexical models, that underpin modern reranking and still sit on top of neural pipelines at search companies. The lecture said the week after turns to reranking with large language models. (At the time of the lecture, Assignment 1 was due on 4 September at 4pm.)
Slide decks: pre-class deck (51 slides, cited as "slides n") and in-class lecture deck (27 slides, cited as "lecture slides n"). The in-class lecture transcript (w6-lec-INFS7410_S2_2026_STL-transcript.txt) is cited as "Lecture transcript"; the pre-class video transcripts are in Pre-class youtube Transcripts/.
13. References
Zhuang, S. and Zuccon, G. (2021). TILDE: Term Independent Likelihood moDEl for passage re-ranking. SIGIR 2021.
Zhuang, S. and Zuccon, G. (2021). Fast passage re-ranking with contextualized exact term matching and efficient passage expansion. arXiv:2108.08513 (TILDEv2).
Formal, T., Piwowarski, B. and Clinchant, S. (2021). SPLADE: Sparse lexical and expansion model for first stage ranking. SIGIR 2021, pp. 2288–2292.
Formal, T., Lassance, C., Piwowarski, B. and Clinchant, S. (2021). SPLADE v2: Sparse lexical and expansion model for information retrieval. arXiv:2109.10086.
Zhuang, S., Ma, X., Koopman, B., Lin, J. and Zuccon, G. (2024). PromptReps: Prompting large language models to generate dense and sparse representations for zero-shot document retrieval. EMNLP 2024.