INFS7410 · Information Retrieval and Web Search · Week 6 · Simplified Technical English edition
Learned Sparse Retrievers: TILDE, TILDEv2, SPLADE and PromptReps
This week covers lexical retrieval models with learned term weights. These models keep the speed of inverted-index search and add the semantic power of transformers. The notes cover query and document expansion (doc2query), the TILDE family, the SPLADE family, a comparison of all retrievers so far, PromptReps (which needs no training), and the exercises from the in-class lecture.
The in-class lecture starts with a review of the ranking models so far. Week 1 defines information retrieval and text processing (tokenisation, stemming, stopwords). Week 2 covers evaluation: set-based and rank-based measures, pooling, and statistical tests. Week 3 covers the term-based lexical models: the Boolean model, TF-IDF, the vector space model, BIM and BM25. Week 4 explains the transformer: encoder, decoder, BERT and GPT. Week 5 uses transformers as retrievers: DPR, ANCE, RepBERT, ColBERT, E5-Mistral, LLM2Vec and RepLLaMA. Week 6 covers learned sparse models (TILDE, TILDEv2, SPLADE, SPLADEv2) and PromptReps.
The lecture timeline puts learned sparse models next to dense retrievers, in the period 2018–2022. Term-based lexical models come first (1950s–1990s), then the transformer (2017–2018). The lecturer's point: a learned sparse model mixes the Week 4 transformer with the ideas of the Week 3 lexical models.
In the past weeks we built a table of relevance model families. Term-based lexical models such as BM25 (Week 3) match exact words with an inverted index (Week 1). Single-vector dense retrievers such as RepBERT and ANCE (Week 5) encode a full query or document into one embedding. Multi-vector dense retrievers such as ColBERT keep one embedding for each token. This week fills the last cell of the table: learned sparse retrievers. Another name for them is lexical retrieval models with learned term weights. They sit between lexical models and dense models. They try to get the best parts of both.
Family
Encoding
Online scoring
Stored representation
BM25
Tokenisation and term statistics
Traverse posting lists for query terms
Sparse term weights
TILDE-QL
BERT on documents offline; tokeniser on queries online
Learn document weights offline; standard SPLADE also encodes queries online
Use shared active vocabulary terms
Sparse document weights
Single-vector dense
Transformer on each text; document encoding offline
One vector comparison per candidate; ANN can select candidates
One dense vector per document
Multi-vector dense
A similar transformer backbone; retain token vectors
Token-to-token comparisons and MaxSim aggregation
Several dense vectors per document
Reading the lecture's cost diagram. Separate encoding a text, scoring one candidate, and searching a collection. The lecturer says that single-vector and multi-vector models can use the same encoder backbone. Keeping token vectors increases storage and scoring work. Full transformer self-attention considers token pairs in either family. The slide's “O(<1)” label is a qualitative efficiency cue. Actual search cost depends on posting-list lengths, active terms, candidate counts, and the index. Dense retrieval can use approximate nearest-neighbour (ANN) search to avoid an exhaustive scan.
The positions are qualitative. Learned sparse retrievers keep the low scoring cost of lexical models. Their effectiveness moves up towards dense retrievers.PracticeWhy does ColBERT scoring cost more than RepBERT scoring? Which column of the table do learned sparse retrievers make better?
Count the vectors that each model keeps for a query and for a document.
AnswerRepBERT keeps one vector for each query and each document. Thus the score of one document is one dot product. ColBERT keeps one vector for each token. It compares every query token vector with every document token vector (MaxSim), so the work increases. Learned sparse retrievers make the scorer column better. Their sparse vectors let the engine score only the documents that share active terms with the query, as BM25 does.
2. Review: lexical retrievers and dense retrievers
2.1 Term-based lexical models give high efficiency
Take the query "machine learning algorithms". A system such as BM25 looks only at documents that contain the exact words machine, learning, or algorithms. The inverted index stores, for each term, a list of the documents that contain it. At query time the engine walks only the lists for the query terms. This candidate set is usually a very small part of the collection. This is the source of the high efficiency of lexical models. The lecture also gives a practical fact: BM25 runs on a CPU in a few milliseconds for each query. An Assignment 1 BM25 run takes minutes. If it takes hours, the code has a bug.
Lexical retrieval examines only the small parts of the corpus that contain query terms.
2.2 Dense retrievers give high effectiveness
Dense retrievers (Week 5) send queries and documents through transformer encoders. The encoders make dense vectors. The retriever ranks by vector similarity: cosine similarity, dot product, or Euclidean distance. Dense retrievers are much more effective because they do semantic matching. They know that machine learning is related to artificial intelligence, neural networks and transformers, also when no word is the same. The cost is a heavy encoder. Also, scoring must compare the query with every document vector in the collection.
Key idea
Lexical models are very efficient, but their effectiveness is limited (exact matching only). Dense models are very effective (semantic matching), but they cost much computation. The central question of Week 6 is: can we combine the advantages of both?
PracticeA user searches for "AI tutoring". A relevant document uses only the words "machine learning tutors". Which retriever family ranks it well? What does the other family pay for that result?
AnswerBM25 matches only "tutoring" (if stemming changes it to "tutor"). It does not see the link between "AI" and "machine learning", so it gives the document a low rank. A dense retriever puts the two texts near each other in the embedding space and gives the document a high rank. The dense retriever pays with a transformer encoding of the query at search time. It also compares the query with every document vector.
Traditional term-based lexical retrievers have three limits. First, their weights come only from term overlap and collection statistics (term frequency, document frequency, document length). Second, they use hand-made formulas. Researchers designed these formulas from intuition and some theory. Third, they learn nothing from data. The same formula applies to every query, domain and document type. Only a small number of parameters can change.
Learned sparse models use a different method:
They learn the best term weights from training data. Thus the model adapts to its content, queries and domain.
They use a strong transformer to predict term importance. This gives much higher effectiveness than simple statistics.
They keep sparsity for efficiency. Scoring can still use the inverted-index method: it touches only documents that share query terms (or expanded query terms).
This week shows two main routes to this goal. It also shows a recent hybrid method that uses an LLM:
Route
Idea in one line
Models
Remove the query encoder
Queries and documents get different treatment (asymmetric). The tokeniser changes the query into a sparse term-existence vector, with no neural network at query time. BERT still encodes the documents offline into dense vectors of vocabulary size. Document encoding time stays the same. Query encoding time almost goes to zero. The sparse query vector also decreases the scoring time.
TILDE, TILDEv2
Keep the encoder, output term weights
The transformer encoder processes queries and documents. It gives weights over the full vocabulary (an "expanded set of terms"). Encoding time is the same as for a dense model. Scoring is cheap because the vectors are sparse.
SPLADE, SPLADEv2
Prompt an LLM, with no training
A prompt tells a generative LLM to summarise a text in one word. One inference gives a dense vector (hidden state) and a sparse vector (next-token logits).
PromptReps
The two main routes. TILDE removes the query encoder (asymmetric). SPLADE keeps both encoders and makes their outputs sparse.
3.1 Why sparsity is the efficiency
Both routes end with a dot product in the ∣V∣-dimensional vocabulary space. The full formula shows the source of the saving:
s(q,d)=i=1∑∣V∣qidi=i:qi=0∑qidi
Where
s(q,d)
relevance score of document d for query q
∣V∣
vocabulary size (30,522 for the BERT vocabulary)
i
index of one vocabulary term (one dimension)
qi
weight of term i in the query vector
di
weight of term i in the document vector
i:qi=0
only the dimensions where the query weight is not zero
Read it aloud
The score of document d for query q is the sum, over every vocabulary term i, of the query weight times the document weight. This is equal to the same sum over only the terms where the query weight is not zero.
Formula in plain words
A dimension with a query weight of zero adds zero to the score. The document value in that dimension has no effect, so the engine skips it. Thus the work for each document depends only on the number of active query terms, which is small. If most weights are exactly zero, the engine can also build posting lists for the non-zero terms. Then it scores only the documents that share active terms with the query, as BM25 does. A query with more active terms (for example after expansion) costs more to score.
Key idea
Sparsity is the efficiency. The speed of a learned sparse model depends on the sparsity of its vectors. If all weights are not zero, the dot product costs as much as a dense dot product with 30,522 dimensions.
PracticeA sparse query vector has 4 non-zero values in 30,522 dimensions. How many multiplications does the score of one document need? How many does a full dense vector with 30,522 dimensions need?
Answer4 multiplications (one for each non-zero query value, plus 3 additions). The dense case needs 30,522. The zero values add nothing, so the engine does not calculate them.
PracticeGive the three limits of BM25-style retrievers that learned sparse models solve. Give the property that they keep from BM25.
Answer(1) The weights come only from term overlap and collection statistics. (2) The formulas are hand-made. (3) The model learns nothing from data. Learned sparse models use a transformer to learn term weights from training data. They keep the sparsity of BM25, and thus the efficiency of the inverted index.
Query and document expansion adds more relevant terms to the user query or to a document in the corpus. The goal is higher effectiveness: the retriever can then match related terms that are not in the original text. Expansion attacks the vocabulary mismatch problem directly. For example, a query says "lower blood sugar" and the document says "reduces blood sugar levels". There are automatic and semi-automatic expansion methods. Semi-automatic methods need the user to select the best expansion terms. Automatic methods need no user input. Query or document rewriting is a related method. Many learned sparse retrievers use expansion to get better effectiveness, so these notes explain it first. The lecturer says that Week 12 covers expansion in more detail.
The in-class lecture uses this example. The original query is "who wrote the book of the lord of the rings". Expansion adds terms that show the meaning of the question: write, created, authored, novel, LOTR, Tolkien. The expanded terms are new words, because a repeat of a query word adds nothing. The expanded query is the original query plus these terms. The same idea applies to documents. Expansion can add "machine" and "learning" to a document about "AI". Then the document matches a query that uses those words.
Expansion adds related terms. Then relevant documents with different words still share terms with the query (lecture slide 7).
4.2 doc2query: document expansion with a seq2seq transformer
Document expansion adds terms (or changes term weights) in the index to improve performance. doc2query uses a sequence-to-sequence transformer for this task. This transformer has an encoder and a decoder, and it changes input text into output text (T5 is a common example). BERT is different: it changes input text into embeddings. The task of the model is this: for a given document, generate a query that the document answers. Training is supervised. It uses pairs of ⟨query, relevant document⟩, for example from MS MARCO.
The doc2query pipeline. Generate 5–40 queries for each document. Join them to the document. Index the expanded documents as usual.
In practice, the model samples 5 to 40 queries for each document (with top-k or nucleus sampling). The system joins each document with its predicted queries. The result is an expanded document. A traditional inverted index stores the expanded documents. When a user searches, the engine (for example BM25) matches against the expanded index. This closes much of the vocabulary gap between query and document.
4.3 Does it work?
Yes. Expansion always gives better results than the same ranking model on documents without expansion. This is true for all tested rankers: a bag-of-words model, BM25 with RM3 pseudo-relevance feedback, and BM25 followed by a monoBERT/T5 reranker.
Ranker
MS MARCO Passage (MRR@10)
TREC-DL 19 (nDCG@10)
BM25
0.184
0.506
BM25 + doc2query
0.277
0.642
BM25 + RM3 (pseudo-relevance feedback)
0.156
0.518
BM25 + RM3 + doc2query
0.214
0.655
BM25 + monoBERT/T5 reranking
0.365
0.738
BM25 + monoBERT/T5 + doc2query
0.379
0.754
The generated queries show that doc2query does two useful things. It adds new words to the document, and this attacks keyword mismatch directly. It also copies keywords that are already in the document. Without stop words, about 69% of the generated words are copies and 31% are new. Copies are important too: they increase the importance (term frequency) of those keywords in the expanded document. The slide gives three examples:
Input document (short form)
Predicted query
Target query
New words
"July is the hottest month in Washington DC with an average temperature of 27C …"
weather in washington dc
what is the temperature in washington
weather
"The Delaware River flows through Philadelphia into the Delaware Bay …"
what river flows through delaware
where does the delaware river start and end
none (all copied)
"sex chromosome … a threadlike strand of DNA in the cell nucleus that carries the genes …"
what is the relationship between genes and chromosomes
which chromosome controls sex characteristics
relationship
Ablation (MS MARCO passage)
MRR@10
R@1000
Original document
0.184
0.853
+ expansion with new words only
0.195
0.907
+ expansion with copied words only
0.221
0.893
+ expansion copied + new
0.277
0.944
Only expansions (original document removed)
0.263
0.927
Takeaways
Both types of expansion help. Copied words help more than new words (0.221 against 0.195 MRR@10). Also, an index of only the generated queries, without the original document (0.263), is better than an index of only the original documents (0.184).
Worked example 1How doc2query expansion changes a BM25 score
Setup (illustrative numbers). The user query is q="lower blood sugar". The passage d is "Researchers are finding that cinnamon reduces blood sugar levels naturally when taken daily". It has 13 terms, so ∣d∣=13. The collection has N=1,000,000 passages with average length avgdl=15. The BM25 parameters are k1=1.2 and b=0.75. The document frequencies are df(lower)=40,000, df(blood)=25,000, df(sugar)=20,000. We use the BM25 form from Week 3:
Formula in plain words
The score is the sum over all query terms of two factors. The IDF factor is the log of "how rare the term is in the collection", so rare terms count more. The term-frequency factor increases when the term occurs more often in the document, but each new occurrence adds less. It also changes for documents that are longer or shorter than average. tf(t,d) is the count of t in d. df(t) is the number of documents that contain t. N is the collection size. k1 controls saturation and b controls length normalisation.
1
Score the original document. The term frequencies in d are tf(lower)=0, tf(blood)=1, tf(sugar)=1. The passage says "reduces", so "lower" does not occur. Its contribution is 0. This is vocabulary mismatch.
2
Calculate the shared parts. Length normalisation:
1−b+bavgdl∣d∣=1−0.75+0.75×1513=0.25+0.65=0.90
Thus the TF factor for a term with tf=1 is
1+1.2×0.901×2.2=2.082.2=1.058
3
Calculate the IDFs.idf(blood)idf(sugar)idf(lower)=ln(25,000+0.51,000,000−25,000+0.5+1)=ln(40.0)=3.689=ln(20,000.5980,000+1)=ln(50.0)=3.912=ln(40,000.5960,000+1)=ln(25.0)=3.219
We do not use the value for "lower" yet, because tf=0.
4
Original score.BM25=3.689×1.058+3.912×1.058=3.903+4.139=8.04. The term "lower" adds nothing.
5
Expand the document. doc2query predicts the query "does cinnamon lower blood sugar" (5 terms). We add it to the end of the document. The expanded document d′ has ∣d′∣=18 terms and new counts: tf(lower)=1, tf(blood)=2, tf(sugar)=2. Both effects from the lecture occur. Expansion adds a new word ("lower"). It also copies existing words ("blood", "sugar") and increases their term frequency.
6
Calculate the parts again for d′.length normalisation:0.25+0.75×1518constant in the denominator:k1×1.15TF factor, tf=1:1+1.382.2TF factor, tf=2:2+1.382×2.2=0.25+0.90=1.15=1.38=0.924=3.384.4=1.302
This is larger than for tf=1, but less than two times larger. The BM25 term frequency saturates.
Result
Expansion increases the score of the document from 8.04 to 12.87. More important, the query term "lower" now matches. Query time does not change: the expansion work occurs one time, offline, at indexing. This mechanism causes the MRR@10 increase from 0.184 to 0.277 for BM25 on MS MARCO.
PracticeThe ablation shows that expansion with only copied words improves MRR@10. How can words that are already in the document help BM25?
Find the BM25 input that the copies change.
AnswerCopies increase the term frequency of the words that the generated queries find important. Thus BM25 gives those words a larger TF factor. The document representation then puts weight on the terms that users actually search for (0.184 → 0.221 MRR@10 with copied words only).
PracticeWhy does doc2query add no cost at query time? Where does its cost go?
AnswerThe seq2seq model runs one time for each document, at indexing time. The expanded text goes into a normal inverted index, and plain BM25 processes the queries. The cost moves offline: generation for every document, and a larger index. Section 6 shows that this offline cost can still be very large.
5.1 The idea: replace the query encoder with a tokeniser
TILDE (Zhuang and Zuccon, SIGIR 2021) uses the first route from Section 3: it removes query encoding fully. A dense bi-encoder (Week 5) runs BERT on both sides. TILDE keeps BERT on the document side. On the query side it uses only the BERT tokeniser. The tokeniser is the first step of the Week 4 input pipeline, before the embedding and encoder layers.
Query: the tokeniser changes the query into a sparse vector of size ∣V∣=30,522 (the BERT vocabulary). Element i is 1 if token ti occurs in the query, and 0 if not. This is a term-existence vector. It is like exact matching over the BERT vocabulary, in place of the collection vocabulary.
Document: BERT encodes the document. A trained linear layer projects the [CLS] token embedding (768 dimensions) to a dense vector of size ∣V∣=30,522. Each dimension holds the learned importance (probability) of that vocabulary term for the document. The system calculates this offline, at indexing time.
Score: the inner product of the sparse query vector and the dense document vector, in the ∣V∣-dimensional space.
s(q,d)=q⊤d=i=1∑∣V∣qidi=ti∈q∑di
Where
q
binary query vector of length ∣V∣, from the BERT tokeniser
d
dense document vector of length ∣V∣: the [CLS] embedding after the linear layer
qi
1 if vocabulary token ti occurs in the query, 0 if not
di
learned weight (probability) of token ti for the document
ti∈q
the vocabulary tokens that occur in the query
∣V∣
BERT vocabulary size, 30,522
Read it aloud
The score s of document d for query q is the transpose of q times d. This is the sum over all vocabulary positions i of q-i times d-i. This is equal to the sum of d-i over the tokens t-i that occur in the query.
Formula in plain words
Query values are only 0 or 1. Thus the inner product becomes one simple operation: add the learned document weights at the positions of the query tokens. A query with 5 tokens costs 5 look-ups and additions for each document. A document gets a higher score when BERT gave high weights to the query tokens. This is also true for tokens that do not occur in its text. Each new query token adds one more term to the sum. The query side has no weights, so all the learning is in the document vector.
TILDE. On the query side, a tokeniser makes a binary sparse vector. On the document side, a linear layer projects the BERT [CLS] embedding to vocabulary size.
Takeaways(1) Semantic matching stays. A query keyword can be absent from the document. The dense document vector, which BERT made, can still give that keyword a high weight. The model has learned which vocabulary terms describe the document. (2) No GPU at query time. The tokeniser is a small lookup table. Query "encoding" is almost free and runs well on a CPU.
Worked example 2Scoring with TILDE's sparse × dense inner product
Setup (illustrative numbers). To keep the numbers easy to read, use a small vocabulary of 8 BERT tokens in place of 30,522: V={apple,store,account,fruit,buy,banana,login,the}. The user query is "apple account". BERT encoded two candidate passages offline. After the linear projection, their [CLS] embeddings give these dense weights (each row is Pθ(ti∣d)):
Passage
apple
store
account
fruit
buy
banana
login
the
d1: "how to sign in to your Apple ID"
0.62
0.28
0.51
0.04
0.19
0.02
0.44
0.30
d2: "apples are a healthy fruit to snack on"
0.66
0.07
0.03
0.58
0.21
0.35
0.01
0.31
1
Tokenise the query (no GPU). "apple account" sets the dimensions apple and account: q=[1,0,1,0,0,0,0,0]. All other dimensions are 0.
2
Inner product with d1. Only the two active dimensions have an effect: s(q,d1)=1×0.62+1×0.51=1.13.
3
Inner product with d2.s(q,d2)=1×0.66+1×0.03=0.69.
4
Rank.d1≻d2. This shows semantic matching. d1 does not contain the word "account". But BERT learned that a passage about "sign in to your Apple ID" is about accounts. Thus its dense vector has a high weight (0.51) on the account dimension. A lexical matcher does not find this. d1 also has a weight of 0.44 on "login", another term that is not in the passage. This is implicit expansion in the document representation.
5
TILDE-QL with log weights. Use log-likelihoods in place of raw probabilities:
TILDE-QL(q∣d1)TILDE-QL(q∣d2)=ln0.62+ln0.51=−0.478−0.673=−1.151=ln0.66+ln0.03=−0.416−3.507=−3.923
Because −1.151>−3.923, the ranking is the same: d1 is first.
Result
The score of each document costs two weight lookups and one addition, after a tokeniser lookup. Thus TILDE has a very short query latency and needs no GPU. It still gives a high score to a document that matches the query only by meaning.
Common trapThe lecture architecture illustrates a sum of term weights. For the TILDE-QL ranking rule in Section 5.4, store log-probabilities as those weights. Adding raw probabilities and adding log-probabilities can produce different rankings. Worked example 2 happens to give the same order with both. Each TILDE output is an independent token probability, so its vocabulary entries can sum to more than one.
5.2 Why TILDE is fast at query time
In Exercise 1 of the lecture (lecture slide 21), the class gave three different reasons why TILDE is more efficient than dense retrievers:
No query encoding. At query time, the system only tokenises the query. The query does not go into the BERT model. Tokenisation is the step before the input embeddings, so it is very fast.
Document encoding occurs offline. Documents still go through BERT, but only one time, at indexing time. If the collection does not change, the index does not change. The number of queries has no effect on this cost.
The query vector is sparse. Only the query tokens have the value 1. All other vocabulary values are 0. A query almost always has far fewer tokens than the vocabulary, so the dot product uses only a small number of dimensions. For each candidate, score only the active query-token dimensions.
Neural work at query time for K candidate passages. TILDE-QL needs no BERT inference. The TILDE-DL signal (Section 5.4) adds one query inference again.
The words "term independent" in the name of TILDE explain why the offline work is possible. TILDE assumes that query tokens are independent of each other. Thus the likelihood of each vocabulary term depends only on the passage: Pθ(t∣d). The model can predict the likelihood of every term for a passage before a query exists. It stores the results and answers a query with look-ups. In a term-dependent model, the probability of each query term also depends on the earlier query terms. That model must run its neural inference after the query arrives.
Term independence changes online neural inference into offline calculation followed by look-ups at query time.
Worked example 3Query-time operation count for 1,000 candidates
Setup (illustrative numbers). A query with three tokens must get scores against 1,000 candidate passages. TILDE-QL has the passage weights in the index. A dense retriever scores the same candidates exactly with 768-dimensional vectors.
1
TILDE-QL. Tokenise the query (no BERT). For each candidate, look up and add 3 stored values: 1,000×3=3,000 look-ups and additions.
2
Dense scoring. Run BERT one time on the query. Then calculate one 768-dimensional dot product for each candidate: 1,000×768=768,000 multiplications.
3
Compare.3,000768,000=256
The dense scorer processes 256 times more values. It also needs a BERT query encoding, which TILDE-QL does not need.
Result
TILDE-QL: 3,000 cheap operations and no neural inference. Dense: 768,000 multiplications and one BERT inference. (This count is for exact candidate scoring. An ANN index changes the dense cost.)
Worked example 3 on a log scale. The dense scorer processes 256 times more values than TILDE-QL.
Watch out
TILDE-QL has a sparse query and a dense document vector. Each candidate can have a weight for every query token, including tokens absent from its text. For 1,000 candidates and 3 distinct query tokens, scoring reads 3,000 stored weights. Scanning all 30,522 dimensions would read 30,522,000 weights. The saving is in dimensions per candidate. A first-stage retriever can supply the candidate set. Document precomputation also helps dense bi-encoders; TILDE-QL additionally removes online query inference.
Query sparsity reduces work within each candidate. Candidate selection is a separate step.
5.3 The training loss: BiQDL
The loss is another important difference from other bi-encoders. Week 5 bi-encoders use a margin loss or a contrastive loss. TILDE uses the traditional query likelihood method from statistical language models. It assumes that query tokens are independent of each other. Query likelihood models make the same assumption. The bi-directional query-document likelihood loss (BiQDL) has two directions. Training uses relevant ⟨query, document⟩ pairs.
Direction 1: query likelihood (QL). The model reads the document text. It must predict which vocabulary tokens occur in the query:
LQL(D)=−(q,d)∈D∑∣V∣1i∑∣V∣[ylog(Pθ(ti∣d))+(1−y)log(1−Pθ(ti∣d))]y={1,0,if vocabulary term ti appears in query q,otherwise.
Where
LQL(D)
query-likelihood loss over the training set D
D
training set of relevant ⟨query, document⟩ pairs
(q,d)
one relevant query–document pair
∣V∣
vocabulary size; ∣V∣1 gives the average over all vocabulary tokens
ti
vocabulary token number i
y
label: 1 if ti is in the query, 0 if not (yi on the lecture slides)
Pθ(ti∣d)
the probability (importance) that the model gives to token ti for the document: entry i of the document vector
θ
the model parameters (BERT and the linear layer) that training changes
Read it aloud
The query-likelihood loss over D is minus the sum over every query–document pair in D of one over the vocabulary size times the sum over every vocabulary token t-i of: y times the log of the probability of t-i given d, plus one minus y times the log of one minus that probability. Here y is 1 if t-i occurs in the query, and 0 if not.
Formula in plain words
For each training pair, calculate a binary cross-entropy for each vocabulary token and take the average over all ∣V∣ tokens. If the token is in the query (y=1), only the logPθ(ti∣d) part is active. A higher probability moves that log nearer to zero and decreases the loss. If the token is not in the query (y=0), only the log(1−Pθ) part is active. Now a higher probability increases the loss. We want the value in the brackets to be as large as possible. The minus sign at the start changes it into a loss that training makes as small as possible. In short: increase the probability of query tokens, and decrease the probability of all other tokens. The lecturer's advice: understand why each part is there. The exam more probably asks you to explain the function than to write it.
The QL direction of BiQDL (lecture slide 10). Training increases the document weights at the query tokens and decreases them at all other tokens.
Direction 2: document likelihood (DL). Change the roles. BERT encodes the query into a dense representation. The tokeniser encodes the document into a sparse representation. From the query text, the model learns which vocabulary terms are important for the relevant document:
LDL(D)=−(q,d)∈D∑∣V∣1i∑∣V∣[ylog(Pθ(ti∣q))+(1−y)log(1−Pθ(ti∣q))]y={1,0,if vocabulary term ti appears in document d,otherwise.
Where
LDL(D)
document-likelihood loss over the training set D
Pθ(ti∣q)
the probability that the model gives to token ti for the query (BERT now encodes the query)
y
label: 1 if ti is in the document, 0 if not
D,(q,d),∣V∣,ti,θ
the same as in LQL
Read it aloud
The document-likelihood loss over D is minus the sum over every pair of one over the vocabulary size times the sum over every vocabulary token of: y times the log of the probability of t-i given q, plus one minus y times the log of one minus that probability. Here y is 1 if t-i occurs in the document.
Formula in plain words
The structure is the same, with the roles changed. From the query text, increase the probability of tokens that occur in the document. Decrease the probability of all other tokens. If you understand the QL direction, you understand this direction too. It trains the query side to predict document terms. This makes the TILDE-DL ranking mode possible (see below).
Combined. The bi-directional loss is the average of the two directions:
LBiQDL(D)=2LQL(D)+LDL(D)
Where
LBiQDL(D)
the bi-directional query-document likelihood loss that trains TILDE
LQL(D),LDL(D)
the query-likelihood loss and the document-likelihood loss above
Read it aloud
The BiQDL loss over D is equal to the query-likelihood loss plus the document-likelihood loss, divided by two.
Formula in plain words
The training loss gives the same weight to the two directions. A lower loss in one direction decreases the total, so training improves the two abilities at the same time. After training with BiQDL, TILDE can estimate the query likelihood for a document text. It can also estimate the document likelihood for a query text.
Worked example 4Computing the BiQDL loss on a tiny vocabulary
Setup (illustrative numbers). There is one training pair: query q= "apple account", relevant document d= "apple store account help". The small vocabulary is V={apple,store,account,banana}, so ∣V∣=4. The current model reads d and predicts these probabilities for each token in the query: Pθ(apple∣d)=0.7, Pθ(store∣d)=0.5, Pθ(account∣d)=0.3, Pθ(banana∣d)=0.2. It reads q and predicts these probabilities for each token in the document: Pθ(apple∣q)=0.8, Pθ(store∣q)=0.4, Pθ(account∣q)=0.6, Pθ(banana∣q)=0.1. (All logs are natural logs.)
1
Make the label vector for the QL direction.yi=1 if token ti is in the query. The query tokens are {apple, account}. Thus y=[1,0,1,0] for (apple, store, account, banana).
2
QL terms for each token. For tokens in the query, use logP. For other tokens, use log(1−P):
apple (y=1):log0.7store (y=0):log(1−0.5)account (y=1):log0.3banana (y=0):log(1−0.2)=−0.357=log0.5=−0.693=−1.204=log0.8=−0.223
3
Average and change the sign.sumaverageLQL=−0.357−0.693−1.204−0.223=−2.477=∣V∣sum=4sum=−0.619=−(average)=0.619
"account" adds the most loss. The model gave a true query token a probability of only 0.3. It also gave the non-query token "store" a probability that is too high (0.5). Training increases P(account∣d) and decreases P(store∣d).
4
Label vector for the DL direction. Now yi=1 if ti is in the document "apple store account help". The document tokens in our small V are {apple, store, account}, so y=[1,1,1,0].
5
DL terms for each token.apple:log0.8store:log0.4account:log0.6banana:log(1−0.1)=−0.223=−0.916=−0.511=−0.105
6
Average and change the sign.sumaverageLDL=−1.755=4sum=−0.439=−(average)=0.439
7
Combine.LBiQDL=20.619+0.439=0.529.
Result
The BiQDL loss for this pair is 0.529. When training decreases this loss, the model learns two things at the same time. It learns to predict query tokens from documents (QL direction) and document tokens from queries (DL direction). Thus a trained TILDE can rank with query likelihood, document likelihood, or both.
5.4 Three ways to rank with TILDE
These two abilities give three ranking settings. Each setting gives a different balance of efficiency and effectiveness. The in-class lecture gives a comment on each setting (lecture slide 13).
TILDE-QL(q∣dk)=i∑∣q∣log(Pθ(qi∣dk))
Where
dk
candidate document number k
∣q∣
number of tokens in the query
qi
query token number i
Pθ(qi∣dk)
stored probability of query token qi for the document (from the offline document vector)
Read it aloud
TILDE-QL of q given d-k is the sum, over each query token q-i, of the log of the probability of q-i given d-k.
Formula in plain words
Rank each document by the likelihood that it gives to the query tokens. The document representations exist before the query arrives. Thus the score needs only a small number of look-ups and additions. It is a dot product: cheap and effective, with no GPU at query time. This is the default setting of TILDE. Logs of probabilities are negative, so scores nearer to zero are better. One query token with a very small probability decreases the full score, because its log is a large negative number.
TILDE-DL(dk∣q)=∣dk∣1i∑∣dk∣log(Pθ(dik∣q))
Where
∣dk∣
number of tokens in document dk
dik
token number i of document dk
Pθ(dik∣q)
probability of that document token for the query (BERT encodes the query at search time)
Read it aloud
TILDE-DL of d-k given q is one over the length of d-k times the sum, over each token of d-k, of the log of the probability of that token given q.
Formula in plain words
Rank by the average log-probability of the document tokens for the query. The division by the document length ∣dk∣ normalises the score. A longer document has more tokens and collects a lower (more negative) total likelihood. Without the division, long documents lose. This direction runs BERT on the query at search time. Thus it needs a GPU, and queries are slow. The lecture calls it an ablation in the paper: TILDE does not need it.
mixing weight in [0,1]: the part of the score that comes from query likelihood
TILDE-QL,TILDE-DL
the two scores above
Read it aloud
TILDE-QDL of q and d-k is equal to alpha times TILDE-QL of q given d-k, plus one minus alpha times TILDE-DL of d-k given q.
Formula in plain words
The final score is a weighted mix of the two directions. α=1 gives pure query likelihood (the fastest setting, tokeniser only). α=0 gives pure document likelihood. Values between them mix the two scores. Any α<1 still needs the online query inference. Thus the mix gives a small increase in effectiveness, but a longer query latency.
Mode
Online work
Lecture comment
When a system designer selects it
TILDE-QL (default)
Tokenise the query. Look up stored document weights.
"Just a dot product (cheap and effective)"
The usual case. The corpus size is acceptable, so encode each document one time offline. Users get the fastest response.
TILDE-DL
BERT inference on the query at search time
An ablation in the paper. Slow at query time.
The document side is very large and you do not want to run BERT on all documents. The number of queries is small.
TILDE-QDL
The two items above
A small gain. Still slow, because it needs online calculation.
Effectiveness is the most important goal and a longer latency is acceptable (it is still faster than a dense model).
5.5 The effect of the document likelihood signal
Plots of nDCG@10 and MAP against α (TREC DL 2019 and 2020) show three facts:
α=0 (document likelihood only): performance is very bad (nDCG@10 ≈ 0.23–0.25, MAP ≈ 0.13–0.20). Document likelihood alone cannot rank documents.
α=0.4–0.5: best performance (nDCG@10 ≈ 0.61–0.62). Document likelihood helps when it is part of the mix.
α=1 (pure query likelihood): a small decrease from the best (≈ 0.58–0.62). This setting has the shortest query latency, because it needs no GPU inference at query time.
nDCG@10 of TILDE-QDL for different values of α (approximate values from slide 26). The MAP plot on the same slide has the same shape.
Worked example 5TILDE-QDL interpolation with α
Setup (illustrative numbers). For a query q and two candidate passages, a trained TILDE gives: TILDE-QL(q∣d1)=−1.9, TILDE-DL(d1∣q)=−0.6, TILDE-QL(q∣d2)=−1.6, TILDE-DL(d2∣q)=−1.4. All are log-likelihoods, so a higher value (nearer to zero) is better. The DL scores are already normalised for length.
1
Pure query likelihood, α=1. The scores are the QL values: d1:−1.9, d2:−1.6. The ranking is d2≻d1. This is the fastest setting: no GPU.
2
Pure document likelihood, α=0. The scores are the DL values: d1:−0.6, d2:−1.4. The ranking is d1≻d2. The lecture shows that this setting alone gives bad performance.
3
Mix at the best value, α=0.4.d1:0.4×(−1.9)+0.6×(−0.6)d2:0.4×(−1.6)+0.6×(−1.4)=−0.76−0.36=−1.12=−0.64−0.84=−1.48
4
Rank.−1.12>−1.48, so d1≻d2. The document-likelihood evidence gave strong support to d1. It changed the ranking of pure QL.
Result
α controls a real trade-off. α=1 gives the shortest latency (tokeniser only). α≈0.4–0.5 gives the best effectiveness, but needs one BERT inference on the query at search time (GPU). The plots show that the effectiveness difference between these two settings is small.
PracticeLecture Exercise 1: how does TILDE get high efficiency, compared with dense retrievers?
The lecture expects three different aspects.
Answer(1) At query time, the system only tokenises the query. No BERT encoding runs online. (2) Documents go through BERT, but offline at indexing time, one time only. The index stays the same while queries arrive. (3) The query vector is sparse (1 only at the query tokens). Thus scoring uses only those few dimensions of each document vector.
PracticeIn LQL, what occurs to the loss if training increases Pθ(ti∣d) for a token that is not in the query? Why is there a minus sign before the sum?
AnswerFor that token, y=0. Thus only log(1−Pθ) is active. A higher Pθ makes 1−Pθ smaller and the log more negative. Thus the loss increases. The sum in the brackets is a log-likelihood that we want to be as large as possible. The minus sign changes it into a loss that training makes as small as possible.
PracticeYour search service has a very large document collection that changes frequently. It gets a small number of queries each day. Which TILDE ranking mode is correct for it? What do you lose?
AnswerTILDE-DL. It does not run BERT on every document offline. The neural work occurs on the queries, and there are few queries. You lose query latency (one BERT inference for each query, with a GPU). You also lose effectiveness, because DL alone ranks badly (nDCG@10 ≈ 0.23 at α=0).
6. TILDEv2: exact matching with learned weights and cheap expansion
TILDEv2 keeps the sparse query encoding from the BERT tokeniser. It changes the document side in two ways:
Token embeddings in place of [CLS]. The contextualised token embeddings represent the document, one weight for each token. Scoring calculates the inner product of the sparse query vector and these token embeddings. In practice, this scores only exact matches between query terms and tokens of the expanded document. Each matched token has a contextualised learned weight in place of a count. This solves the semantic-gap problem.
Document expansion before encoding. Exact matching alone has the query–document vocabulary mismatch problem. Thus the system expands the document before encoding. It uses doc2query or TILDE. A trained TILDE can generate the most probable query terms for a document, as doc2query does, at a much lower cost.
TILDEv2: exact term matching between the sparse query vector and contextualised token weights of the expanded document. "account" is in the document only because expansion added it.
Worked example 6Scoring a query with TILDEv2
Setup (illustrative numbers). The query is "apple account". The original passage d is "apple store". At indexing time, TILDE expansion proposed the extra tokens {account, iphone}. These are probable query terms for this passage. After filtering (see 6.2), the system added both. BERT then encoded the expanded passage and gave one learned weight for each token: apple → 4.7, store → 0.2, account → 3.1, iphone → 2.4.
1
Tokenise the query. The active terms are {apple, account}. This is the sparse query vector. Scoring uses only these terms.
2
Exact-match lookup. For each query term, find the same token in the expanded document and take its contextualised weight. "apple" matches with weight 4.7. "account" matches with weight 3.1, only because expansion added it. (If a term occurs more than one time, use its stored match weight from the index.)
3
Add the matched weights.s(q,d)=4.7+3.1=7.8.
4
Compare with the passage without expansion. Without expansion, "account" matches nothing and s=4.7. The passage then gets a rank as if it has no information about accounts. Also compare with plain BM25. BM25 gives "apple" a weight from counts and rarity only. The TILDEv2 weight 4.7 is contextualised. The same word "apple" in a passage about fruit gets a different weight.
Result
TILDEv2 = exact term matching (fast, good for inverted indexes) + learned contextualised term weights (meaning) + offline expansion (solves vocabulary mismatch). Scoring again costs only a small number of additions for each document.
6.2 How important is the expansion component?
The table gives results (MRR@10, MS MARCO dev) for different expansion methods. TILDE generates m candidate terms. The system adds only terms that are not stop words and not already in the document. The same filter applies to doc2query. More added terms also make a larger index:
Measure
No expansion
doc2query
TILDE m=128
TILDE m=150
TILDE m=200
MRR@10
0.299
0.333
0.326
0.327
0.330
Avg added tokens
–
19.0
13.0
25.2
61.6
Index size
4.3 GB
5.2 GB
5.2 GB
5.6 GB
6.9 GB
Expansion cost
–
320 hours, $768 (TPU)
7.22 hours, $5.34 (GPU)
7.25 hours, $5.37
7.33 hours, $5.42
How to read the table: expansion is always better than no expansion (0.299 → 0.32+). doc2query gives the best effectiveness (0.333). TILDE must add many more terms to get near that value. With m = 200, TILDE adds 61.6 tokens on average, and doc2query adds 19. This also makes the index larger (6.9 GB against 5.2 GB). Thus each term that TILDE generates is a little weaker than a doc2query term. The cost column gives the other part of the story:
Collection
doc2query (TPU)
TILDE expansion (GPU)
MS MARCO
320 hours (about 13 days), $768
about 7¼ hours, about $5
MS MARCO v2 (15× larger)
about 200 days, more than $11,000
about 4.5 days, about $80
Key idea
Offline work also has a cost. doc2query ran on Google TPUs. TPUs are about 15 times more powerful than GPUs, and they are expensive. TILDE expansion runs on cheap GPUs. The difference increases for commercial search engines (Google, Bing, Baidu). At that scale, a wait of 13 days (or 200 days) for an index is not possible.
PracticeWhy does TILDEv2 need document expansion, when the original TILDE works without it?
AnswerThe TILDE document vector is dense over the full vocabulary. Thus every query token gets a learned weight (implicit expansion). TILDEv2 scores only exact matches between query tokens and the document tokens. A query term that is not in the document adds nothing. Expansion puts probable query terms into the document before encoding. This gives matches for those terms again.
PracticeIn the TILDEv2 expansion table, doc2query gives the best MRR@10. Give two reasons to select TILDE expansion.
Answer(1) Cost. On MS MARCO, TILDE takes about 7 hours and $5 on GPUs, and doc2query takes 320 hours and $768 on TPUs. On MS MARCO v2, the difference is about 4.5 days against 200 days. (2) The effectiveness difference is small (0.330 with m = 200 against 0.333). Thus the saving costs very little quality. The disadvantage is a larger index when m is large.
7. SPLADE: expansion and weights from the MLM head
SPLADE (Formal, Piwowarski and Clinchant, SIGIR 2021) uses the second route. It keeps the transformer encoder for queries and documents. The encoder predicts term-importance weights over the full vocabulary. Take the query "Who wrote the book of the lord of the rings". A traditional lexical model can match only those exact words. SPLADE activates related vocabulary terms with learned importance weights, for example:
Activated term
write
created
authored
novel
LOTR
Tolkien
Contribution
3.5
1.2
3.3
3.3
5
4
These terms can come from outside the query. Tolkien gets weight 4 only from its semantic relation to the query. The vector is still sparse: almost all of the 30,522 vocabulary dimensions stay at zero. If we can predict how much related vocabulary terms help the ranking, we get a much better lexical retriever. The transformer gives these weights. Collection statistics have no part in them.
7.2 The mechanism: reuse the BERT MLM head
BERT pre-training uses a masked language modelling (MLM) head to predict vocabulary tokens. SPLADE reuses this head at every input position. It obtains vocabulary logits, converts them to non-negative term activations, then pools each vocabulary column into one weight. The lecture uses “probability” as a simple explanation of term importance. Here pij denotes the supplied non-negative activation at input position i for vocabulary term j.
wj=i=1∑npijj=1,…,∣V∣
Where
wj
importance weight of vocabulary term j for this query or document
n
number of token positions in the input text
i
index of one input token position
pij
weight that the MLM head at position i gives to vocabulary term j
j,∣V∣
index of a vocabulary term, and the vocabulary size (30,522)
Read it aloud
The weight w-j is equal to the sum, over every input position i from 1 to n, of p-i-j. This applies to every vocabulary term j from 1 to the vocabulary size.
Formula in plain words
A vocabulary term becomes important when many positions of the text "point to it". This is also true if the term does not occur in the text. Each position adds its weight for the term. Thus a term gets weight if many positions support it weakly, or if one position supports it strongly. Longer inputs have more positions, so their sums are usually larger. The result is one vector of vocabulary size for each query or document.
SPLADE. Each token position gives an MLM activation vector over the vocabulary. Pooling over the positions gives the term-importance vector.
Technical detail beyond the lecture shorthandThe original SPLADE paper, Section 3.2 defines the activation before pooling as follows.
pij=log(1+max(0,zij))
Where
zij
raw MLM logit at input position i for vocabulary term j
pij
non-negative activation used in the pooling formulas
max(0,zij)
ReLU: keep a positive logit, otherwise return zero
log
natural logarithm
Read it aloudp-i-j equals the natural log of one plus the maximum of zero and z-i-j.
Formula in plain wordsReLU removes non-positive logits. The logarithm reduces the effect of large positive logits. Zero remains zero. Pool these activations to obtain vocabulary weights; the weights need no normalisation to a sum of one.
7.3 How SPLADE keeps weights sparse
A softmax gives positive probability to every vocabulary term. SPLADE instead applies ReLU to raw logits, then log saturation, before pooling. ReLU makes non-positive logits exactly zero. Sparsity regularisation encourages the model to activate fewer terms; the ranking loss keeps useful matching terms. FLOPS regularisation discourages frequently active dimensions that make posting lists costly. L1 penalises total activation magnitude. The lecture mentions both approaches; their detailed equations are optional reading.
Exam angleExplain the roles separately: the ranking objective learns relevance, ReLU creates zeros, and regularisation encourages efficient sparse representations. Sum or max pooling combines positions. Max pooling alone does not create zeros when every input value is positive.
7.4 SPLADE core ideas, in summary
Every vocabulary term can get a weight. Traditional models give weights only to terms in the text.
A BERT-based model predicts term importance for the query or document. It uses all the semantic knowledge from pre-training.
Implicit expansion: the weights are free of the original text terms. Thus related terms become active and help the ranking.
Sparsity stays because of ReLU and regularisation. This keeps the efficiency of sparse representations.
Worked example 7SPLADE pooling (sum vs max) and scoring
Setup (illustrative numbers). The document is "apple reduces sugar" (3 token positions). The small vocabulary has 6 terms. Use these example non-negative activations. These example rows happen to sum to 1:
Position
apple
fruit
reduces
lowers
sugar
car
pos 1 · "apple"
0.50
0.30
0.02
0.02
0.10
0.06
pos 2 · "reduces"
0.05
0.05
0.45
0.30
0.10
0.05
pos 3 · "sugar"
0.10
0.15
0.05
0.05
0.60
0.05
1
Sum pooling (SPLADEv1). Add each column:
applefruitreduceslowerssugarcar=0.50+0.05+0.10=0.65=0.30+0.05+0.15=0.50=0.02+0.45+0.05=0.52=0.02+0.30+0.05=0.37=0.10+0.10+0.60=0.80=0.06+0.05+0.05=0.16
Document vector (sum): [0.65,0.50,0.52,0.37,0.80,0.16]. fruit (0.50) and lowers (0.37) get real weight, but these words are not in the document. This is implicit expansion.
2
Max pooling (SPLADEv2). Take the largest value of each column:
applefruitreduceslowerssugarcar=max(0.50,0.05,0.10)=0.50=0.30=0.45=0.30=0.60=0.06
Document vector (max): [0.50,0.30,0.45,0.30,0.60,0.06]. Max pooling keeps the strongest single signal for each term. Many weak signals do not add up. "car" has no real support from any position, and it decreases from 0.16 to 0.06.
3
Illustrate optional pruning. For this arithmetic example only, set pooled weights below 0.2 to zero. This threshold is an extra illustration; SPLADE learns sparsity through its activation and regularisation. The sum-pooled vector becomes [0.65,0.50,0.52,0.37,0.80,0]. "car" is now inactive. Thus this document is never in the posting list for "car".
4
Encode a query in the same way. The query "lowers sugar" goes through the same SPLADE encoder. Assume that its sparse vector is [0,0,0.3,0.9,1.1,0]. Expansion activates reduces, and lowers and sugar are active too.
5
Score with a dot product over the shared non-zero terms. Use the sparse sum-pooled document vector: s(q,d)=0.3×0.52+0.9×0.37+1.1×0.80=0.156+0.333+0.880=1.369. Scoring uses only 3 of 6 dimensions. With a real vocabulary, only a few dozen active terms have an effect. This makes SPLADE scoring as fast as an inverted index.
Result
The query word "lowers" is not in the document. The document word "reduces" is not in the query. But the two match through their expanded, weighted representations. This is semantic matching inside a sparse, lexical-style dot product.
7.5 TILDE versus SPLADE
The in-class lecture compares the two families side by side (lecture slide 18). The lecturer recommends this comparison figure for revision:
The source of the weights. TILDE takes only the [CLS] vector. A trained linear layer projects it to vocabulary size. SPLADE keeps all token vectors (as ColBERT does in Week 5). It sends each vector through the pre-trained MLM head and pools the activation vectors (sum in v1, max in v2). The MLM head gives SPLADE a better start.
The cause of sparsity. TILDE is sparse because of its architecture. The tokenised query vector is sparse by design, and TILDE has no sparsity loss. SPLADE is sparse because of training. The encoder makes both vectors, and the sparsity loss makes them sparse.
Speed. TILDE is usually much faster than SPLADE, because its query side needs only the tokeniser. SPLADE always sends the query through the encoder.
Common points. Both give every vocabulary term a chance to get a weight. Both use a BERT-based model to predict term importance. Both score with a dot product that skips the zeros.
TILDE and SPLADE for the query "deep learning" (from lecture slide 18). TILDE weights come from [CLS] and a linear layer. SPLADE weights come from the MLM activation vector of every token, after pooling.PracticeLecture Exercise 1.1: how does SPLADE get its term weights?
Give the representation that it takes from BERT, the head that it uses, and how it combines them.
AnswerSPLADE sends the query or document through BERT and keeps the output vector of every token. Each vector goes through the pre-trained MLM head. The head gives logits over the full vocabulary for that position. ReLU and log saturation turn these into an activation vector. SPLADE then pools these activation vectors into one vector of vocabulary size: sum pooling in SPLADEv1, max pooling in SPLADEv2. The value in each dimension is the importance of that term. During training, ReLU and the FLOPS/L1 sparsity losses make most weights exactly zero.
PracticeBoth models score with a sparse dot product. Why is TILDE usually faster than SPLADE at query time?
AnswerThe TILDE query side is only the BERT tokeniser, a lookup table with no neural inference. SPLADE must send the query through the full BERT encoder and MLM head before it can score. Thus each SPLADE query needs one transformer inference.
SPLADEv2 comes from the same research group. It has three main improvements:
Aspect
SPLADEv1
SPLADEv2
Pooling
Sum pooling combines the token activation vectors
Max pooling. For each vocabulary dimension, take the highest weight over all input token positions. Tests show that this keeps the most important signal better.
Training loss
Basic contrastive training
Adds a distillation loss. The model also learns from a teacher model (in addition to the relevance labels). This improves weighting and expansion.
Weighting and expansion
The model weights and expands the query and the document. Thus the full SPLADE inference runs at query time.
A new variant with weighting and expansion of the document only (similar to TILDE). Queries stay simple and near their original form.
Max pooling replaces the sum in the SPLADE weight formula:
wj=i=1,…,nmaxpijj=1,…,∣V∣
Where
wj
importance weight of vocabulary term j
maxi
the largest value over all input positions i=1,…,n
pij
MLM weight of term j at position i
n,∣V∣
number of input positions, and the vocabulary size
Read it aloud
The weight w-j is equal to the maximum, over all input positions i from 1 to n, of p-i-j, for every vocabulary term j.
Formula in plain words
For each vocabulary term, keep only the strongest vote from one position. A term with strong support from one position keeps that full weight. A term with weak support from many positions stays small, because weak votes do not add up. Weights can exceed 1. More text increases a weight only if a new position gives a stronger vote than all earlier positions.
Worked example 8 for four of the ten terms. Sum pooling stacks the weights of the two positions. Max pooling keeps the larger weight.
Reading slides 24–25The two supplied rows sum to 2.2 and 1.9. Treat them as term weights for this pooling exercise. Keep their numbers unchanged. Slide 25 repeats “SPLADEv1” in its opening line, but its title and question ask for SPLADEv2, so use max pooling. Pool each vocabulary column across input positions.
Worked example 8Lecture Exercises 2 and 2.1: SPLADEv1 and SPLADEv2 pooling for “deep learning”
Setup (lecture slides 24–25; a very small vocabulary on purpose). The full vocabulary has 10 terms: [deep, apple, learning, neural, networks, fruit, like, machine, map, shallow]. The query is "deep learning". The MLM head predicts one activation vector for each query position:
Position
deep
apple
learning
neural
networks
fruit
like
machine
map
shallow
“deep”
0.9
0
0.3
0.5
0.2
0
0
0.2
0
0.1
“learning”
0.1
0.4
0.5
0
0.2
0
0.4
0.3
0
0
1
SPLADEv1: sum pooling. Add the two rows, term by term:
deepapplelearningneuralnetworksfruitlikemachinemapshallow=0.9+0.1=1.0=0+0.4=0.4=0.3+0.5=0.8=0.5+0=0.5=0.2+0.2=0.4=0+0=0=0+0.4=0.4=0.2+0.3=0.5=0+0=0=0.1+0=0.1
wv1=[1.0,0.4,0.8,0.5,0.4,0,0.4,0.5,0,0.1]
2
SPLADEv2: max pooling. Take the larger value in each column:
deepapplelearningneuralnetworksfruitlikemachinemapshallow=max(0.9,0.1)=0.9=max(0,0.4)=0.4=max(0.3,0.5)=0.5=max(0.5,0)=0.5=0.2=0=0.4=max(0.2,0.3)=0.3=0=0.1
wv2=[0.9,0.4,0.5,0.5,0.2,0,0.4,0.3,0,0.1]
3
Read the vectors. Both vectors keep neural, networks and machine active, but the query does not contain these words. This is implicit expansion. Terms with support from both positions (learning, networks, machine) get the largest increase from sum pooling. fruit and map stay at 0 in both vectors. Thus the query never touches their posting lists.
Result
SPLADEv1 (sum): [1.0, 0.4, 0.8, 0.5, 0.4, 0, 0.4, 0.5, 0, 0.1]. SPLADEv2 (max): [0.9, 0.4, 0.5, 0.5, 0.2, 0, 0.4, 0.3, 0, 0.1]. The lecturer's summary: the MLM head gives many vectors, and you combine them. If you combine with sum, it is v1. If you combine with max, it is v2.
The variant for documents only is important, because the asymmetric argument from TILDE applies here too. Document expansion occurs offline at indexing time. There is much time to process each document fully. Query processing occurs online while a user waits. Thus a simple query side gives much faster search responses. Most of the semantic-matching benefit comes from the document side. Thus the removal of query-side expansion gives efficiency at a small cost.
PracticeTwo positions give MLM activation vectors over four terms: p1=[0.6,0.2,0.0,0.4] and p2=[0.1,0.5,0.3,0.4]. Calculate the SPLADEv1 vector and the SPLADEv2 vector.
AnswerSum (v1): [0.7,0.7,0.3,0.8]. Max (v2): [0.6,0.5,0.3,0.4]. Both positions support the fourth term equally. It is the largest weight with sum pooling, and only a medium weight with max pooling.
PracticeWhy does the SPLADEv2 variant for documents only search faster than SPLADEv1?
AnswerSPLADEv1 weights and expands the query with a full SPLADE inference at query time. The variant for documents only does all weighting and expansion offline, on the documents. The online query stays near its original tokens. There is no expensive query inference, and there are fewer active query terms to score.
All numbers below are for the MS MARCO dev set with MRR@10 (mean reciprocal rank over the top 10, from Week 2). MRR@10 is the standard measure for MS MARCO dev because its relevance judgements are sparse. Usually each query has one judged relevant passage. The values are the approximate numbers from Video 6. Where the video gives no number, the values come from the chart bars.
MRR@10 on MS MARCO dev (approximate). Grey: term-based lexical. Blue: encoder-based dense (single-vector and multi-vector). Orange: decoder-based dense. Green: learned sparse.
Watch outThe bar chart on slides 41–46 shows RepBERT near 0.30 and ColBERT near 0.36. Video 6 gives about 0.33 and 0.39. The order is the same in both sources. Give approximate values and put the focus on the order of the families.
The lecture goes through the chart in this order:
BM25 (≈ 0.18): the traditional baseline. It is fast and efficient, but exact matching limits it.
Single-vector dense: RepBERT, ANCE (≈ 0.33): almost two times BM25. This improvement made the field pay attention to transformer-based retrieval. Semantic matching does the work.
Multi-vector dense: ColBERT (≈ 0.39): better again, because it has one vector for each token and finer matching. It is much slower to encode and score than single-vector models.
Decoder-based: RepLLaMA, LLM2Vec (≈ 0.41–0.42): the most recent models. Their backbones have about 7B parameters (BERT has 110M), and their vectors have more dimensions. Thus encoding and scoring cost much more.
Learned sparse: BM25+TILDE ≈ 0.27, BM25+TILDEv2 ≈ 0.33, SPLADEv1 ≈ 0.32, SPLADEv2 ≈ 0.34: a large increase over BM25 (TILDEv2 is much better than TILDE). They come near single-vector dense effectiveness. SPLADE is a little above the TILDE family (it is also a later model). SPLADE is less efficient than TILDE, because it still runs a query encoder at search time.
Takeaways
Learned sparse models are a little less effective than the best dense retrievers. But they are usually much more efficient. As traditional lexical models do, they score only documents that share (expanded) terms with the query. This gives fast search and low computation needs. They are in a good middle position: much more effective than BM25, and they keep much of the efficiency that makes lexical models practical at large scale.
PracticeLLM2Vec gets about 0.42 MRR@10 and SPLADEv2 gets about 0.34. Give two reasons why a production search team can still select SPLADEv2.
Answer(1) Efficiency. SPLADEv2 scores through an inverted index and touches only documents that share active terms. LLM2Vec compares dense vectors with many dimensions. (2) Model size. SPLADE uses an encoder of BERT size (about 110M parameters), and LLM2Vec uses a decoder with 7B parameters. Thus encoding the collection and each query costs much less GPU time and memory. The variant for documents only decreases the query cost more.
10. PromptReps: representations from an LLM with no training
The LLM-based dense retrievers from Week 5 all need training before they can retrieve:
E5(-Mistral) takes the embedding from the last hidden layer of an [EOS] token that it adds to the query or document. It needs (supervised) contrastive fine-tuning. It can also use synthetic data that an LLM generates.
LLM2Vec takes the embedding from mean pooling over the last hidden states of all sequence tokens. It needs (unsupervised or supervised) contrastive fine-tuning.
Contrastive fine-tuning is expensive in two ways. You need labels (or synthetic data), and you need a large GPU infrastructure. A larger decoder backbone makes training longer and needs more memory. PromptReps (Zhuang et al., EMNLP 2024) asks this question: can prompts make the LLM give an effective retrieval representation with no contrastive fine-tuning? This is "zero-shot" generation of representations: there is no training after the LLM pre-training, and there are no in-context examples.
10.2 How it works
Tell the LLM to represent the text with one word. For queries, the prompt is a little different ("represent the query"):
<System> You are an AI assistant that can understand human language. <User> Passage: "[text]". Use one word to represent the passage in a retrieval task. Make sure your word is in lowercase. <Assistant> The word is: "
Run one inference. Take two representations from the position of the last quote character. At this position, the model is about to generate its one summary word:
Dense: the last hidden state at that position. It encodes the word that the model will generate. The system uses it for dense retrieval with an ANN index. The lecture says that this is the same type of embedding that RepLLaMA uses.
Sparse: continue through the LM head to get the next-token logits. These are raw prediction scores over the full LLM vocabulary. Keep the top-k predicted terms as a sparse vector. The system uses it for inverted-index matching.
One inference gives two representations. The hidden state gives a dense vector. The next-token logits give a sparse vector over the LLM vocabulary. A linear combination of the two scores gives the final score.
At retrieval time, the dense representation does dense retrieval. The sparse representation does inverted-index matching. Each gives a score, and a linear combination (an interpolation) of the two scores gives the final relevance score. The result is a hybrid retriever from one encoder. The lecture gives these points: PromptReps can run as a sparse model only, as a dense model only, or as the hybrid. It works zero-shot. If you train it with a standard contrastive loss, it becomes much better. A later model in Week 9 uses a text-diffusion model and applies the same idea with multi-vector representations.
The two scores have different scales. Cosine similarity is in [−1, 1], and a sparse dot product has no upper limit. Thus a common method first applies min–max normalisation to each score list over the candidates. Then it mixes the two scores:
the smallest and the largest raw score of that type in the candidate list
s~
the normalised score, between 0 and 1
s~dense,s~sparse
normalised dense (hidden-state) score and normalised sparse (logits) score
λ
mixing weight in [0,1] for the dense score
shybrid
final hybrid relevance score
Read it aloud
s-tilde is equal to s minus s-min, divided by s-max minus s-min. The hybrid score is equal to lambda times the normalised dense score, plus one minus lambda times the normalised sparse score.
Formula in plain words
First, change the scale of each score list: its smallest value becomes 0 and its largest value becomes 1. To do this, subtract the minimum and divide by the range. The two signals then have the same scale. Then calculate a weighted sum: a part λ of the dense score plus the remaining part of the sparse score. λ=1 is pure dense retrieval. λ=0 is pure sparse retrieval. Values between them give a high score to documents that are strong on both signals. The slides say only "combine linearly". Min–max normalisation is one standard method to do it.
Worked-example conventionThe lecture specifies linear combination. The min–max scaling here is an illustrative choice. When a score list has equal minimum and maximum, give that list a zero normalised contribution to avoid division by zero. Check effectiveness with relevance judgements after choosing a fusion rule.
10.3 Does zero-shot retrieval work?
These results are for MS MARCO (MRR@10, shown ×100 on the slide). All systems are zero-shot: no in-context examples and no contrastive training of the representations.
System
MRR@10 (approx.)
Notes
LLM2Vec (unsupervised, SimCSE-trained baseline)
≈ 13.5
still needs unsupervised contrastive training
PromptReps (Dense only)
≈ 17.5
below BM25
BM25 (dotted reference line)
≈ 18.5
also needs no training
PromptReps (Sparse only)
≈ 21
above BM25
PromptReps (Hybrid)
≈ 24.5
the combination of dense and sparse is the key
Key ideaThe combination of the dense and sparse representations gives very effective zero-shot retrieval. The hybrid is clearly better than BM25 and the unsupervised LLM2Vec baseline, with no training of the representations.
Extension: more than one word. The prompt can ask for two words, three words, a sentence or a paragraph. This needs more than one autoregressive inference and gives more than one token embedding or distribution. The system can combine them (max pooling over the sparse distributions, mean pooling over the dense vectors). It can also use them in the ColBERT style, with MaxSim between the query-side and passage-side representations. The lecture stops at this point, because these multi-token representations of paragraphs and passages are weak.
Worked example 9Combining dense and sparse scores in a hybrid retriever
Setup (illustrative numbers). A user sends a query to a PromptReps system. Three candidate passages get a dense score (cosine similarity from the ANN index) and a sparse score (inverted-index match on the logits vectors, on an illustrative scale). Use the normalisation and mix above with λ=0.5:
Passage
Dense score sdense
Sparse score ssparse
d1
0.82
6.0
d2
0.78
12.0
d3
0.70
9.0
1
Normalise the dense scores. Min = 0.70, max = 0.82, range = 0.12.
d1:0.120.82−0.70d2:0.120.78−0.70d3:0.120.70−0.70=1.00=0.667=0.00
2
Normalise the sparse scores. Min = 6.0, max = 12.0, range = 6.0.
d1:66−6d2:612−6d3:69−6=0.00=1.00=0.50
3
Mix with λ=0.5.d1:0.5×1.00+0.5×0.00d2:0.5×0.667+0.5×1.00d3:0.5×0.00+0.5×0.50=0.50=0.83=0.25
4
Rank.d2≻d1≻d3. No single signal gives this ranking. With the dense score only, d1 is first. With the sparse score only, d1 is last. The hybrid gives a high score to d2 because it is strong on both signals. This shows how score fusion changes the order. The measured MS MARCO results appear separately above.
Result
A linear mix gives a combined ranking. Relevance judgements are needed to measure whether it improves effectiveness. These example scores show the calculation only.
PracticeLecture Exercise 1.2: why is PromptReps a "hybrid" method?
Give the source of the dense part and the source of the sparse part.
AnswerOne LLM inference with the prompt gives two representations. The dense part is the last-layer hidden state at the last prompt token. The system uses it as a RepLLaMA-style embedding in an ANN index. The sparse (lexical) part is the next-token scores over the vocabulary. The prompt asks for one word that represents the text, so the top-k predicted terms show its meaning. These terms go into an inverted index. The system then combines the dense and sparse scores. Thus one encoder gives a hybrid retriever.
PracticeDo Worked example 9 again with λ=0.8. Does the ranking change?
Answerd1:0.8×1.00+0.2×0.00d2:0.8×0.667+0.2×1.00d3:0.8×0.00+0.2×0.50=0.80=0.733=0.10
The ranking changes to d1≻d2≻d3. Most of the weight is on the dense signal, and d1 has the highest dense score.
11. Summary and study checklist
Exam angleThe lecturer emphasised explaining each model from its architecture. Practise these four tasks: explain TILDE's three efficiency gains; trace SPLADE from token vectors to vocabulary weights; identify both PromptReps outputs; calculate sum and max pooling column by column. For BiQDL, explain which token probabilities each direction raises or lowers and how ranking choices affect online work (lecture slides 10–14, 21–25).
Model
Query side (online)
Document side (offline)
Expansion
Training signal
Online neural inference?
BM25 + doc2query
Plain terms
Text + generated queries in a normal inverted index
Explicit, generated by seq2seq (T5)
Supervised ⟨query, doc⟩ pairs (for doc2query)
No
TILDE
BERT tokeniser → binary sparse vector
[CLS] → linear → dense |V| vector
Implicit (the dense document vector covers the full vocabulary)
BiQDL (bi-directional query/document likelihood)
No (with α = 1). Yes with DL.
TILDEv2
BERT tokeniser → binary sparse vector
Contextualised token weights of the expanded document
Explicit, with doc2query or TILDE (cheap)
Contextualised exact-match training
No
SPLADE
Full encoder → sparse weighted vector
Full encoder → sparse weighted vector (sum pooling)
Implicit, with the MLM head over the vocabulary
Contrastive + ReLU/FLOPS/L1 for sparsity
Yes (query encoder)
SPLADEv2 (document-only variant)
Simple query, near the original
Full encoder → sparse weighted vector (max pooling)
Document only
+ distillation from a teacher
Less
PromptReps
One LLM inference for each text → dense (hidden state) + sparse (top-k logits)
Through the summary word of the LLM
None (zero-shot). Optional contrastive training helps more.
Yes (LLM inference)
After this week you can do these things:
Put learned sparse retrievers in the model landscape. Explain the efficiency and effectiveness trade-off that they target.
Use the sparse dot product to explain why sparsity gives efficiency.
Explain query and document expansion (with the Lord of the Rings example). Describe the doc2query pipeline. Explain why copied words help, in addition to new words.
Draw the TILDE architecture. Give the three reasons for its speed at query time. Explain the BiQDL loss in words. Describe the three ranking modes, their costs, and the role of α.
Explain the TILDEv2 changes: token embeddings, exact matching and expansion. Give the cost argument for TILDE expansion against doc2query.
Explain how SPLADE changes MLM-head predictions into term weights. Explain the sparsity risk and how ReLU with FLOPS/L1 regularisation gives sparsity back.
Calculate SPLADEv1 (sum) and SPLADEv2 (max) pooled vectors by hand. List the other changes from SPLADEv1 to SPLADEv2.
Compare TILDE and SPLADE: [CLS] and a linear layer against all tokens and the MLM head; sparse by architecture against sparse by training; the difference in speed.
Compare all retriever families on MS MARCO. Give the position of learned sparse models.
Describe PromptReps: the prompt, the source of the dense and sparse representations, and why the hybrid works zero-shot.
Do every worked example again with different numbers.
Next weeks. Week 7 moves from retrieval models to reranking with learning to rank. These are traditional methods, often used on top of term-based lexical models. They are the base of modern reranking, and search companies still use them on top of neural pipelines. The lecture says that the week after that covers reranking with large language models. (At the time of the lecture, Assignment 1 was due on 4 September at 4pm.)
Slide decks: pre-class deck (51 slides, "slides n" in these notes) and in-class lecture deck (27 slides, "lecture slides n"). "Lecture transcript" refers to the in-class lecture transcript (w6-lec-INFS7410_S2_2026_STL-transcript.txt). The pre-class video transcripts are in Pre-class youtube Transcripts/.
13. References
Zhuang, S. and Zuccon, G. (2021). TILDE: Term Independent Likelihood moDEl for passage re-ranking. SIGIR 2021.
Zhuang, S. and Zuccon, G. (2021). Fast passage re-ranking with contextualized exact term matching and efficient passage expansion. arXiv:2108.08513 (TILDEv2).
Formal, T., Piwowarski, B. and Clinchant, S. (2021). SPLADE: Sparse lexical and expansion model for first stage ranking. SIGIR 2021, pp. 2288–2292.
Formal, T., Lassance, C., Piwowarski, B. and Clinchant, S. (2021). SPLADE v2: Sparse lexical and expansion model for information retrieval. arXiv:2109.10086.
Zhuang, S., Ma, X., Koopman, B., Lin, J. and Zuccon, G. (2024). PromptReps: Prompting large language models to generate dense and sparse representations for zero-shot document retrieval. EMNLP 2024.