INFS7410 · Information Retrieval and Web Search · Week 6

Learned Sparse Retrievers: TILDE, TILDEv2, SPLADE & PromptReps

Lexical-based retrieval models with learned term weights. This week combines the efficiency of inverted-index lexical search with the semantic power of transformers, covering query/document expansion (doc2query), the TILDE family, the SPLADE family, an effectiveness comparison across every retriever seen so far, and the training-free LLM-based PromptReps.

1. Where Week 6 fits: the landscape of relevance models

Over the past weeks we have built up a table of relevance model families. Term-based lexical models such as BM25 (Weeks 1–2) match exact words through an inverted index. Single-vector dense retrievers such as RepBERT and ANCE (Week 5) encode a whole query or document into one embedding. Multi-vector dense retrievers such as ColBERT keep one embedding per token. This week we fill in the remaining cell: learned sparse retrievers, also called lexical-based retrieval models with learned term weights. They sit between traditional lexical models and dense embedding models and try to get the best of both.

Model family	Example	Encoder cost	Scorer cost	Representation
Term-based lexical	BM25	$O(1)$ (no neural encoding, just tokenising and counting)	$O(\lt 1)$ — near constant; only documents sharing query terms are touched	Sparse (vocabulary-sized, mostly zeros)
Learned sparse (this week)	TILDE, TILDEv2, SPLADE	Between the two neighbours — depends on the model	Cheap — sparse vectors keep the inverted-index trick	Sparse, but with learned weights
Single-vector dense	RepBERT, ANCE	$O(\|d\|)$ — linear in document length	$O(1)$ — one dot product per document	Dense (e.g. 768 dims)
Multi-vector dense	ColBERT	$O(\|d\|^2)$ — token-level interactions	Expensive — MaxSim across all query–document token pairs	Dense, one vector per token

Reading the costs in plain words "Encoder cost

O(|d|)

" means the work grows in proportion to the number of tokens in the document; "

O(|d|^2)

" means the work grows with the number of token pairs, which is why ColBERT-style models are so much heavier; "

O(1)

" means the cost stays roughly constant no matter how large the collection or document is.

2. Recap: term-based lexical vs dense retrievers

2.1 Term-based lexical models: high efficiency

For the query "machine learning algorithms", a traditional system like BM25 looks only at documents containing the exact words machine, learning, or algorithms. The corpus is organised in an inverted index: for each term, a posting list of the documents that contain it. At query time the engine only loops over the posting lists of the query terms, and this set of candidate documents is usually a tiny fraction of the whole collection. That is the source of the high efficiency of lexical models.

Lexical retrieval only examines the small subsets of the corpus whose documents contain query terms.

2.2 Dense retrievers: high effectiveness

Dense embedding-based retrievers (Week 5) push both queries and documents through transformer encoders to produce dense vectors, and rank by vector similarity. They are much more effective because they do semantic matching: they understand that machine learning relates to artificial intelligence, neural networks or transformers even when no word overlaps. The price is a computationally heavy encoder, and scoring that in principle involves every document vector.

The fundamental trade-off Lexical models: highly efficient, limited in effectiveness (exact matching only). Dense models: highly effective (semantic matching), computationally expensive. The central question of Week 6: can we combine their advantages?

3. The core idea of learned sparse retrievers

Traditional term-based lexical retrievers have three limitations. First, their weighting is based solely on term overlap and collection statistics (term frequency, document frequency, document length). Second, they use hand-crafted formulas designed from intuition and some theoretical foundations. Third, there is no learning from data: apart from a couple of tunable parameters, the same formula is applied to every query, domain, and document type.

Learned sparse models take a different approach:

Learn optimal term weights from training data, so the model adapts to the content, queries, and domain it is trained on.
Use a powerful transformer to predict term importance, which gives far higher effectiveness than simple statistical measures.
Maintain sparsity for efficiency, so scoring can still use the inverted-index trick of only touching documents that share (possibly expanded) terms with the query.

This week presents two main routes to this goal, plus a recent LLM-based hybrid:

Route	Idea in one line	Models
Drop the query encoder	Treat queries and documents asymmetrically: the query is only tokenised into a sparse term-existence vector (no neural network at query time), while documents are still encoded into dense vocabulary-sized vectors offline.	TILDE, TILDEv2
Keep the encoder, output term weights	Use the transformer encoder for both queries and documents, but make it output weights over the whole vocabulary (an expanded set of terms). No encoding time is saved, but scoring is cheap because the vectors are sparse.	SPLADE, SPLADEv2
Prompt an LLM, no training at all	Prompt a generative LLM to summarise a text in one word, and harvest a dense vector (hidden state) plus a sparse vector (next-token logits) from that single inference.	PromptReps

Why sparsity matters In a dot product between a sparse query vector and any document vector, dimensions where the query is zero contribute nothing. If most weights are exactly zero, the engine can precompute posting lists over the non-zero terms and score only documents that share active terms with the query — exactly like BM25. Sparsity is the efficiency.

4. Query & document expansion, and doc2query

4.1 What is expansion?

Query/document expansion means augmenting the user query, or a document in the corpus, with additional relevant terms. The goal is to improve effectiveness by matching related terms that do not appear in the original text — directly attacking the vocabulary mismatch problem (a query says "lower blood sugar", the document says "reduces blood sugar levels"). A variety of automatic and semi-automatic expansion techniques exist; semi-automatic ones require user interaction to select the best expansion terms, automatic ones need no user input. Query/document rewriting is a closely related technique. Many learned sparse retrievers rely on expansion for better effectiveness, which is why we cover it first.

4.2 doc2query: document expansion with a seq2seq transformer

With document expansion we add terms (or modify term weights) in the index to improve performance. doc2query does this with a sequence-to-sequence transformer — a transformer with both an encoder and a decoder that maps input text to output text (T5 is a common example). This differs from BERT, which maps input text to embeddings. The model's task: given a document, generate a query that this document would answer. Training is supervised, using pairs of ⟨query, relevant document⟩ (e.g. from MS MARCO).

The doc2query pipeline: generate 5–40 queries per document, concatenate them onto the document, index the expanded documents as usual.

In practice, between 5 and 40 queries are sampled per document (with top-k or nucleus sampling). Each document is concatenated with its predicted queries to form an expanded document, which is indexed into a traditional inverted index. When a user searches, the engine (e.g. BM25) matches against the expanded index, hopefully bridging the vocabulary gap between query and document.

4.3 Does it work?

Yes — the expansion strategy consistently improves results over the same ranking model on non-expanded documents, regardless of the ranker:

Ranker	MS MARCO Passage (MRR@10)	TREC-DL 19 (nDCG@10)
BM25	0.184	0.506
BM25 + doc2query	0.277	0.642
BM25 + RM3 (pseudo-relevance feedback)	0.156	0.518
BM25 + RM3 + doc2query	0.214	0.655
BM25 + monoBERT/T5 reranking	0.365	0.738
BM25 + monoBERT/T5 + doc2query	0.379	0.754

Inspecting the generated queries shows doc2query does two useful things at once: it injects new words into the document (directly addressing keyword mismatch — e.g. adding weather to a document about temperatures), and it copies/repeats keywords already in the document (about 69% of generated non-stop-words are copied, 31% are new). Repetition matters too: it increases the importance (term frequency) of those keywords in the expanded document.

Ablation (MS MARCO passage)	MRR@10	R@1000
Original document	0.184	0.853
+ expansion with new words only	0.195	0.907
+ expansion with copied words only	0.221	0.893
+ expansion copied + new	0.277	0.944
Only expansions (original document removed)	0.263	0.927

Takeaways from the ablation Both kinds of expansion help, but copied words contribute more than new words (0.221 vs 0.195). Strikingly, indexing only the generated queries with no original document (0.263) beats indexing the original documents alone (0.184).

Worked example 1 How doc2query expansion changes a BM25 score

Setup. A user query $q =$ "lower blood sugar" and a passage $d$ : "Researchers are finding that cinnamon reduces blood sugar levels naturally when taken daily" (13 terms, so $|d| = 13$ ). Assume the collection has $N = 1{,}000{,}000$ passages, average length $avgdl = 15$ , BM25 parameters $k_1 = 1.2$ , $b = 0.75$ , and document frequencies $df(\text{lower}) = 40{,}000$ , $df(\text{blood}) = 25{,}000$ , $df(\text{sugar}) = 20{,}000$ . We use the BM25 form from Week 2:

\mathrm{BM25}(q,d) = \sum_{t \in q} \ln\!\Big(\frac{N - df(t) + 0.5}{df(t) + 0.5} + 1\Big) \cdot \frac{tf(t,d)\,(k_1+1)}{tf(t,d) + k_1\big(1 - b + b\,\frac{|d|}{avgdl}\big)}

Formula in plain words The score is the sum over every query term of two factors: an IDF factor (the log of roughly "how rare the term is across the collection", so rarer terms count more) multiplied by a saturating term-frequency factor (the more often the term appears in the document the higher the factor, but with diminishing returns, and adjusted for whether the document is longer or shorter than average).

Score the original document. Term frequencies in

d

tf(\text{lower}) = 0

tf(\text{blood}) = 1

tf(\text{sugar}) = 1

. The word "lower" never appears — the passage says "reduces". Its contribution is 0: this is vocabulary mismatch.

Compute the shared pieces. Length normalisation:

1 - b + b\frac{|d|}{avgdl} = 1 - 0.75 + 0.75 \times \frac{13}{15} = 0.25 + 0.65 = 0.90

. So the TF factor for a term with

tf = 1

\frac{1 \times 2.2}{1 + 1.2 \times 0.90} = \frac{2.2}{2.08} = 1.058

Compute IDFs.

\mathrm{idf}(\text{blood}) = \ln\big(\frac{1{,}000{,}000 - 25{,}000 + 0.5}{25{,}000 + 0.5} + 1\big) = \ln(40.0) = 3.689

\mathrm{idf}(\text{sugar}) = \ln\big(\frac{980{,}000}{20{,}000.5} + 1\big) = \ln(50.0) = 3.912

\mathrm{idf}(\text{lower}) = \ln\big(\frac{960{,}000}{40{,}000.5} + 1\big) = \ln(25.0) = 3.219

(unused for now since

tf = 0

Original score.

\mathrm{BM25} = 3.689 \times 1.058 + 3.912 \times 1.058 = 3.903 + 4.139 = \mathbf{8.04}

. The term "lower" contributed nothing.

Expand the document. doc2query predicts the query "does cinnamon lower blood sugar" (5 terms), which we append. The expanded document

d'

has

|d'| = 18

terms and new counts:

tf(\text{lower}) = 1

tf(\text{blood}) = 2

tf(\text{sugar}) = 2

. Note both effects from the lecture: a new word ("lower") was injected, and existing words ("blood", "sugar") were copied, raising their term frequency.

Recompute the pieces for $d'$ . Length normalisation:

0.25 + 0.75 \times \frac{18}{15} = 0.25 + 0.90 = 1.15

, so the denominator constant is

k_1 \times 1.15 = 1.38

. TF factor for

tf = 1

\frac{2.2}{1 + 1.38} = 0.924

. TF factor for

tf = 2

\frac{2 \times 2.2}{2 + 1.38} = \frac{4.4}{3.38} = 1.302

— larger than for

tf=1

, but less than double: BM25's term frequency saturates.

Expanded score.

\mathrm{BM25} = \underbrace{3.219 \times 0.924}_{\text{lower}} + \underbrace{3.689 \times 1.302}_{\text{blood}} + \underbrace{3.912 \times 1.302}_{\text{sugar}} = 2.974 + 4.803 + 5.093 = \mathbf{12.87}

Result Expansion lifted the document's score for this query from 8.04 to 12.87, and — more importantly — the previously unmatched query term "lower" now matches. Nothing changed at query time: the expansion work happened once, offline, at indexing. This is exactly the mechanism behind the 0.184 → 0.277 MRR@10 jump for BM25 on MS MARCO.

5. TILDE: Term Independent Likelihood moDEl

5.1 The idea: replace the query encoder with a tokeniser

TILDE (Zhuang & Zuccon, SIGIR 2021) exploits the first route from Section 3: remove query encoding entirely. In a dense bi-encoder, both towers run BERT. TILDE keeps the document tower but replaces the query tower with the plain BERT tokeniser:

Query: tokenised into a sparse vector of size $|V| = 30{,}522$ (the BERT vocabulary). Element $i$ is 1 if token $t_i$ appears in the query and 0 otherwise — a term-existence vector, like exact matching but over the BERT vocabulary instead of the collection vocabulary.
Document: encoded by BERT; the [CLS] token embedding (768 dims) is projected by a linear layer up to a dense vector of size $|V| = 30{,}522$ . This can be fully precomputed offline at indexing time.
Score: the inner product between the sparse query vector and the dense document vector, in the $|V|$ -dimensional space.

s(q, d) = \mathbf{q}^{\top} \mathbf{d} = \sum_{i=1}^{|V|} q_i \, d_i = \sum_{t_i \in q} d_i

Formula in plain words The score is the sum, over all vocabulary tokens, of the query entry times the document entry. Because query entries are only 0 or 1, this collapses to: add up the document's learned weights at exactly the positions of the query's tokens. With a 5-token query that is 5 additions — extremely fast.

TILDE: the query side is just a tokeniser producing a binary sparse vector; the document side is BERT's [CLS] embedding projected to vocabulary size.

Two key observations (1) Semantic matching survives. Even if a query keyword never appears in the document, the document's dense vector (produced by BERT) can still assign that keyword a high weight — the model has learned which vocabulary terms the document is about. (2) No GPU at query time. The tokeniser is a small lookup table, so query "encoding" is nearly free and runs fine on a CPU.

5.2 The training loss: BiQDL

Another key difference from other bi-encoders is the loss. Instead of a margin or contrastive loss (Week 5), TILDE is trained with the traditional query likelihood paradigm from statistical language models, assuming query tokens are independent of each other (the same assumption query likelihood models make). The bi-directional query-document likelihood loss (BiQDL) has two directions.

Direction 1 — query likelihood (QL). The model reads the document text and must predict which vocabulary tokens appear in the query:

\mathcal{L}_{QL}(D) = - \sum_{(q,d) \in D} \frac{1}{|V|} \sum_{i}^{|V|} \Big[ y \log\big(P_\theta(t_i \mid d)\big) + (1-y)\log\big(1 - P_\theta(t_i \mid d)\big) \Big]

y = \begin{cases} 1, & \text{if } t_i \text{ in } q, \\ 0, & \text{otherwise.} \end{cases}

Formula in plain words For every ⟨query, relevant document⟩ pair in the training set

D

, take the average over all $|V|$ vocabulary tokens of a binary cross-entropy: if the token does occur in the query (

y = 1

), reward the model for assigning it a high probability given the document (the

\log P_\theta(t_i \mid d)

part); if it does not occur (

y = 0

), reward the model for assigning it a low probability (the

\log(1 - P_\theta)

part). Sum this over all training pairs and negate, since we minimise the loss. In short: push up the probability of query tokens, push down the probability of every other token.

Direction 2 — document likelihood (DL). Simply swap the roles: the query is encoded by BERT into a dense representation, and the document becomes the sparse tokenised side:

\mathcal{L}_{DL}(D) = - \sum_{(q,d) \in D} \frac{1}{|V|} \sum_{i}^{|V|} \Big[ y \log\big(P_\theta(t_i \mid q)\big) + (1-y)\log\big(1 - P_\theta(t_i \mid q)\big) \Big]

Formula in plain words Identical structure, mirrored: given the query text, push up the probability of tokens that appear in the document and push down the probability of tokens that do not (here

y = 1

t_i

is in

d

Combined. The bi-directional loss is the average of both directions:

\mathcal{L}_{BiQDL}(D) = \frac{\mathcal{L}_{QL}(D) + \mathcal{L}_{DL}(D)}{2}

Formula in plain words The final training loss is the sum of the query-likelihood loss and the document-likelihood loss, divided by two — an equal-weight average of the two directions.

After training with BiQDL, TILDE can estimate the query likelihood given a document text and the document likelihood given a query text.

5.3 Three ways of ranking with TILDE

These two abilities give three ranking settings that trade efficiency for effectiveness:

\text{TILDE-QL}(q \mid d^k) = \sum_i^{|q|} \log\big(P_\theta(q_i \mid d^k)\big)

Formula in plain words Rank document

d^k

by the sum, over each token in the query, of the log of the probability the document assigns to that token. Because document representations are precomputed, this is just a few lookups and additions — no GPU at query time. (Logs of probabilities are negative numbers; scores closer to zero are better.)

\text{TILDE-DL}(d^k \mid q) = \frac{1}{|d^k|} \sum_i^{|d^k|} \log\big(P_\theta(d_i^k \mid q)\big)

Formula in plain words Rank by the average, over each token of the document, of the log-probability of that token given the query. The division by document length

|d^k|

normalises the score, because longer documents contain more tokens and would otherwise accumulate lower (more negative) likelihoods. This direction needs a BERT inference on the query at search time, so a GPU is required.

\text{TILDE-QDL}(q, d^k) = \alpha \cdot \text{TILDE-QL}(q \mid d^k) + (1 - \alpha) \cdot \text{TILDE-DL}(d^k \mid q)

Formula in plain words The final score is a weighted mix of the two directions: a fraction

\alpha

of the query-likelihood score plus the remaining fraction

1 - \alpha

of the document-likelihood score. The hyperparameter

\alpha \in [0,1]

controls the balance.

5.4 Impact of the document likelihood signal

Plotting nDCG@10 and MAP against $\alpha$ (on TREC DL 2019/2020) shows:

$\alpha = 0$ (document likelihood only): performance is very poor (nDCG@10 ≈ 0.23) — documents cannot be ranked with document likelihood alone.
$\alpha = 0.4$ – $0.5$ : best performance (nDCG@10 ≈ 0.61–0.62) — document likelihood is beneficial when mixed in.
$\alpha = 1$ (pure query likelihood): a small drop from the best (≈ 0.58–0.62), but this setting has minimum query latency because no GPU inference is needed at query time.

Worked example 2 Scoring with TILDE's sparse × dense inner product

Setup. To keep the numbers readable, use a toy vocabulary of 8 BERT tokens instead of 30,522: $V = \{\text{apple},\ \text{store},\ \text{account},\ \text{fruit},\ \text{buy},\ \text{banana},\ \text{login},\ \text{the}\}$ . The user query is "apple account". Two candidate passages were encoded offline; their [CLS] embeddings, after the linear projection, give these dense vocabulary-space weights (each row is $P_\theta(t_i \mid d)$ ):

	apple	store	account	fruit	buy	banana	login	the
$d_1$ : "how to sign in to your Apple ID"	0.62	0.28	0.51	0.04	0.19	0.02	0.44	0.30
$d_2$ : "apples are a healthy fruit to snack on"	0.66	0.07	0.03	0.58	0.21	0.35	0.01	0.31

Tokenise the query (no GPU). "apple account" activates dimensions apple and account:

\mathbf{q} = [1, 0, 1, 0, 0, 0, 0, 0]

. Every other dimension is 0.

Inner product with $d_1$ . Only the two active dimensions matter:

s(q, d_1) = 1 \times 0.62 + 1 \times 0.51 = \mathbf{1.13}

Inner product with $d_2$ .

s(q, d_2) = 1 \times 0.66 + 1 \times 0.03 = \mathbf{0.69}

Rank.

d_1 \succ d_2

. Notice the semantic matching at work:

d_1

never contains the literal word "account", yet BERT learned that a passage about "signing in to your Apple ID" is about accounts, so its dense vector carries a high weight (0.51) on the account dimension. A purely lexical matcher would have missed this. Also notice

d_1

's weight of 0.44 on "login" — a term not in the passage either: this is implicit expansion in the document representation.

Equivalent TILDE-QL view. Using log-likelihoods instead of raw probabilities:

\text{TILDE-QL}(q \mid d_1) = \ln 0.62 + \ln 0.51 = -0.478 - 0.673 = -1.151

, and

\text{TILDE-QL}(q \mid d_2) = \ln 0.66 + \ln 0.03 = -0.416 - 3.507 = -3.923

. Since

-1.151 > -3.923

, the ranking is the same:

d_1

first.

Result Scoring each document cost exactly two additions (one per query token) after a tokeniser lookup — this is why TILDE's query latency is tiny and GPU-free, while still rewarding a document that matches the query only semantically.

Worked example 3 Computing the BiQDL loss on a tiny vocabulary

Setup. One training pair: query $q =$ "apple account", relevant document $d =$ "apple store account help". Toy vocabulary $V = \{\text{apple}, \text{store}, \text{account}, \text{banana}\}$ , so $|V| = 4$ . The current model, reading $d$ , predicts these probabilities of each token appearing in the query: $P_\theta(\text{apple} \mid d) = 0.7$ , $P_\theta(\text{store} \mid d) = 0.5$ , $P_\theta(\text{account} \mid d) = 0.3$ , $P_\theta(\text{banana} \mid d) = 0.2$ . Reading $q$ , it predicts these probabilities of each token appearing in the document: $P_\theta(\text{apple} \mid q) = 0.8$ , $P_\theta(\text{store} \mid q) = 0.4$ , $P_\theta(\text{account} \mid q) = 0.6$ , $P_\theta(\text{banana} \mid q) = 0.1$ . (All logs are natural logs.)

Build the label vector for the QL direction.

y_i = 1

if token

t_i

is in the query. Query tokens: {apple, account}. So

y = [1, 0, 1, 0]

for (apple, store, account, banana).

Per-token QL terms. For tokens in the query use

\log P

; for tokens not in the query use

\log(1-P)

:
apple (

y=1

\log 0.7 = -0.357

store (

y=0

\log(1 - 0.5) = \log 0.5 = -0.693

account (

y=1

\log 0.3 = -1.204

banana (

y=0

\log(1 - 0.2) = \log 0.8 = -0.223

Average and negate. Sum

= -0.357 - 0.693 - 1.204 - 0.223 = -2.477

. Divide by

|V| = 4

-0.619

. Negate:

\mathcal{L}_{QL} = \mathbf{0.619}

. (The biggest contributor is "account": the model gave a true query token only probability 0.3, and gave the non-query token "store" a too-high 0.5 — training will push

P(\text{account} \mid d)

up and

P(\text{store} \mid d)

down.)

Label vector for the DL direction. Now

y_i = 1

t_i

is in the document "apple store account help". Document tokens within our toy

V

: {apple, store, account}, so

y = [1, 1, 1, 0]

Per-token DL terms.
apple:

\log 0.8 = -0.223

· store:

\log 0.4 = -0.916

· account:

\log 0.6 = -0.511

· banana:

\log(1 - 0.1) = -0.105

Average and negate. Sum

= -1.755

; divided by 4:

-0.439

; negated:

\mathcal{L}_{DL} = \mathbf{0.439}

Combine.

\mathcal{L}_{BiQDL} = \frac{0.619 + 0.439}{2} = \mathbf{0.529}

Result The BiQDL loss for this pair is 0.529. Minimising it simultaneously teaches the model to predict query tokens from documents (QL direction) and document tokens from queries (DL direction) — which is precisely what lets a trained TILDE rank with query likelihood, document likelihood, or both.

Worked example 4 TILDE-QDL interpolation with α

Setup. For a query $q$ and two candidate passages, a trained TILDE gives: $\text{TILDE-QL}(q\mid d_1) = -1.9$ , $\text{TILDE-DL}(d_1\mid q) = -0.6$ ; and $\text{TILDE-QL}(q\mid d_2) = -1.6$ , $\text{TILDE-DL}(d_2\mid q) = -1.4$ . (All are log-likelihoods, so higher — closer to zero — is better. The DL scores are already length-normalised.)

Pure query likelihood, $\alpha = 1$ . Scores are just QL:

d_1 : -1.9

d_2 : -1.6

→ ranking

d_2 \succ d_1

. Fastest setting: no GPU needed.

Pure document likelihood, $\alpha = 0$ . Scores are just DL:

d_1 : -0.6

d_2 : -1.4

→ ranking

d_1 \succ d_2

. The lecture showed this setting alone performs badly overall.

Interpolate at the sweet spot $\alpha = 0.4$ .

d_1: 0.4 \times (-1.9) + 0.6 \times (-0.6) = -0.76 - 0.36 = \mathbf{-1.12}

d_2: 0.4 \times (-1.6) + 0.6 \times (-1.4) = -0.64 - 0.84 = \mathbf{-1.48}

Rank.

-1.12 > -1.48

, so

d_1 \succ d_2

. The document-likelihood evidence (which strongly favoured

d_1

) flipped the ranking relative to pure QL.

Result α controls a real trade-off:

\alpha = 1

gives minimum latency (tokeniser only), while

\alpha \approx 0.4\text{–}0.5

gives the best effectiveness at the cost of one BERT inference on the query at search time (GPU needed). The empirical plots in the lecture show the effectiveness gap between these two settings is quite small.

6. TILDEv2: contextualised exact matching + cheap expansion

6.1 What changes from TILDE

TILDEv2 keeps the BERT-tokeniser sparse query encoding, but changes the document side in two ways:

Token embeddings instead of [CLS]. The document is represented by its contextualised token embeddings, not one pooled [CLS] vector. Scoring computes the inner product between the sparse query vector and these token embeddings — in practice this only scores exact matches between query terms and tokens of the (expanded) document, but each matched token carries a contextualised learned weight rather than a count. This handles the semantic-gap problem through the contextualised weights.
Document expansion before encoding. To overcome query–document vocabulary mismatch (which pure exact matching would suffer from), the document is expanded before encoding — using doc2query, or using TILDE itself: a trained TILDE can generate the most likely query terms for a document, just as doc2query does, but far more cheaply.

TILDEv2: exact term matching between the sparse query vector and contextualised token weights of the expanded document. Here "account" only exists in the document because expansion added it.

Worked example 5 Scoring a query with TILDEv2

Setup. Query: "apple account". Original passage $d$ : "apple store". At indexing time, TILDE expansion proposed the extra tokens {account, iphone} (likely query terms for this passage); after filtering (see 6.2), both were appended. BERT then encoded the expanded passage and produced one learned weight per token: apple → 4.7, store → 0.2, account → 3.1, iphone → 2.4.

Tokenise the query. Active terms: {apple, account}. This is the sparse query vector; nothing else will be scored.

Exact-match lookup. For each query term, find matching tokens in the expanded document and take their contextualised weights: "apple" matches with weight 4.7; "account" matches with weight 3.1 — only because expansion added it. (If a term appeared several times, we would take its best/matched occurrence weights as stored in the index.)

Sum the matched weights.

s(q,d) = 4.7 + 3.1 = \mathbf{7.8}

Contrast with the unexpanded passage. Without expansion, "account" matches nothing and

s = 4.7

: the passage would rank as if it had nothing to say about accounts. And contrast with plain BM25: BM25 would give "apple" a weight from counts and rarity alone, while TILDEv2's 4.7 is contextualised — the same word "apple" in a fruit passage would get a different weight than in this tech-support passage.

Result TILDEv2 = exact term matching (fast, inverted-index friendly) + learned contextualised term weights (semantics) + offline expansion (fixes vocabulary mismatch). Scoring cost is again a handful of additions per document.

6.2 How important is the expansion component?

Empirical results (MRR@10, MS MARCO dev) as the expansion method varies. When TILDE generates $m$ candidate terms, only terms that are not stop words and not already in the document are actually added (the same filtering is applied to doc2query):

	No expansion	doc2query	TILDE m=128	TILDE m=150	TILDE m=200
MRR@10	0.299	0.333	0.326	0.327	0.330
Avg added tokens	–	19.0	13.0	25.2	61.6
Index size	4.3 GB	5.2 GB	5.2 GB	5.6 GB	6.9 GB
Expansion cost	–	320 hours, $768 (TPU)	7.22 hours, $5.34 (GPU)	7.25 hours, $5.37	7.33 hours, $5.42

Reading the table: expansion always beats no expansion (0.299 → 0.32+). doc2query gives the best effectiveness (0.333); TILDE needs many more added terms to get close (m = 200 adds 61.6 tokens on average vs doc2query's 19), which also grows the index (6.9 GB vs 5.2 GB) — suggesting TILDE's individual generated terms are somewhat weaker than doc2query's. But that is not the whole story:

The efficiency story Expanding all of MS MARCO with doc2query took 320 hours on Google TPUs ≈ $768 (TPUs are ~15× more powerful than GPUs, but expensive). TILDE expansion runs on cheap GPUs in about 7¼ hours for ~$5. For MS MARCO v2 (15× bigger), doc2query would need ~200 days and > $11,000, versus ~4.5 days and ~$80 for TILDE — and the gap keeps growing at commercial-search-engine scale (Google, Bing, Baidu). Offline does not mean free.

7. SPLADE: expansion and weighting from the MLM head

7.1 Intuition

SPLADE (Formal, Piwowarski & Clinchant, SIGIR 2021) takes the second route: keep the transformer encoder for both queries and documents, but use it to predict term-importance weights over the entire vocabulary. For the query "Who wrote the book of the lord of the rings", a traditional lexical model can only match those exact words. SPLADE instead activates related vocabulary terms with learned importance weights — for example:

Activated term	write	created	authored	novel	LOTR	Tolkien
Contribution	3.5	1.2	3.3	3.3	5	4

None of these terms needs to be in the query — Tolkien gets weight 4 purely from semantic association — and yet the vector is still sparse: nearly all of the 30,522 vocabulary dimensions stay at zero. If we can predict the importance of highly related vocabulary terms for their contribution to ranking, we get a much better lexical term-based retriever. The weights come from the transformer, not from collection statistics.

7.2 The mechanism: reuse BERT's MLM head

Recall BERT's pre-training task from earlier weeks: masked language modelling (MLM) — predict a [MASK] token from its surrounding context. The MLM head outputs, at each token position, a probability distribution over all 30,522 vocabulary terms, and those distributions encode exactly the semantic association we need (contextually related words get high probability). SPLADE's key innovation: run the input through BERT with its MLM head, obtain one vocabulary-sized probability distribution per input token position, and then aggregate across positions with sum pooling to get a single importance vector:

w_j = \sum_{i=1}^{n} p_{ij} \qquad j = 1, \dots, |V|

Formula in plain words The importance weight of vocabulary term

j

is the sum, over every token position $i$ of the input text (of length $n$ ), of the probability that position $i$ 's MLM distribution assigns to term $j$ . A vocabulary term becomes important if many positions of the text "point at it" — even if the term itself never occurs in the text. This gives one vocabulary-sized vector per query or document.

SPLADE: each token position yields an MLM probability distribution over the vocabulary; pooling across positions gives the term-importance vector.

7.3 The sparsity problem

There is a catch. Every token position assigns some probability to every vocabulary term, however small. Summing them means every one of the 30,522 dimensions is non-zero — which is a dense 30,522-dimensional vector, not a sparse one, and the inverted-index efficiency evaporates. SPLADE restores sparsity with several techniques:

ReLU activation — zero out all negative raw weights (only positively-activated terms survive);
FLOPS regularisation and L1 regularisation — training penalties that push most weights to exactly zero, keeping only the important terms active.

Scope note The mathematical details of the FLOPS/L1 regularisers are not covered in this course (the SPLADE paper is cited on the slides for interested readers). What matters is the idea: these techniques force most term weights to exactly zero while keeping important terms active, so the learned representation stays sparse and efficient.

7.4 SPLADE core ideas, summarised

Every vocabulary term can get a weight — unlike traditional models that only weight terms present in the text.
A BERT-based model predicts term importance given the query or document, reusing all the semantic knowledge from pre-training.
Implicit expansion: weights are not tied to the original text's terms, so related terms get activated and help ranking.
Sparsity is maintained through ReLU + regularisation, preserving the efficiency of sparse representations.

Worked example 6 SPLADE pooling (sum vs max) and scoring

Setup. Document: "apple reduces sugar" (3 token positions). Toy vocabulary of 6 terms. The MLM head gives each position a probability distribution over the vocabulary (rows sum to 1):

Position	apple	fruit	reduces	lowers	sugar	car
pos 1 · "apple"	0.50	0.30	0.02	0.02	0.10	0.06
pos 2 · "reduces"	0.05	0.05	0.45	0.30	0.10	0.05
pos 3 · "sugar"	0.10	0.15	0.05	0.05	0.60	0.05

Sum pooling (SPLADEv1). Add each column:
apple:

0.50 + 0.05 + 0.10 = 0.65

· fruit:

0.30 + 0.05 + 0.15 = 0.50

· reduces:

0.02 + 0.45 + 0.05 = 0.52

· lowers:

0.02 + 0.30 + 0.05 = 0.37

· sugar:

0.10 + 0.10 + 0.60 = 0.80

· car:

0.06 + 0.05 + 0.05 = 0.16

.
Document vector (sum):

[0.65, 0.50, 0.52, 0.37, 0.80, 0.16]

. Note fruit (0.50) and lowers (0.37) got real weight although neither word is in the document — implicit expansion.

Max pooling (SPLADEv2). Take each column's largest value across positions:
apple:

\max(0.50, 0.05, 0.10) = 0.50

· fruit: 0.30 · reduces: 0.45 · lowers: 0.30 · sugar: 0.60 · car: 0.06.
Document vector (max):

[0.50, 0.30, 0.45, 0.30, 0.60, 0.06]

. Max pooling keeps the strongest single signal per term instead of accumulating many weak ones — notice how "car", which no position really supports, drops from 0.16 to 0.06 relative to the others.

Sparsify. During training, ReLU plus FLOPS/L1 regularisation would push tiny weights to exactly zero. Suppose weights below 0.2 are zeroed: sum-pooled vector →

[0.65, 0.50, 0.52, 0.37, 0.80, 0]

. "car" is now inactive, so this document never even enters the posting list for "car".

Encode a query the same way. Query "lowers sugar" passes through the same SPLADE encoder; suppose its sparse vector comes out as

[0, 0, 0.3, 0.9, 1.1, 0]

(activating reduces by expansion, plus lowers and sugar).

Score with a dot product over shared non-zero terms. Using the sparsified sum-pooled document vector:

s(q,d) = 0.3 \times 0.52 + 0.9 \times 0.37 + 1.1 \times 0.80 = 0.156 + 0.333 + 0.880 = \mathbf{1.369}

. Only 3 of 6 dimensions were touched — with a real vocabulary, only the few dozen active terms matter, which is what makes SPLADE scoring inverted-index fast.

Result The query word "lowers" never appears in the document and the document word "reduces" never appears in the query, yet the two matched through their expanded, weighted representations — semantic matching carried out entirely inside a sparse, lexical-style dot product.

8. SPLADEv2: what changed

SPLADEv2 (same research group) made three main improvements:

	SPLADEv1	SPLADEv2
Pooling	Sum pooling to aggregate token distributions	Max pooling — for each vocabulary dimension, take the highest probability across all input token positions; empirically captures the most important signal better
Training loss	Basic contrastive training	Adds distillation loss — the model also learns from a teacher model (not just relevance labels), improving weighting and expansion
Weighting & expansion	Both query and document are weighted and expanded → full SPLADE inference needed at query time	New variant with document-only weighting and expansion (similar in spirit to TILDE) → queries stay simple and close to their original form

The document-only variant matters because the asymmetric argument from TILDE applies here too: document expansion happens offline at indexing time, where there is plenty of time to process thoroughly, while query processing happens online while a user waits — keeping the query side simple means much faster search response times. If most of the semantic-matching benefit comes from the document side anyway, dropping query-side expansion buys efficiency at little cost.

9. Effectiveness comparison of all retrievers so far

All numbers below are on the MS MARCO dev set with MRR@10 (mean reciprocal rank over the top 10 — standard for MS MARCO dev because its relevance judgements are sparse, usually one judged relevant passage per query). Values are approximate readings from the lecture's chart.

MRR@10 on MS MARCO dev (approximate values read from the lecture chart). Grey: term-based lexical. Blue: encoder-based dense (single- and multi-vector). Orange: decoder-based dense. Green: learned sparse.

Walking through the chart the way the lecture does:

BM25 (≈ 0.18) — the traditional baseline: fast and efficient, limited by exact matching.
Single-vector dense — RepBERT, ANCE (≈ 0.33) — nearly double BM25; the improvement that made the field take notice of transformer-based retrieval. Semantic matching does the work.
Multi-vector dense — ColBERT (≈ 0.39) — better again, thanks to one vector per token and finer-grained matching; much slower to encode and score than single-vector models.
Decoder-based — RepLLaMA, LLM2Vec (≈ 0.41–0.42) — the most recent advances, but the backbones are ~7B-parameter decoders (vs BERT's 110M), with higher-dimensional vectors, so both encoding and scoring cost far more.
Learned sparse — BM25+TILDE ≈ 0.27, BM25+TILDEv2 ≈ 0.33, SPLADEv1 ≈ 0.32, SPLADEv2 ≈ 0.34 — a large jump over BM25 (TILDEv2 improves substantially over TILDE), approaching single-vector dense effectiveness. SPLADE lands a little above TILDE-family models (it is also a later model), but is less efficient than TILDE because it still runs a query encoder at search time.

The overall insight Learned sparse models do not quite match the very best dense retrievers on effectiveness, but they are generally much more efficient: like traditional lexical models they only score documents that share (expanded) terms with the query, giving fast search and low computational requirements. They occupy a sweet spot — significantly better effectiveness than BM25 while keeping much of the efficiency that makes lexical models practical at large scale.

10. PromptReps: training-free dense + sparse representations from LLMs

10.1 Why another approach?

The LLM-based dense retrievers from Week 5 all need training to become retrievers:

E5(-Mistral): takes the embedding from the last hidden layer of an [EOS] token appended to the query/document; requires (supervised) contrastive fine-tuning (it can also use LLM-generated synthetic data).
LLM2Vec: takes the embedding from mean pooling over all sequence tokens' last hidden states; requires (unsupervised or supervised) contrastive fine-tuning.

Contrastive fine-tuning is expensive twice over: you need labels (or synthetic data), and you need serious GPU infrastructure — the larger the decoder backbone, the longer the training and the more memory required. The question PromptReps (Zhuang et al., EMNLP 2024) asks: can we engineer prompts so the LLM produces an effective retrieval representation with no contrastive fine-tuning at all? — "zero-shot" generation of representations: no training beyond the LLM's own pre-training, no in-context examples either.

10.2 How it works

Prompt the LLM to represent the given text with a single word:

<System> You are an AI assistant that can understand human language.
<User> Passage: "[text]". Use one word to represent the passage in a retrieval task. Make sure your word is in lowercase.
<Assistant> The word is: "

Run one inference and harvest two representations from the position of that final quote character — the position where the model is about to generate its one summarising word:

Dense: the last hidden state at that position (it encodes the word the model is about to say) → used for dense retrieval via an ANN index.
Sparse: continue through the LM head to get the next-token logits — a distribution of probability mass over the LLM's entire vocabulary — and use that as a sparse vector → used for classic inverted-index matching.

One inference, two representations: the hidden state gives a dense vector; the next-token logits give a sparse vector over the LLM vocabulary. Their scores are combined linearly.

At retrieval time, the dense representation performs dense retrieval and the sparse representation performs typical inverted-index matching; each gives a score, and the final relevance score combines them linearly (an interpolation) — a hybrid retriever.

10.3 Does zero-shot retrieval work?

On MS MARCO (MRR@10), with everything zero-shot (no in-context examples, no contrastive training of the representations):

System	MRR@10 (approx.)	Notes
LLM2Vec (unsupervised, SimCSE-trained baseline)	≈ 13.5	still needs unsupervised contrastive training
PromptReps (Dense only)	≈ 17.5	below BM25
BM25 (dotted reference line)	≈ 18.5	no training either, of course
PromptReps (Sparse only)	≈ 21	above BM25
PromptReps (Hybrid)	≈ 24.5	combining dense + sparse is the key

Main finding Combining the dense and sparse representations allows very effective zero-shot retrieval — the hybrid clearly beats BM25 and the unsupervised LLM2Vec baseline, with no training of the representations at all.

Extension — more than one word. The prompt can ask for two words, three words, a sentence, a paragraph… which requires multiple autoregressive inferences and yields multiple token embeddings/distributions. These can be aggregated (max pooling over the sparse distributions, mean pooling over the dense vectors) or even used ColBERT-style with MaxSim between query-side and passage-side multi-token representations. The lecture does not go deeper here, because these multi-token paragraph/passage representations turn out not to be very effective.

Worked example 7 Combining dense and sparse scores in a hybrid retriever

Setup. A query is issued to a PromptReps system. Three candidate passages receive a dense score (cosine similarity from the ANN index) and a sparse score (inverted-index match on the logits vectors, BM25-style scale):

Passage	Dense score $s_{dense}$	Sparse score $s_{sparse}$
$d_1$	0.82	6.0
$d_2$	0.78	12.0
$d_3$	0.70	9.0

The two scores live on different scales (cosine ∈ [−1, 1]; the sparse dot product is unbounded), so we first min–max normalise each within the candidate list, then interpolate with weight $\lambda = 0.5$ :

\tilde{s} = \frac{s - s_{\min}}{s_{\max} - s_{\min}}, \qquad s_{hybrid} = \lambda\, \tilde{s}_{dense} + (1 - \lambda)\, \tilde{s}_{sparse}

Formulas in plain words First rescale each score list so its smallest value becomes 0 and its largest becomes 1 (subtract the minimum, then divide by the range). Then the hybrid score is a weighted sum of the two normalised scores — a fraction

\lambda

of the dense score plus the remaining fraction of the sparse score.

Normalise dense scores. Min = 0.70, max = 0.82, range = 0.12.

d_1: \frac{0.82-0.70}{0.12} = 1.00

d_2: \frac{0.78-0.70}{0.12} = 0.667

d_3: \frac{0.70-0.70}{0.12} = 0.00

Normalise sparse scores. Min = 6.0, max = 12.0, range = 6.0.

d_1: \frac{6-6}{6} = 0.00

d_2: \frac{12-6}{6} = 1.00

d_3: \frac{9-6}{6} = 0.50

Interpolate with $\lambda = 0.5$ .

d_1: 0.5 \times 1.00 + 0.5 \times 0.00 = \mathbf{0.50}

d_2: 0.5 \times 0.667 + 0.5 \times 1.00 = \mathbf{0.83}

d_3: 0.5 \times 0.00 + 0.5 \times 0.50 = \mathbf{0.25}

Rank.

d_2 \succ d_1 \succ d_3

. Neither signal alone gives this ranking: dense alone would put

d_1

first, sparse alone would put

d_1

last. The hybrid rewards

d_2

for being strong on both signals — which is exactly why PromptReps (Hybrid) outperforms both PromptReps (Dense) and PromptReps (Sparse) on MS MARCO.

Result A linear interpolation of normalised dense and sparse scores produced a better ranking than either representation alone, with zero training of the representations.

11. Summary and study checklist

Model	Query side (online)	Document side (offline)	Expansion	Training signal	GPU at query time?
BM25 + doc2query	Plain terms	Text + generated queries in a normal inverted index	Explicit, generated by seq2seq (T5)	Supervised ⟨query, doc⟩ pairs (for doc2query)	No
TILDE	BERT tokeniser → binary sparse vector	[CLS] → linear → dense \|V\| vector	Implicit (dense doc vector covers whole vocabulary)	BiQDL (bi-directional query/document likelihood)	No (with α = 1); yes if using DL
TILDEv2	BERT tokeniser → binary sparse vector	Contextualised token weights of the expanded document	Explicit, via doc2query or TILDE (cheap)	Contextualised exact-match training	No
SPLADE	Full encoder → sparse weighted vector	Full encoder → sparse weighted vector	Implicit, via MLM head over vocabulary	Contrastive + ReLU/FLOPS/L1 for sparsity	Yes (query encoder)
SPLADEv2 (doc-only variant)	Simple, near-original query	Full encoder → sparse weighted vector (max pooling)	Document-only	+ distillation from a teacher	Reduced
PromptReps	One LLM inference each → dense (hidden state) + sparse (logits)		Via the LLM's summarising word	None (zero-shot)	Yes (LLM inference)

You should now be able to:

Place learned sparse retrievers in the relevance-model landscape and explain the efficiency/effectiveness trade-off they target.
Explain query/document expansion and describe the doc2query pipeline, including why copied words help as well as new words.
Draw the TILDE architecture, explain why query encoding needs no GPU, write down and explain the BiQDL loss, and describe the three ranking modes and the role of α.
Explain what TILDEv2 changes (token embeddings, exact matching, expansion) and argue the cost case for TILDE expansion over doc2query.
Explain how SPLADE turns MLM-head predictions into term weights, why sparsity is at risk, and how ReLU + FLOPS/L1 regularisation restore it; list the SPLADEv1 → v2 changes.
Compare all retriever families on MS MARCO and articulate where learned sparse models sit.
Describe PromptReps: the prompt, where the dense and sparse representations come from, and why the hybrid works zero-shot.
Reproduce every worked example above with different numbers.