INFS7410 · Information Retrieval and Web Search · Week 5

Exercise 3: InfoNCE loss, worked solution

Step-by-step calculation of the InfoNCE (negative log likelihood) loss for one query with one positive and three negative documents: the training loss of DPR, ANCE, Contriever, ColBERT and RepLLaMA. The page states the formula, works the numbers, and then checks how the loss moves when the model improves.

1. The problem and the InfoNCE formula

Sources: slides 19, 21, 31 · Video 2 (DPR), 02:11–05:50 · Week 5 in-class exercise 3 (Week 5 lecture notes, in-class exercises)

Given the following similarity scores for a query qq, calculate the InfoNCE loss. The similarity is the dot product of the query embedding and the document embedding (slide 21), as in DPR's bi-encoder.

CandidateRoleSimilarity to q
p⁺positive (relevant) document0.9
p₁⁻negative document0.3
p₂⁻negative document0.1
p₃⁻negative document0.4
L=−log⁡esim(q,p+)esim(q,p+)+∑j=1nesim(q,pj−) L = -\log \frac{e^{\mathrm{sim}(q,p^+)}}{e^{\mathrm{sim}(q,p^+)} + \sum_{j=1}^{n} e^{\mathrm{sim}(q,p_j^-)}}
Where
LL
the InfoNCE loss for this one query (also called the negative log likelihood loss)
qq
the query embedding
p+p^+
the embedding of the positive (relevant) document
pj−p_j^-
the embedding of the jj-th negative (non-relevant) document
nn
number of negatives (here n=3n = 3)
sim(q,p)\mathrm{sim}(q,p)
similarity score; in DPR the dot product q⋅pq \cdot p
exe^{x}
the exponential function (e≈2.718e \approx 2.718)
log⁡\log
the natural logarithm, ln⁡\ln, by convention
Read it aloud L equals minus the log of a fraction: on top, e to the power of the similarity between q and the positive; underneath, that same term plus the sum, over the n negatives, of e to the power of the similarity between q and each negative.
Formula in plain words Take ee raised to the similarity of the query with the positive document: the “score mass” of the correct answer. Divide it by the sum of that same value plus ee raised to the similarity with every negative document. The fraction is the probability that the model picks the correct document out of all four candidates (a softmax, as in the maths refresher of the Week 4 notes). Then take the negative natural logarithm of that probability. Raising sim(q,p+)\mathrm{sim}(q,p^+) pushes the probability towards 1 and the loss towards 0; raising any negative's similarity, or adding more negatives, enlarges the denominator and raises the loss. The loss is always at least 0, and it grows without limit as the probability falls towards 0.
Similarities (dot products) p⁺: 0.9 · p₁⁻: 0.3 · p₂⁻: 0.1 · p₃⁻: 0.4 Exponentiate every score all terms positive, gaps amplified Positive term ÷ sum of all terms the sum includes the positive's own term P(p⁺ | q): softmax probability between 0 and 1 L = −ln P(p⁺ | q) 0 when P = 1; large when P is small
InfoNCE is a softmax over the candidates followed by a negative log: the loss is small only when the positive takes most of the probability.
PracticeWhy does the denominator include the positive's own term?
AnswerSo that the fraction is a probability over all candidates, the positive included: the four probabilities sum to 1 and each lies between 0 and 1. That keeps −ln⁡P≥0-\ln P \ge 0, with 0 reached only when the positive takes all the mass. Leaving the positive out would give a ratio that can exceed 1 and a loss that can turn negative.

2. Worked solution

Sources: slides 19–21 · Video 2 (DPR), 03:34–05:04 · Week 5 in-class exercise 3

Worked example 1InfoNCE for one query, one positive, three negatives

1

Exponentiate every similarity. This makes every term positive and amplifies the gaps between scores:

e0.9≈2.4596 (positive) e^{0.9} \approx 2.4596 \ \text{(positive)}

Negatives:

e0.3≈1.3499e0.1≈1.1052e0.4≈1.4918 \begin{aligned} e^{0.3} &\approx 1.3499 \\ e^{0.1} &\approx 1.1052 \\ e^{0.4} &\approx 1.4918 \end{aligned}
2

Build the denominator (call it DD). It is the positive's term plus the sum of all three negatives' terms: the total score mass of every candidate.

D=e0.9+(e0.3+e0.1+e0.4)=2.4596+(1.3499+1.1052+1.4918)=2.4596+3.9469=6.4065 \begin{aligned} D &= e^{0.9} + (e^{0.3} + e^{0.1} + e^{0.4}) \\ &= 2.4596 + (1.3499 \\ &\qquad + 1.1052 + 1.4918) \\ &= 2.4596 + 3.9469 \\ &= 6.4065 \end{aligned}
3

Form the softmax probability P=P(p+∣q)P = P(p^+ \mid q) of picking the positive: positive mass over total mass.

P=e0.9e0.9+e0.3+e0.1+e0.4=2.45966.4065≈0.3839 \begin{aligned} P &= \frac{e^{0.9}}{e^{0.9} + e^{0.3} + e^{0.1} + e^{0.4}} \\ &= \frac{2.4596}{6.4065} \approx 0.3839 \end{aligned}

Interpretation: the model gives the correct document only a ≈38% chance. The positive is the single highest-scored candidate, but the three negatives together hold most of the probability mass.

4

Take the negative natural log.

L=−ln⁡(0.3839)≈0.957 L = -\ln(0.3839) \approx 0.957
ResultL ≈ 0.96. A confident model (P → 1) would have loss −ln(1) = 0, so this fairly large loss means training still has work to do: the gradient pushes sim(q, p⁺) up and the negative similarities down.
Share of the total score mass (6.4065) p⁺ 38.4% p₁⁻ 21.1% p₂⁻ 17.3% p₃⁻ 23.3% negatives together: 61.6% P(p⁺ | q) = 0.384 exp terms: 2.4596 | 1.3499 | 1.1052 | 1.4918 loss L = −ln 0.384 ≈ 0.96
The positive is the largest single slice, yet the negatives together hold most of the mass, so the probability of the positive is only 0.38.
PracticeRecompute the loss for sim(q, p⁺) = 1.2 and negatives 0.2, 0.5, 0.1.
Follow the same four steps.
AnswerThe four steps give: e1.2=3.3201e0.2=1.2214e0.5=1.6487e0.1=1.1052denominator=3.3201+3.9753=7.2954P=3.32017.2954≈0.4551L=−ln⁡(0.4551)≈0.787 \begin{aligned} e^{1.2} &= 3.3201 \\ e^{0.2} &= 1.2214 \\ e^{0.5} &= 1.6487 \\ e^{0.1} &= 1.1052 \\ \text{denominator} &= 3.3201 + 3.9753 = 7.2954 \\ P &= \frac{3.3201}{7.2954} \approx 0.4551 \\ L &= -\ln(0.4551) \approx 0.787 \end{aligned}

3. How the loss responds, and exam tips

Sources: slide 21 · Video 2 (DPR), 04:12–05:50 · Week 5 in-class exercise 3

The loss depends on the positive's probability alone: L=−ln⁡PL = -\ln P. The curve below shows why a confident model is rewarded with a near-zero loss while an uncertain one is penalised heavily; Video 2 makes the same point (a probability near 1 gives a loss near 0, while a probability around 0.5 gives a much higher loss).

0123 00.51 P(p⁺ | q) loss L Exercise 3: P = 0.38, L = 0.96 sim(q,p⁺) = 2.0: P = 0.65, L = 0.43 steep: small P is punished hard
The loss −ln P falls towards 0 as the positive's probability approaches 1; improving the positive's similarity moves the point down and to the right.

Worked example 2Sanity check the direction

If training lifted sim(q,p+)\mathrm{sim}(q,p^+) to 2.0 with the same negatives:

P=e2.0e2.0+3.9469=7.389111.3360≈0.652L=−ln⁡(0.652)≈0.428 \begin{aligned} P &= \frac{e^{2.0}}{e^{2.0} + 3.9469} \\ &= \frac{7.3891}{11.3360} \approx 0.652 \\ L &= -\ln(0.652) \approx 0.428 \end{aligned}
ResultHigher confidence in the positive → lower loss (0.957 → 0.428), as expected.
Exam angleInfoNCE uses the natural logarithm (ln) by convention. With log base 10 the answer would be about 0.416; write L≈0.96L \approx 0.96 unless the question explicitly asks for base 10.
Common trapThe denominator includes the positive's own term. Forgetting it is the most common mistake: 2.45963.9469\frac{2.4596}{3.9469} gives 0.623 and the wrong loss 0.473.
PracticeA classmate reports L ≈ 0.416 for this exercise. What did they most likely do?
AnswerThey used the base-10 logarithm: −log⁡10(0.3839)≈0.416-\log_{10}(0.3839) \approx 0.416. With the natural log, the convention for InfoNCE, the answer is −ln⁡(0.3839)≈0.957-\ln(0.3839) \approx 0.957.
PracticeWhat is the loss when the model gives the positive exactly a 50% probability?
AnswerL=−ln⁡(0.5)=ln⁡2≈0.693L = -\ln(0.5) = \ln 2 \approx 0.693.

4. Summary and study checklist