Week 5: Exercise 3, InfoNCE loss (worked solution)Explainer5
INFS7410 · Information Retrieval and Web Search · Week 5
Exercise 3: InfoNCE loss, worked solution
Step-by-step calculation of the InfoNCE (negative log likelihood) loss for one query with one positive and three negative documents: the training loss of DPR, ANCE, Contriever, ColBERT and RepLLaMA. The page states the formula, works the numbers, and then checks how the loss moves when the model improves.
Given the following similarity scores for a query q, calculate the InfoNCE loss. The similarity is the dot product of the query embedding and the document embedding (slide 21), as in DPR's bi-encoder.
Candidate
Role
Similarity to q
p⁺
positive (relevant) document
0.9
p₁⁻
negative document
0.3
p₂⁻
negative document
0.1
p₃⁻
negative document
0.4
L=−logesim(q,p+)+∑j=1nesim(q,pj−)esim(q,p+)
Where
L
the InfoNCE loss for this one query (also called the negative log likelihood loss)
q
the query embedding
p+
the embedding of the positive (relevant) document
pj−
the embedding of the j-th negative (non-relevant) document
n
number of negatives (here n=3)
sim(q,p)
similarity score; in DPR the dot product q⋅p
ex
the exponential function (e≈2.718)
log
the natural logarithm, ln, by convention
Read it aloud
L equals minus the log of a fraction: on top, e to the power of the similarity between q and the positive; underneath, that same term plus the sum, over the n negatives, of e to the power of the similarity between q and each negative.
Formula in plain words
Take e raised to the similarity of the query with the positive document: the “score mass” of the correct answer. Divide it by the sum of that same value pluse raised to the similarity with every negative document. The fraction is the probability that the model picks the correct document out of all four candidates (a softmax, as in the maths refresher of the Week 4 notes). Then take the negative natural logarithm of that probability. Raising sim(q,p+) pushes the probability towards 1 and the loss towards 0; raising any negative's similarity, or adding more negatives, enlarges the denominator and raises the loss. The loss is always at least 0, and it grows without limit as the probability falls towards 0.
InfoNCE is a softmax over the candidates followed by a negative log: the loss is small only when the positive takes most of the probability.PracticeWhy does the denominator include the positive's own term?
AnswerSo that the fraction is a probability over all candidates, the positive included: the four probabilities sum to 1 and each lies between 0 and 1. That keeps −lnP≥0, with 0 reached only when the positive takes all the mass. Leaving the positive out would give a ratio that can exceed 1 and a loss that can turn negative.
Form the softmax probabilityP=P(p+∣q) of picking the positive: positive mass over total mass.
P=e0.9+e0.3+e0.1+e0.4e0.9=6.40652.4596≈0.3839
Interpretation: the model gives the correct document only a ≈38% chance. The positive is the single highest-scored candidate, but the three negatives together hold most of the probability mass.
4
Take the negative natural log.
L=−ln(0.3839)≈0.957
ResultL ≈ 0.96. A confident model (P → 1) would have loss −ln(1) = 0, so this fairly large loss means training still has work to do: the gradient pushes sim(q, p⁺) up and the negative similarities down.
The positive is the largest single slice, yet the negatives together hold most of the mass, so the probability of the positive is only 0.38.PracticeRecompute the loss for sim(q, p⁺) = 1.2 and negatives 0.2, 0.5, 0.1.
Follow the same four steps.
AnswerThe four steps give:
e1.2e0.2e0.5e0.1denominatorPL=3.3201=1.2214=1.6487=1.1052=3.3201+3.9753=7.2954=7.29543.3201≈0.4551=−ln(0.4551)≈0.787
The loss depends on the positive's probability alone: L=−lnP. The curve below shows why a confident model is rewarded with a near-zero loss while an uncertain one is penalised heavily; Video 2 makes the same point (a probability near 1 gives a loss near 0, while a probability around 0.5 gives a much higher loss).
The loss −ln P falls towards 0 as the positive's probability approaches 1; improving the positive's similarity moves the point down and to the right.
Worked example 2Sanity check the direction
If training lifted sim(q,p+) to 2.0 with the same negatives:
ResultHigher confidence in the positive → lower loss (0.957 → 0.428), as expected.
Exam angleInfoNCE uses the natural logarithm (ln) by convention. With log base 10 the answer would be about 0.416; write L≈0.96 unless the question explicitly asks for base 10.
Common trapThe denominator includes the positive's own term. Forgetting it is the most common mistake: 3.94692.4596 gives 0.623 and the wrong loss 0.473.
PracticeA classmate reports L ≈ 0.416 for this exercise. What did they most likely do?
AnswerThey used the base-10 logarithm: −log10(0.3839)≈0.416. With the natural log, the convention for InfoNCE, the answer is −ln(0.3839)≈0.957.
PracticeWhat is the loss when the model gives the positive exactly a 50% probability?
AnswerL=−ln(0.5)=ln2≈0.693.
4. Summary and study checklist
InfoNCE = softmax probability of the positive among all candidates, then the negative natural log.
Four steps: exponentiate, sum (positive included), divide, take −ln.
Exercise answer: P≈0.3839, L≈0.96.
The loss falls as the positive's similarity rises and grows as negatives get closer to the query.
Used by DPR, ANCE, Contriever, ColBERT and RepLLaMA (slides 19, 28, 31, 39, 48). Compare it with the margin loss of Exercise 4.