Diversity, and who consumes the results

The retrieval half: what each policy delivers

In short: Coverage costs relevance. The harvest buys it cheaper than MMR and admits fewer distractors than a DPP.

Ground truth comes from construction rather than from a model. Each of 120 queries is associated with several distinct facts, and each candidate document realises at most one of them, so the sub-topics a selection covers can be counted exactly; deriving sub-topics by clustering would have been circular, since one of the policies under test selects by cluster. Candidates are gated to the upper half by relevance first, as a first pass would do, a distractor pool of answerless documents is mixed in, and each policy then selects ten. S-recallCoverage metrics (S-recall, alpha-nDCG). Scores that reward a result list for covering different sub-topics rather than repeating the same one. S-recall is the share of known sub-topics that appear in the top ten; alpha-nDCG discounts repeats. counts the share of known sub-topics covered; nDCG@10nDCG@10. A 0 to 1 score of how good the first ten search results are. It rewards putting the right documents near the top and counts every relevant document the judges marked. A difference of 0.03 is small but real on a few hundred queries; 0.2 is large. measures relevance; the distractor rate counts answerless documents admitted.

The two harvest settings sit above and to the right of the MMR and cluster policies at comparable coverage. The DPP reaches the most coverage and pays the most relevance for it.

Table view

Point	Relevance (nDCG@10)	Sub-topic coverage (S-recall)
Relevance only	0.827	0.372
MMR, lambda 0.7	0.802	0.405
Harvest 20%	0.808	0.567
Cluster round-robin	0.774	0.545
Harvest 40%	0.781	0.686
MMR, lambda 0.3	0.729	0.686
DPP	0.699	0.746

Source: Benchmark repository, results/e3_diversity/metrics.json, commit 92c18cb (2026-08-22). Reproduced on a second machine to within 5e-8.

Read along the rows: the 40 percent harvest and MMR at lambda 0.3 reach the same coverage (0.686), and the harvest pays 0.046 of relevance for it where MMR pays 0.098. The harvest also admits distractors at 0.236 against MMR's 0.285 and the DPP's 0.328. The mechanism is visible in the construction: MMR and the DPPDiversity selection (MMR, DPP, harvest). Rules for choosing which ten candidates to pass on so that they cover different sub-topics rather than repeating the top hit. MMR and DPP re-rank the whole list; the harvest keeps the most relevant head and spends a fixed quota on the least similar tail. diversify the whole list, so the top of the list drifts off relevance, while the harvest keeps a relevance head and spends only the tail. All coverage gains over relevance ranking are Holm-significant at p=0.0001; so are all the relevance costs.

An earlier version of this experiment reported coverage as nearly free. It was wrong: an integer-division bug gave two distractors per task instead of forty, so nothing could be lost and every relevance metric sat at its ceiling. The corrected trade is the one above.

The consumer half: does the right amount depend on the reader?

In short: Three judges, three answers that shrink to nothing as the judge moves away from the generator's model family.

The same gated pool, the same query, and the same synthesis model were used throughout; only the selection policy and the consumer varied. For the machine consumer the selection is assembled as evidence for a model that must synthesise an account of everything the material covers. For the human consumer the same selection is rendered as a short brief for a person who will read it once and act on it. Sixty tasks, two policies (relevance, harvest) and two consumers give 240 generations. The prediction was an interaction, not a main effect: harvesting should raise coverage for the machine consumer by more than for the human one.

Coverage of the known sub-topics in the generated text was scored by three judges: a model from the same family as the generator, a larger model from that family, and a model from a different family.

+0.071 (p=0.009), +0.047 (p=0.211), +0.008 (p=0.763). The estimate shrinks monotonically with distance from the generator's own family and vanishes at the cross-family judge, which is the signature of a judging artefact. Not disproved: all three point estimates are positive, and 60 tasks cannot resolve an effect this small. But no evidence may be cited for it.

Table view

Category	Interaction	95% interval
Same-family judge, small	+0.071	+0.022 to +0.125
Same-family judge, large	+0.047	−0.024 to +0.117
Cross-family judge	+0.008	−0.039 to +0.056

Source: Benchmark repository, results/e3b_consumer, results/e3b_consumer_judge_pro, results/e3b_consumer_judge_azure (metrics.json), commit 92c18cb (2026-08-22).

One finding from the same study needs no judge and survived every check. Counting words in the output that no selected document supports, harvesting reduces unsupported content: −0.067 for the human-facing brief (p=0.0018) and −0.048 for the machine-facing synthesis (p=0.067). A selection that covers more of the material gives the writer less reason to invent. This was not predicted, it is judge-independent, and it is the most defensible thing the consumer study produced.

What this establishes

Mixed evidence The retrieval half is independently measured on constructed ground truth; the consumer half depends on model judges and is reported as unresolved.

The research behind this page

• Benchmark repository: results/e3_diversity, results/e3b_consumer and the two additional judge runs, commit 92c18cb, 2026-08-22.
• Carbonell and Goldstein, MMR, SIGIR 1998; diverse multi-answer retrieval with determinantal point processes, COLING 2022; DIVA, NAACL 2025; the Distracting Effect, ACL 2025; The Power of Noise (SIGIR 2024) and its retraction (SIGIR 2026). All on the reading list.
• Alpha-nDCG, Clarke et al., SIGIR 2008, and sub-question coverage (Salesforce and Georgia Tech, Oct 2024) for why coverage has to be measured with coverage metrics rather than recall.