The papers behind the recommendation

The published work read during the programme, grouped by the question it answers, with what each paper shows and how the experiments here relate to it: confirmed, contradicted, or bounded.

Several vectors per document, and the unit of indexing

Paper	What it shows	How the results here relate
Weller, Boratko, Naim, Lee, On the Theoretical Limitations of Embedding-Based Retrieval, arXiv:2508.21038, Aug 2025. https://arxiv.org/abs/2508.21038	For a fixed embedding dimension, some top-k result sets are unreachable by any single-vector index; state-of-the-art embedders fail the LIMIT dataset built to show it	Confirmed: pooled dense 0.314 on LIMIT-small; per-attribute cards 0.988, fine blind windows 0.924
Khattab and Zaharia, ColBERT, SIGIR 2020. https://arxiv.org/abs/2004.12832 ; Santhanam et al., ColBERTv2, NAACL 2022. https://arxiv.org/abs/2112.01488	Token-level multi-vector retrieval (late interaction); storage 10 to 30 times dense retrieval is the standing critique	Bounds the design: coarse purpose views are the compromise between one vector and per-token vectors; storage is a property of the corpus (44 cards per document on LIMIT, 2.8 on SciFact)
Zhang et al., Multi-View Document Representation Learning, ACL 2022. https://aclanthology.org/2022.acl-long.414/	Several learned viewer embeddings per document for different query views	The closest academic phrase to the idea; the views there are learned and fixed, not conditioned on an objective
Kong et al., Multi-Aspect Dense Retrieval, KDD 2022	Explicit aspect embeddings from product metadata	Same family: fixed aspects, set by available metadata
Dong et al., Multi-view Content-aware Indexing, EMNLP Findings 2024. https://arxiv.org/abs/2404.15103	Each section indexed under three fixed views (raw text, keywords, summary)	A fixed-view multi-representation index; consistent with corpus-derived views being the durable positive
Chen et al., Dense X Retrieval, EMNLP 2024. https://arxiv.org/abs/2312.06648	Proposition-level indexing: +10.1 recall at 20 unsupervised	Granularity matters and is workload-dependent, which is what the SciFact boundary shows
Duarte et al., LumberChunker, EMNLP Findings 2024. https://arxiv.org/abs/2406.17526 ; Zhong et al., Mix-of-Granularity, 2024. https://arxiv.org/abs/2406.00456	Model-placed chunk boundaries; routing among granularities	Both decide a granularity globally and require reading every item; the recommendation keeps the model off the per-item path
Anthropic, Contextual Retrieval, Sep 2024 (vendor). https://www.anthropic.com/news/contextual-retrieval	Prepending situating context to each chunk before embedding; about $1.02 per million document tokens with caching	The published floor for touching every chunk once with a model; the two-pass design avoids that cost by construction
Weller et al., When do Generative Query and Document Expansions Fail?, EACL 2024. https://arxiv.org/abs/2309.08541	Expansions help weak retrievers and can hurt strong ones	Consistent with purpose cards hurting on SciFact under a capable hybrid

Paper: Weller, Boratko, Naim, Lee, On the Theoretical Limitations of Embedding-Based Retrieval, arXiv:2508.21038, Aug 2025. https://arxiv.org/abs/2508.21038
What it shows: For a fixed embedding dimension, some top-k result sets are unreachable by any single-vector index; state-of-the-art embedders fail the LIMIT dataset built to show it
How the results here relate: Confirmed: pooled dense 0.314 on LIMIT-small; per-attribute cards 0.988, fine blind windows 0.924

Paper: Khattab and Zaharia, ColBERT, SIGIR 2020. https://arxiv.org/abs/2004.12832 ; Santhanam et al., ColBERTv2, NAACL 2022. https://arxiv.org/abs/2112.01488
What it shows: Token-level multi-vector retrieval (late interaction); storage 10 to 30 times dense retrieval is the standing critique
How the results here relate: Bounds the design: coarse purpose views are the compromise between one vector and per-token vectors; storage is a property of the corpus (44 cards per document on LIMIT, 2.8 on SciFact)

Paper: Zhang et al., Multi-View Document Representation Learning, ACL 2022. https://aclanthology.org/2022.acl-long.414/
What it shows: Several learned viewer embeddings per document for different query views
How the results here relate: The closest academic phrase to the idea; the views there are learned and fixed, not conditioned on an objective

Paper: Kong et al., Multi-Aspect Dense Retrieval, KDD 2022
What it shows: Explicit aspect embeddings from product metadata
How the results here relate: Same family: fixed aspects, set by available metadata

Paper: Dong et al., Multi-view Content-aware Indexing, EMNLP Findings 2024. https://arxiv.org/abs/2404.15103
What it shows: Each section indexed under three fixed views (raw text, keywords, summary)
How the results here relate: A fixed-view multi-representation index; consistent with corpus-derived views being the durable positive

Paper: Chen et al., Dense X Retrieval, EMNLP 2024. https://arxiv.org/abs/2312.06648
What it shows: Proposition-level indexing: +10.1 recall at 20 unsupervised
How the results here relate: Granularity matters and is workload-dependent, which is what the SciFact boundary shows

Paper: Duarte et al., LumberChunker, EMNLP Findings 2024. https://arxiv.org/abs/2406.17526 ; Zhong et al., Mix-of-Granularity, 2024. https://arxiv.org/abs/2406.00456
What it shows: Model-placed chunk boundaries; routing among granularities
How the results here relate: Both decide a granularity globally and require reading every item; the recommendation keeps the model off the per-item path

Paper: Anthropic, Contextual Retrieval, Sep 2024 (vendor). https://www.anthropic.com/news/contextual-retrieval
What it shows: Prepending situating context to each chunk before embedding; about $1.02 per million document tokens with caching
How the results here relate: The published floor for touching every chunk once with a model; the two-pass design avoids that cost by construction

Paper: Weller et al., When do Generative Query and Document Expansions Fail?, EACL 2024. https://arxiv.org/abs/2309.08541
What it shows: Expansions help weak retrievers and can hurt strong ones
How the results here relate: Consistent with purpose cards hurting on SciFact under a capable hybrid

Conditioning the index on an instruction or objective

Paper	What it shows	How the results here relate
Su et al., INSTRUCTOR, ACL Findings 2023. https://arxiv.org/abs/2212.09741	One encoder that produces task-specific embeddings from an instruction	The execution layer that subsumes objective-conditioned views
Weller et al., FollowIR, EMNLP 2024. https://arxiv.org/abs/2403.15246	A benchmark with two sets of human judgements per query, before and after the instruction changes, and the p-MRR score	Used as the real-data test in round 5; card re-selection scored p-MRR about zero
Weller et al., Promptriever, ICLR 2025. https://arxiv.org/abs/2409.11136	Retrievers trained to follow instructions like language models	The strong baseline any objective-conditioning proposal must beat; deliberately deferred here because it would widen the negative
Feng et al., GSTransform, 2025. https://arxiv.org/abs/2505.24754	Per-instruction embedding views without re-encoding the corpus	Delivers the embed-once economics the round-5 architecture aimed at, without the view-design step
Qidwai et al., CAMI, 2026. https://arxiv.org/abs/2606.28365	An agent proposes candidate enrichment indexes and selects a portfolio under a cost budget	Conditions index construction on the corpus and a budget, not on the objective; consistent with the negative here
Microsoft Research, GraphRAG auto-tuning, 2024 (vendor). https://www.microsoft.com/en-us/research/blog/graphrag-auto-tuning-provides-rapid-adaptation-to-new-domains/	Extraction schema generated from roughly a one percent corpus sample	Corpus-derived design, the form that worked here

Paper: Su et al., INSTRUCTOR, ACL Findings 2023. https://arxiv.org/abs/2212.09741
What it shows: One encoder that produces task-specific embeddings from an instruction
How the results here relate: The execution layer that subsumes objective-conditioned views

Paper: Weller et al., FollowIR, EMNLP 2024. https://arxiv.org/abs/2403.15246
What it shows: A benchmark with two sets of human judgements per query, before and after the instruction changes, and the p-MRR score
How the results here relate: Used as the real-data test in round 5; card re-selection scored p-MRR about zero

Paper: Weller et al., Promptriever, ICLR 2025. https://arxiv.org/abs/2409.11136
What it shows: Retrievers trained to follow instructions like language models
How the results here relate: The strong baseline any objective-conditioning proposal must beat; deliberately deferred here because it would widen the negative

Paper: Feng et al., GSTransform, 2025. https://arxiv.org/abs/2505.24754
What it shows: Per-instruction embedding views without re-encoding the corpus
How the results here relate: Delivers the embed-once economics the round-5 architecture aimed at, without the view-design step

Paper: Qidwai et al., CAMI, 2026. https://arxiv.org/abs/2606.28365
What it shows: An agent proposes candidate enrichment indexes and selects a portfolio under a cost budget
How the results here relate: Conditions index construction on the corpus and a budget, not on the objective; consistent with the negative here

Paper: Microsoft Research, GraphRAG auto-tuning, 2024 (vendor). https://www.microsoft.com/en-us/research/blog/graphrag-auto-tuning-provides-rapid-adaptation-to-new-domains/
What it shows: Extraction schema generated from roughly a one percent corpus sample
How the results here relate: Corpus-derived design, the form that worked here

The model designs; classical machinery executes

Paper	What it shows	How the results here relate
Wan et al., TnT-LLM, KDD 2024. https://arxiv.org/abs/2403.12173	A model builds a taxonomy from corpus samples and pseudo-labels; embedding plus logistic regression reaches 0.658 accuracy against 0.655 for the frontier model used directly	The economics plank of the recommendation, and the closest published design to the corpus-sample step that worked
Pham et al., TopicGPT, NAACL 2024. https://aclanthology.org/2024.naacl-long.164/	Model-generated topics align better with human categories than classical topic models	Supports model naming and merging of discovered topics
Hsieh et al., Distilling Step-by-Step, ACL Findings 2023. https://arxiv.org/abs/2305.02301 ; SetFit, 2022. https://github.com/huggingface/setfit	Small models taught by large ones can beat the teacher at execution; few-shot small models beat GPT-3 at 1,600 times smaller	Why the gate and the classifiers should be small models, not frontier calls
Databricks, Scaling document classification to 100k+ labels, Jul 2026 (vendor). https://www.databricks.com/blog/scaling-document-classification-100k-labels	Vector shortlist plus small model at about one hundredth the per-document cost of direct frontier classification, at higher accuracy (0.81 against 0.76)	A production-scale parallel to the two-pass constant factor
Arora et al., EVAPORATE, VLDB 2023. https://arxiv.org/abs/2304.09433	A model synthesises extraction functions from a sample that then run over the whole corpus, cutting token cost by two orders of magnitude	The same shape of saving, measured independently
Sarthi et al., RAPTOR, ICLR 2024. https://arxiv.org/abs/2401.18059	Cluster embeddings, summarise clusters recursively, index all levels	The closest cluster-then-summarise system; uses a Gaussian mixture where this programme used density clustering
Grootendorst, BERTopic, 2022. https://maartengr.github.io/BERTopic/	Density clustering over reduced embeddings with outlier reduction, topic merging, and guided or zero-shot topic lists	The established machinery used in rounds 2 to 5, as the maintainer required, rather than a home-grown variant
Edge et al., GraphRAG, 2024 (Microsoft). https://arxiv.org/abs/2404.16130 ; TREX, Mar 2025. https://arxiv.org/abs/2503.02922 ; LightRAG, EMNLP 2025. https://arxiv.org/abs/2410.05779	Model-intensive graph indexing for global sensemaking; indexing bills of $51 to $389 per benchmark corpus; lightweight alternatives	The cost of letting a model read everything, and the space a topics-plus-clusters vector design competes in; see the guide's knowledge-graph study

Paper: Wan et al., TnT-LLM, KDD 2024. https://arxiv.org/abs/2403.12173
What it shows: A model builds a taxonomy from corpus samples and pseudo-labels; embedding plus logistic regression reaches 0.658 accuracy against 0.655 for the frontier model used directly
How the results here relate: The economics plank of the recommendation, and the closest published design to the corpus-sample step that worked

Paper: Pham et al., TopicGPT, NAACL 2024. https://aclanthology.org/2024.naacl-long.164/
What it shows: Model-generated topics align better with human categories than classical topic models
How the results here relate: Supports model naming and merging of discovered topics

Paper: Hsieh et al., Distilling Step-by-Step, ACL Findings 2023. https://arxiv.org/abs/2305.02301 ; SetFit, 2022. https://github.com/huggingface/setfit
What it shows: Small models taught by large ones can beat the teacher at execution; few-shot small models beat GPT-3 at 1,600 times smaller
How the results here relate: Why the gate and the classifiers should be small models, not frontier calls

Paper: Databricks, Scaling document classification to 100k+ labels, Jul 2026 (vendor). https://www.databricks.com/blog/scaling-document-classification-100k-labels
What it shows: Vector shortlist plus small model at about one hundredth the per-document cost of direct frontier classification, at higher accuracy (0.81 against 0.76)
How the results here relate: A production-scale parallel to the two-pass constant factor

Paper: Arora et al., EVAPORATE, VLDB 2023. https://arxiv.org/abs/2304.09433
What it shows: A model synthesises extraction functions from a sample that then run over the whole corpus, cutting token cost by two orders of magnitude
How the results here relate: The same shape of saving, measured independently

Paper: Sarthi et al., RAPTOR, ICLR 2024. https://arxiv.org/abs/2401.18059
What it shows: Cluster embeddings, summarise clusters recursively, index all levels
How the results here relate: The closest cluster-then-summarise system; uses a Gaussian mixture where this programme used density clustering

Paper: Grootendorst, BERTopic, 2022. https://maartengr.github.io/BERTopic/
What it shows: Density clustering over reduced embeddings with outlier reduction, topic merging, and guided or zero-shot topic lists
How the results here relate: The established machinery used in rounds 2 to 5, as the maintainer required, rather than a home-grown variant

Paper: Edge et al., GraphRAG, 2024 (Microsoft). https://arxiv.org/abs/2404.16130 ; TREX, Mar 2025. https://arxiv.org/abs/2503.02922 ; LightRAG, EMNLP 2025. https://arxiv.org/abs/2410.05779
What it shows: Model-intensive graph indexing for global sensemaking; indexing bills of $51 to $389 per benchmark corpus; lightweight alternatives
How the results here relate: The cost of letting a model read everything, and the space a topics-plus-clusters vector design competes in; see the guide's knowledge-graph study

Diversity, noise, and distractors

Paper	What it shows	How the results here relate
Carbonell and Goldstein, MMR, SIGIR 1998. https://dl.acm.org/doi/10.1145/290941.291025	The standard relevance-diversity re-ranking; ships in most vector stores	The harvest bought the same coverage as MMR at lambda 0.3 for half the relevance cost
Diverse Multi-Answer Retrieval with DPPs, COLING 2022. https://aclanthology.org/2022.coling-1.194/ ; SMART-RAG, 2024. https://arxiv.org/abs/2409.13992	Determinantal point processes for diverse context selection	The DPP reached the most coverage and admitted the most distractors (0.328)
DIVA, NAACL 2025. https://aclanthology.org/2025.naacl-long.56/ ; ARAGOG, 2024. https://arxiv.org/abs/2404.01037	Diversified retrieval wins on ambiguous, multi-answer questions; MMR adds nothing on standard single-answer QA	Scopes where diversification belongs: coverage-driven workloads only
Sub-Question Coverage, Salesforce and Georgia Tech, Oct 2024. https://arxiv.org/html/2410.15531v1 ; Beyond Relevance: Information Coverage, ICTIR 2026. https://arxiv.org/abs/2603.08819	Coverage predicts answer completeness; coverage metrics correlate with nugget coverage	Why S-recall and alpha-nDCG, not recall, are the measurement
Cuconasu et al., The Power of Noise, SIGIR 2024. https://arxiv.org/pdf/2401.14887 ; The Powerless Noise, SIGIR 2026. https://arxiv.org/pdf/2607.03615	Random documents appeared to help generation; the effect was walked back	A caution against deliberate lateral inclusion without measurement
Amiraz et al., The Distracting Effect, ACL 2025. https://arxiv.org/abs/2505.06914	Topically adjacent answerless passages are the most damaging additions	Why the pool must be gated first and the distractor rate measured; the harvest's 0.236 against the DPP's 0.328
Clarke et al., alpha-nDCG, SIGIR 2008. https://dl.acm.org/doi/10.1145/1390334.1390446	The coverage-aware ranking metric	Used throughout the diversity experiment

Paper: Carbonell and Goldstein, MMR, SIGIR 1998. https://dl.acm.org/doi/10.1145/290941.291025
What it shows: The standard relevance-diversity re-ranking; ships in most vector stores
How the results here relate: The harvest bought the same coverage as MMR at lambda 0.3 for half the relevance cost

Paper: Diverse Multi-Answer Retrieval with DPPs, COLING 2022. https://aclanthology.org/2022.coling-1.194/ ; SMART-RAG, 2024. https://arxiv.org/abs/2409.13992
What it shows: Determinantal point processes for diverse context selection
How the results here relate: The DPP reached the most coverage and admitted the most distractors (0.328)

Paper: DIVA, NAACL 2025. https://aclanthology.org/2025.naacl-long.56/ ; ARAGOG, 2024. https://arxiv.org/abs/2404.01037
What it shows: Diversified retrieval wins on ambiguous, multi-answer questions; MMR adds nothing on standard single-answer QA
How the results here relate: Scopes where diversification belongs: coverage-driven workloads only

Paper: Sub-Question Coverage, Salesforce and Georgia Tech, Oct 2024. https://arxiv.org/html/2410.15531v1 ; Beyond Relevance: Information Coverage, ICTIR 2026. https://arxiv.org/abs/2603.08819
What it shows: Coverage predicts answer completeness; coverage metrics correlate with nugget coverage
How the results here relate: Why S-recall and alpha-nDCG, not recall, are the measurement

Paper: Cuconasu et al., The Power of Noise, SIGIR 2024. https://arxiv.org/pdf/2401.14887 ; The Powerless Noise, SIGIR 2026. https://arxiv.org/pdf/2607.03615
What it shows: Random documents appeared to help generation; the effect was walked back
How the results here relate: A caution against deliberate lateral inclusion without measurement

Paper: Amiraz et al., The Distracting Effect, ACL 2025. https://arxiv.org/abs/2505.06914
What it shows: Topically adjacent answerless passages are the most damaging additions
How the results here relate: Why the pool must be gated first and the distractor rate measured; the harvest's 0.236 against the DPP's 0.328

Paper: Clarke et al., alpha-nDCG, SIGIR 2008. https://dl.acm.org/doi/10.1145/1390334.1390446
What it shows: The coverage-aware ranking metric
How the results here relate: Used throughout the diversity experiment

Benchmarks and evaluation

Paper	What it shows	How the results here relate
Thakur et al., BEIR, NeurIPS 2021. https://arxiv.org/abs/2104.08663	The zero-shot retrieval benchmark family; SciFact's home	The human-judged rung of the corpus ladder
Weller et al., LIMIT dataset, 2025 (above)	The single-vector stress test	The capacity rung, and the site of the programme's largest correction
Es et al., RAGAS, EACL 2024. https://arxiv.org/abs/2309.15217 ; RAGChecker, 2024. https://arxiv.org/abs/2408.08067	Reference-free and fine-grained evaluation of generated answers	The unsupported-content measure in the consumer study follows the same principle with a direct word-level check rather than a judge

Paper: Thakur et al., BEIR, NeurIPS 2021. https://arxiv.org/abs/2104.08663
What it shows: The zero-shot retrieval benchmark family; SciFact's home
How the results here relate: The human-judged rung of the corpus ladder

Paper: Weller et al., LIMIT dataset, 2025 (above)
What it shows: The single-vector stress test
How the results here relate: The capacity rung, and the site of the programme's largest correction

Paper: Es et al., RAGAS, EACL 2024. https://arxiv.org/abs/2309.15217 ; RAGChecker, 2024. https://arxiv.org/abs/2408.08067
What it shows: Reference-free and fine-grained evaluation of generated answers
How the results here relate: The unsupported-content measure in the consumer study follows the same principle with a direct word-level check rather than a judge

How this list was assembled

The literature sweep ran in two passes: a retrieval-literature sprint on 2026-08-19 covering multi-view indexing, diversity, taxonomy economics and graph-versus-vector evidence, and a prior-art review on 2026-08-20 that added the instruction-conditioning and generative-index-design lines. Every arXiv entry in the benchmark repository's bibliography was checked programmatically against its abstract page on 2026-08-20 (title, first author, submission date). Vendor-published results are labelled; conference venues are as published and are re-checked before any submission.