我们是否应该使用检索方法预训练自回归语言模型？一项全面研究（英文版）

英文标题：Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study中文摘要：通过引入检索机制可以提高大型解码器语言模型的困惑度和生成质量，本文提出了一种可扩展的预训练检索增强语言模型，它在实验证明了在知识密集型任务上优于标准 GPT，未来这可能是预训练自回归 LM 的有前途的方向。英文摘要：Large decoder-only language models (LMs) can be largely improved in terms ofperplexity by retrieval (e.g., RETRO), but its impact on text generationquality and downstream task accuracy is unclear. Thus, it is still an openquestion: shall we pretrain large autoregressive LMs with retrieval? To a