针对数据效率的语言模型 MiniPile 挑战（英文版）

英文标题：The MiniPile Challenge for Data-Efficient Language Models中文摘要：本文提出 MiniPile 挑战，呈现一种使用文本语料库的小数据集进行语言模型预训练的方法，其适用性通过在 GLUE 和 SNI 基准测试中得到论证。英文摘要：The ever-growing diversity of pre-training text corpora has equipped languagemodels with generalization capabilities across various downstream tasks.However, such diverse datasets are often too large for academic budgets; hence,most research on Transformer architectures, training procedures, optimizers,etc. gets conducted on smaller, homogeneous datas