UniMax: 大规模多语言预训练中更公平、更有效的语言采样（英文版）

英文标题：UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining中文摘要：本文提出了一种新的采样方法 UniMax，可在平衡语言之间的差异并减轻尾部语言的过拟合的同时，提供更加均匀的头部语言覆盖，并在一系列多语言评估基准测试中证明了 UniMax 的优越性和其随着模型规模的增加而持续的优点。英文摘要：Pretrained multilingual large language models have typically used heuristictemperature-based sampling to balance between different languages. Howeverprevious work has not systematically evaluated the efficacy of differentpretraining language distributions across model scales. In thi