CodeGen2：训练大型语言模型处理编程和自然语言的经验教训（英文版）

英文标题：CodeGen2: Lessons for Training LLMs on Programming and Natural Languages中文摘要：本文研究如何通过整合模型架构、学习方法、填充采样和数据分布等四个关键组件来提高大型语言模型在程序综合方面的训练效率，并在 1B LLMs 上开展了一系列实验，提炼出四个教训并发布了 CodeGen2 模型和训练框架。英文摘要：Large language models (LLMs) have demonstrated remarkable abilities inrepresentation learning for program synthesis and understanding tasks. Thequality of the learned representations appears to be dictated by the neuralscaling laws as a function of the number of model parameters and observations,while im