对大型语言模型生成的合成数据中质量、多样性和复杂性的影响进行调查

Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models

December 4, 2024

作者: Alex Havrilla, Andrew Dai, Laura O'Mahony, Koen Oostermeijer, Vera Zisler, Alon Albalak, Fabrizio Milo, Sharath Chandra Raparthy, Kanishk Gandhi, Baber Abbasi, Duy Phung, Maia Iyer, Dakota Mahan, Chase Blagden, Srishti Gureja, Mohammed Hamdy, Wen-Ding Li, Giovanni Paolini, Pawan Sasanka Ammanamanchi, Elliot Meyerson

cs.AI

摘要

利用大型语言模型进行合成数据生成是一种有前途的范式，可用于增加几乎无限范围的任务的自然数据。鉴于这种多样性，合成数据生成算法之间的直接比较很少，这使得难以理解改进来自何处以及存在哪些瓶颈。我们建议通过每个算法生成的合成数据的数据质量、多样性和复杂性来评估算法。我们选择这三个特征是因为它们在开放式过程中的重要性以及它们对下游模型能力的影响。我们发现质量对于分布内模型泛化至关重要，多样性对于分布外泛化至关重要，而复杂性对两者都有益。此外，我们强调在训练数据中存在质量-多样性的权衡以及对模型性能的下游影响。然后，我们检查合成数据管道中各个组件对每个数据特征的影响。这种检查使我们能够通过它们利用的组件以及对数据QDC组成产生的影响来对合成数据生成算法进行分类和比较。这种分析延伸到对在合成数据中平衡QDC对于高效强化学习和自我改进算法的重要性的讨论。类似于训练数据中的QD权衡，通常存在模型输出质量和输出多样性之间的权衡，这些权衡影响合成数据的组成。我们观察到目前许多模型仅被评估和优化为输出质量，从而限制了输出多样性和自我改进的潜力。我们认为平衡这些权衡对于未来自我改进算法的发展至关重要，并强调一些在这方向取得进展的工作。

English

Synthetic data generation with Large Language Models is a promising paradigm for augmenting natural data over a nearly infinite range of tasks. Given this variety, direct comparisons among synthetic data generation algorithms are scarce, making it difficult to understand where improvement comes from and what bottlenecks exist. We propose to evaluate algorithms via the makeup of synthetic data generated by each algorithm in terms of data quality, diversity, and complexity. We choose these three characteristics for their significance in open-ended processes and the impact each has on the capabilities of downstream models. We find quality to be essential for in-distribution model generalization, diversity to be essential for out-of-distribution generalization, and complexity to be beneficial for both. Further, we emphasize the existence of Quality-Diversity trade-offs in training data and the downstream effects on model performance. We then examine the effect of various components in the synthetic data pipeline on each data characteristic. This examination allows us to taxonomize and compare synthetic data generation algorithms through the components they utilize and the resulting effects on data QDC composition. This analysis extends into a discussion on the importance of balancing QDC in synthetic data for efficient reinforcement learning and self-improvement algorithms. Analogous to the QD trade-offs in training data, often there exist trade-offs between model output quality and output diversity which impact the composition of synthetic data. We observe that many models are currently evaluated and optimized only for output quality, thereby limiting output diversity and the potential for self-improvement. We argue that balancing these trade-offs is essential to the development of future self-improvement algorithms and highlight a number of works making progress in this direction.