机器学习的进化与Knightian盲点

Evolution and The Knightian Blindspot of Machine Learning

January 22, 2025

作者: Joel Lehman, Elliot Meyerson, Tarek El-Gaaly, Kenneth O. Stanley, Tarin Ziyaee

cs.AI

摘要

本文声称，机器学习（ML）在很大程度上忽视了普遍智能的一个重要方面：对未知未来的鲁棒性，尤其是在一个开放世界中。这种鲁棒性与经济学中的Knightian不确定性（KU）有关，即无法量化的不确定性，在ML的关键形式化中被排除在考虑之外。本文旨在识别这一盲点，论证其重要性，并催生研究以解决这一问题，我们认为这对于创造真正鲁棒的开放世界人工智能是必要的。为了帮助阐明这一盲点，我们将ML的一个领域，强化学习（RL），与生物进化过程进行对比。尽管RL取得了惊人的持续进展，但在开放世界的情况下仍然面临困难，经常在意想不到的情况下失败。例如，目前将仅在美国训练过的自动驾驶汽车政策零-shot转移到英国的想法似乎过于雄心勃勃。戏剧性的对比是，生物进化经常产生在开放世界中茁壮成长的个体，有时甚至适应了非常不同的情况（例如入侵物种；或者人类，他们确实进行了这种零-shot国际驾驶）。有趣的是，进化在没有明确理论、形式化或数学梯度的情况下实现了这种鲁棒性。我们探讨了支撑RL典型形式化的假设，展示了它们如何限制了RL与不断变化的复杂世界特征中的未知未知的接触。此外，我们确定了进化过程促进对新颖和不可预测挑战的鲁棒性的机制，并讨论了在算法上体现这些机制的潜在途径。结论是，ML仍然存在引人注目的脆弱性可能是由于其形式化中的盲点，直接面对KU挑战可能会带来显著收益。

English

This paper claims that machine learning (ML) largely overlooks an important facet of general intelligence: robustness to a qualitatively unknown future in an open world. Such robustness relates to Knightian uncertainty (KU) in economics, i.e. uncertainty that cannot be quantified, which is excluded from consideration in ML's key formalisms. This paper aims to identify this blind spot, argue its importance, and catalyze research into addressing it, which we believe is necessary to create truly robust open-world AI. To help illuminate the blind spot, we contrast one area of ML, reinforcement learning (RL), with the process of biological evolution. Despite staggering ongoing progress, RL still struggles in open-world situations, often failing under unforeseen situations. For example, the idea of zero-shot transferring a self-driving car policy trained only in the US to the UK currently seems exceedingly ambitious. In dramatic contrast, biological evolution routinely produces agents that thrive within an open world, sometimes even to situations that are remarkably out-of-distribution (e.g. invasive species; or humans, who do undertake such zero-shot international driving). Interestingly, evolution achieves such robustness without explicit theory, formalisms, or mathematical gradients. We explore the assumptions underlying RL's typical formalisms, showing how they limit RL's engagement with the unknown unknowns characteristic of an ever-changing complex world. Further, we identify mechanisms through which evolutionary processes foster robustness to novel and unpredictable challenges, and discuss potential pathways to algorithmically embody them. The conclusion is that the intriguing remaining fragility of ML may result from blind spots in its formalisms, and that significant gains may result from direct confrontation with the challenge of KU.