擲骰子並三思而後行：突破下一個詞預測的創造力界限

摘要

我們設計了一套最小化的算法任務，這些任務是對開放式現實世界任務的鬆散抽象。這使我們能夠清晰且可控地量化當今語言模型的創造力極限。與需要創造性、遠見性思維躍遷的現實世界任務類似，我們的任務需要一個隱含的、開放式的隨機規劃步驟，該步驟要麼（a）在抽象的知識圖譜中發現新的聯繫（如文字遊戲、類比推理或研究），要麼（b）構建新的模式（如設計數學問題或新蛋白質）。在這些任務中，我們從經驗和概念上論證了下一詞學習的短視性及其過度記憶的傾向；相比之下，多詞方法，即無教師訓練和擴散模型，在生成多樣且原創的輸出方面表現出色。其次，在我們的任務中，我們發現要在不損害連貫性的前提下從Transformer中激發隨機性，最好在輸入層直接注入噪聲（通過我們稱之為哈希條件化的方法），而非依賴於輸出層的溫度採樣。因此，我們的工作為分析開放式創造技能提供了一個原則性的最小化測試平台，並為超越下一詞學習和基於softmax的採樣提供了新的論據。我們在https://github.com/chenwu98/algorithmic-creativity上公開了部分代碼。

English

We design a suite of minimal algorithmic tasks that are a loose abstraction of open-ended real-world tasks. This allows us to cleanly and controllably quantify the creative limits of the present-day language model. Much like real-world tasks that require a creative, far-sighted leap of thought, our tasks require an implicit, open-ended stochastic planning step that either (a) discovers new connections in an abstract knowledge graph (like in wordplay, drawing analogies, or research) or (b) constructs new patterns (like in designing math problems or new proteins). In these tasks, we empirically and conceptually argue how next-token learning is myopic and memorizes excessively; comparatively, multi-token approaches, namely teacherless training and diffusion models, excel in producing diverse and original output. Secondly, in our tasks, we find that to elicit randomness from the Transformer without hurting coherence, it is better to inject noise right at the input layer (via a method we dub hash-conditioning) rather than defer to temperature sampling from the output layer. Thus, our work offers a principled, minimal test-bed for analyzing open-ended creative skills, and offers new arguments for going beyond next-token learning and softmax-based sampling. We make part of the code available under https://github.com/chenwu98/algorithmic-creativity

擲骰子並三思而後行：突破下一個詞預測的創造力界限

Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction

摘要

Summary

Support

Support