MineWorld：一个基于《我的世界》的实时开源交互式世界模型

MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft

April 11, 2025

作者: Junliang Guo, Yang Ye, Tianyu He, Haoyu Wu, Yushu Jiang, Tim Pearce, Jiang Bian

cs.AI

摘要

世界建模是使智能体能够有效与人类互动并在动态环境中运作的关键任务。在本研究中，我们提出了MineWorld，一个基于Minecraft的实时交互式世界模型，Minecraft作为一个开放式的沙盒游戏，已被广泛用作世界建模的通用测试平台。MineWorld由一个视觉-动作自回归Transformer驱动，该模型以成对的游戏场景及相应动作为输入，生成遵循这些动作的后续新场景。具体而言，通过使用图像分词器和动作分词器分别将视觉游戏场景和动作转换为离散的token ID，我们将这两类ID交错拼接构成模型输入。随后，模型通过下一token预测进行训练，以同时学习游戏状态的丰富表示以及状态与动作之间的条件关系。在推理阶段，我们开发了一种新颖的并行解码算法，该算法同时预测每帧中的空间冗余token，使得不同规模的模型能够每秒生成4至7帧，实现与游戏玩家的实时交互。在评估中，我们提出了新的指标，不仅评估生成新场景的视觉质量，还评估其动作跟随能力，这对于世界模型至关重要。我们的全面评估展示了MineWorld的有效性，显著超越了当前最先进的基于扩散的开源世界模型。代码与模型已公开发布。

English

World modeling is a crucial task for enabling intelligent agents to effectively interact with humans and operate in dynamic environments. In this work, we propose MineWorld, a real-time interactive world model on Minecraft, an open-ended sandbox game which has been utilized as a common testbed for world modeling. MineWorld is driven by a visual-action autoregressive Transformer, which takes paired game scenes and corresponding actions as input, and generates consequent new scenes following the actions. Specifically, by transforming visual game scenes and actions into discrete token ids with an image tokenizer and an action tokenizer correspondingly, we consist the model input with the concatenation of the two kinds of ids interleaved. The model is then trained with next token prediction to learn rich representations of game states as well as the conditions between states and actions simultaneously. In inference, we develop a novel parallel decoding algorithm that predicts the spatial redundant tokens in each frame at the same time, letting models in different scales generate 4 to 7 frames per second and enabling real-time interactions with game players. In evaluation, we propose new metrics to assess not only visual quality but also the action following capacity when generating new scenes, which is crucial for a world model. Our comprehensive evaluation shows the efficacy of MineWorld, outperforming SoTA open-sourced diffusion based world models significantly. The code and model have been released.