超稀疏记忆网络

摘要

众所周知，Transformer模型的性能与其参数数量和计算复杂度呈指数关系。虽然像专家混合（MoE）这样的方法将参数数量与计算复杂度分离，但由于高内存访问成本，它们在推断过程中仍面临挑战。本研究引入了UltraMem，将大规模、超稀疏内存层融入其中，以解决这些限制。我们的方法显著降低了推断延迟，同时保持模型性能。我们还研究了这种新架构的扩展规律，表明它不仅具有良好的扩展特性，而且优于传统模型。在我们的实验中，我们训练了具有多达2000万个内存槽的网络。结果显示，我们的方法在给定的计算预算内实现了最先进的推断速度和模型性能。

English

It is widely acknowledged that the performance of Transformer models is exponentially related to their number of parameters and computational complexity. While approaches like Mixture of Experts (MoE) decouple parameter count from computational complexity, they still face challenges in inference due to high memory access costs. This work introduces UltraMem, incorporating large-scale, ultra-sparse memory layer to address these limitations. Our approach significantly reduces inference latency while maintaining model performance. We also investigate the scaling laws of this new architecture, demonstrating that it not only exhibits favorable scaling properties but outperforms traditional models. In our experiments, we train networks with up to 20 million memory slots. The results show that our method achieves state-of-the-art inference speed and model performance within a given computational budget.

超稀疏记忆网络

Ultra-Sparse Memory Network

摘要

Summary

Support