傅立葉分析網絡：Fourier Analysis Networks

摘要

儘管神經網絡，特別是MLP和Transformer所代表的神經網絡取得了顯著的成功，我們揭示了它們在建模和推理週期性方面存在潛在缺陷，即它們傾向於記憶週期性數據，而非真正理解週期性的基本原則。然而，週期性是各種形式推理和泛化的關鍵特徵，在自然和工程系統中透過觀察中的重複模式支撐可預測性。在本文中，我們提出了基於傅立葉分析的新型網絡架構FAN，它賦予了有效地建模和推理週期性現象的能力。通過引入傅立葉級數，週期性被自然地融入神經網絡的結構和計算過程中，從而實現對週期性模式更準確的表達和預測。作為多層感知器（MLP）的一個有前途的替代方案，FAN可以在各種模型中無縫地取代MLP，並具有更少的參數和FLOPs。通過大量實驗，我們展示了FAN在建模和推理週期函數方面的有效性，以及FAN在一系列現實任務中的優越性和泛化能力，包括符號公式表示、時間序列預測和語言建模。

English

Despite the remarkable success achieved by neural networks, particularly those represented by MLP and Transformer, we reveal that they exhibit potential flaws in the modeling and reasoning of periodicity, i.e., they tend to memorize the periodic data rather than genuinely understanding the underlying principles of periodicity. However, periodicity is a crucial trait in various forms of reasoning and generalization, underpinning predictability across natural and engineered systems through recurring patterns in observations. In this paper, we propose FAN, a novel network architecture based on Fourier Analysis, which empowers the ability to efficiently model and reason about periodic phenomena. By introducing Fourier Series, the periodicity is naturally integrated into the structure and computational processes of the neural network, thus achieving a more accurate expression and prediction of periodic patterns. As a promising substitute to multi-layer perceptron (MLP), FAN can seamlessly replace MLP in various models with fewer parameters and FLOPs. Through extensive experiments, we demonstrate the effectiveness of FAN in modeling and reasoning about periodic functions, and the superiority and generalizability of FAN across a range of real-world tasks, including symbolic formula representation, time series forecasting, and language modeling.

傅立葉分析網絡：Fourier Analysis Networks

FAN: Fourier Analysis Networks

摘要

Summary

Support

Support