訓練噪聲標記修剪

Training Noise Token Pruning

November 27, 2024

作者: Mingxing Rao, Bohan Jiang, Daniel Moyer

cs.AI

摘要

在本研究中，我們提出了用於視覺Transformer的訓練噪聲標記（TNT）剪枝。我們的方法將離散標記丟棄條件放寬為連續的添加性噪聲，在訓練中提供平滑的優化，同時在部署設置中保留離散丟棄的計算優勢。我們在ImageNet數據集上使用ViT和DeiT架構進行理論連接到速率失真文獻的實證評估，展示了TNT相對於先前剪枝方法的優勢。

English

In the present work we present Training Noise Token (TNT) Pruning for vision transformers. Our method relaxes the discrete token dropping condition to continuous additive noise, providing smooth optimization in training, while retaining discrete dropping computational gains in deployment settings. We provide theoretical connections to Rate-Distortion literature, and empirical evaluations on the ImageNet dataset using ViT and DeiT architectures demonstrating TNT's advantages over previous pruning methods.