訓練方法如何影響視覺模型的利用？

How Do Training Methods Influence the Utilization of Vision Models?

October 18, 2024

作者: Paul Gavrikov, Shashank Agnihotri, Margret Keuper, Janis Keuper

cs.AI

摘要

並非所有可學習的參數（例如權重）對神經網絡的決策功能貢獻相同。事實上，有時整個層的參數可以被重置為隨機值，對模型的決策幾乎沒有影響。我們重新審視早期研究，探討架構和任務複雜性如何影響這一現象，並提出問題：這一現象是否也受我們訓練模型的方式影響？我們對多個ImageNet-1k分類模型進行了實驗評估，探索這一問題，保持架構和訓練數據恆定，但變化訓練流程。我們的研究發現顯示，訓練方法強烈影響哪些層對於特定任務的決策功能至關重要。例如，改進的訓練制度和自監督訓練增加了早期層的重要性，同時明顯地未充分利用更深層。相反，諸如對抗訓練等方法呈現相反的趨勢。我們的初步結果擴展了先前的研究發現，提供了對神經網絡內部機制更細緻的理解。 代碼：https://github.com/paulgavrikov/layer_criticality

English

Not all learnable parameters (e.g., weights) contribute equally to a neural network's decision function. In fact, entire layers' parameters can sometimes be reset to random values with little to no impact on the model's decisions. We revisit earlier studies that examined how architecture and task complexity influence this phenomenon and ask: is this phenomenon also affected by how we train the model? We conducted experimental evaluations on a diverse set of ImageNet-1k classification models to explore this, keeping the architecture and training data constant but varying the training pipeline. Our findings reveal that the training method strongly influences which layers become critical to the decision function for a given task. For example, improved training regimes and self-supervised training increase the importance of early layers while significantly under-utilizing deeper layers. In contrast, methods such as adversarial training display an opposite trend. Our preliminary results extend previous findings, offering a more nuanced understanding of the inner mechanics of neural networks. Code: https://github.com/paulgavrikov/layer_criticality