最小熵耦合與瓶頸

Minimum Entropy Coupling with Bottleneck

October 29, 2024

作者: M. Reza Ebrahimi, Jun Chen, Ashish Khisti

cs.AI

摘要

本文研究了一種在對數損失下運作的新型有損壓縮框架，旨在處理重建分佈與來源分佈不一致的情況。該框架尤其適用於需要聯合壓縮和檢索的應用，以及涉及由於處理而導致分佈變化的情況。我們展示了所提出的公式通過整合瓶頸，將經典的最小熵耦合框架擴展，從而允許在耦合中控制一定程度的隨機性。我們探索了將最小熵耦合與瓶頸（MEC-B）分解為兩個不同優化問題的方法：對於編碼器的熵限制信息最大化（EBIM）和對於解碼器的最小熵耦合（MEC）。通過廣泛的分析，我們提供了一種對EBIM的貪婪算法，保證性能，并對接近功能映射的最優解進行了表徵，從而對這個問題的結構復雜性提供了重要的理論見解。此外，我們通過在速率限制下的馬可夫編碼遊戲（MCGs）中的實驗來說明MEC-B的實際應用。這些遊戲模擬了馬可夫決策過程中的通信情景，其中一個代理必須通過其動作將壓縮消息從發送者傳輸給接收者。我們的實驗突顯了在各種壓縮速率下MDP獎勵和接收者準確性之間的權衡，展示了我們的方法相對於傳統壓縮基準的有效性。

English

This paper investigates a novel lossy compression framework operating under logarithmic loss, designed to handle situations where the reconstruction distribution diverges from the source distribution. This framework is especially relevant for applications that require joint compression and retrieval, and in scenarios involving distributional shifts due to processing. We show that the proposed formulation extends the classical minimum entropy coupling framework by integrating a bottleneck, allowing for a controlled degree of stochasticity in the coupling. We explore the decomposition of the Minimum Entropy Coupling with Bottleneck (MEC-B) into two distinct optimization problems: Entropy-Bounded Information Maximization (EBIM) for the encoder, and Minimum Entropy Coupling (MEC) for the decoder. Through extensive analysis, we provide a greedy algorithm for EBIM with guaranteed performance, and characterize the optimal solution near functional mappings, yielding significant theoretical insights into the structural complexity of this problem. Furthermore, we illustrate the practical application of MEC-B through experiments in Markov Coding Games (MCGs) under rate limits. These games simulate a communication scenario within a Markov Decision Process, where an agent must transmit a compressed message from a sender to a receiver through its actions. Our experiments highlight the trade-offs between MDP rewards and receiver accuracy across various compression rates, showcasing the efficacy of our method compared to conventional compression baseline.