最小熵耦合与瓶颈

Minimum Entropy Coupling with Bottleneck

October 29, 2024

作者: M. Reza Ebrahimi, Jun Chen, Ashish Khisti

cs.AI

摘要

本文研究了一种新颖的损失压缩框架，其在对数损失下运行，旨在处理重建分布与源分布不一致的情况。该框架特别适用于需要联合压缩和检索的应用程序，以及涉及由于处理而导致分布偏移的场景。我们展示了所提出的公式通过集成瓶颈扩展了经典的最小熵耦合框架，从而允许在耦合中控制一定程度的随机性。我们探讨了最小熵耦合与瓶颈（MEC-B）的分解为两个不同的优化问题：编码器的熵约束信息最大化（EBIM）和解码器的最小熵耦合（MEC）。通过广泛的分析，我们提供了一种具有保证性能的贪婪算法用于EBIM，并表征了在功能映射附近的最优解，为这一问题的结构复杂性提供了重要的理论见解。此外，我们通过在速率限制下的马尔可夫编码游戏（MCGs）中的实验展示了MEC-B的实际应用。这些游戏模拟了马尔可夫决策过程中的通信场景，其中一个代理必须通过其动作将压缩消息从发送方传输到接收方。我们的实验突显了在各种压缩率下MDP奖励和接收方准确性之间的权衡，展示了我们的方法相对于传统压缩基准的有效性。

English

This paper investigates a novel lossy compression framework operating under logarithmic loss, designed to handle situations where the reconstruction distribution diverges from the source distribution. This framework is especially relevant for applications that require joint compression and retrieval, and in scenarios involving distributional shifts due to processing. We show that the proposed formulation extends the classical minimum entropy coupling framework by integrating a bottleneck, allowing for a controlled degree of stochasticity in the coupling. We explore the decomposition of the Minimum Entropy Coupling with Bottleneck (MEC-B) into two distinct optimization problems: Entropy-Bounded Information Maximization (EBIM) for the encoder, and Minimum Entropy Coupling (MEC) for the decoder. Through extensive analysis, we provide a greedy algorithm for EBIM with guaranteed performance, and characterize the optimal solution near functional mappings, yielding significant theoretical insights into the structural complexity of this problem. Furthermore, we illustrate the practical application of MEC-B through experiments in Markov Coding Games (MCGs) under rate limits. These games simulate a communication scenario within a Markov Decision Process, where an agent must transmit a compressed message from a sender to a receiver through its actions. Our experiments highlight the trade-offs between MDP rewards and receiver accuracy across various compression rates, showcasing the efficacy of our method compared to conventional compression baseline.