Hypencoder：用于信息检索的超网络

Hypencoder: Hypernetworks for Information Retrieval

February 7, 2025

作者: Julian Killingback, Hansi Zeng, Hamed Zamani

cs.AI

摘要

绝大多数检索模型依赖于向量内积来生成查询和文档之间的相关性分数。这自然地限制了可用的相关性分数的表达能力。我们提出了一种新的范式，不是生成一个向量来表示查询，而是生成一个作为学习相关性函数的小型神经网络。这个小型神经网络接收文档的表示，本文中我们使用一个单一向量，并生成一个标量相关性分数。为了生成这个小型神经网络，我们使用一个超网络，即一个生成其他网络权重的网络，作为我们的查询编码器或者我们称之为Hypencoder。在领域内搜索任务上的实验表明，Hypencoder能够显著优于强大的密集检索模型，并且比重新排序模型和规模大一个数量级的模型具有更高的指标。Hypencoder还表现出对领域外搜索任务的良好泛化能力。为了评估Hypencoder的能力程度，我们在一组困难的检索任务上进行评估，包括“差一点就想起来”的检索和遵循指令的检索任务，并发现与标准检索任务相比，性能差距显著扩大。此外，为了展示我们方法的实用性，我们实现了一个近似搜索算法，并展示我们的模型能够在不到60毫秒的时间内搜索880万个文档。

English

The vast majority of retrieval models depend on vector inner products to produce a relevance score between a query and a document. This naturally limits the expressiveness of the relevance score that can be employed. We propose a new paradigm, instead of producing a vector to represent the query we produce a small neural network which acts as a learned relevance function. This small neural network takes in a representation of the document, in this paper we use a single vector, and produces a scalar relevance score. To produce the little neural network we use a hypernetwork, a network that produce the weights of other networks, as our query encoder or as we call it a Hypencoder. Experiments on in-domain search tasks show that Hypencoder is able to significantly outperform strong dense retrieval models and has higher metrics then reranking models and models an order of magnitude larger. Hypencoder is also shown to generalize well to out-of-domain search tasks. To assess the extent of Hypencoder's capabilities, we evaluate on a set of hard retrieval tasks including tip-of-the-tongue retrieval and instruction-following retrieval tasks and find that the performance gap widens substantially compared to standard retrieval tasks. Furthermore, to demonstrate the practicality of our method we implement an approximate search algorithm and show that our model is able to search 8.8M documents in under 60ms.