CypherBench：朝向在LLM時代對全尺度現代知識圖進行精確檢索

摘要

從圖形數據中檢索對於增強大型語言模型（LLM）具有關鍵意義，因為它能夠提供開放領域知識和私人企業數據，同時也是最近的GraphRAG系統（edge等，2024年）的關鍵組成部分。儘管在知識圖和知識庫問答方面進行了數十年的研究，但領先的LLM框架（例如Langchain和LlamaIndex）對於從現代百科知識圖（如Wikidata）中檢索僅提供了最低限度的支持。在本文中，我們分析了根本原因並建議，現代RDF知識圖（例如Wikidata、Freebase）對於LLM來說效率較低，原因在於其過於龐大的架構遠超過典型的LLM上下文窗口，使用資源識別符、重疊的關係類型和缺乏規範化。作為解決方案，我們提出在底層RDF圖之上的屬性圖視圖，可以通過使用Cypher有效地由LLM進行查詢。我們在Wikidata上實現了這一想法，並引入了CypherBench，這是第一個具有11個大規模、多領域屬性圖的基準測試，其中包含780萬個實體和超過10,000個問題。為實現此目標，我們應對了幾個關鍵挑戰，包括開發一個RDF到屬性圖轉換引擎、創建一個系統化的文本到Cypher任務生成流程，以及設計新的評估指標。

English

Retrieval from graph data is crucial for augmenting large language models (LLM) with both open-domain knowledge and private enterprise data, and it is also a key component in the recent GraphRAG system (edge et al., 2024). Despite decades of research on knowledge graphs and knowledge base question answering, leading LLM frameworks (e.g. Langchain and LlamaIndex) have only minimal support for retrieval from modern encyclopedic knowledge graphs like Wikidata. In this paper, we analyze the root cause and suggest that modern RDF knowledge graphs (e.g. Wikidata, Freebase) are less efficient for LLMs due to overly large schemas that far exceed the typical LLM context window, use of resource identifiers, overlapping relation types and lack of normalization. As a solution, we propose property graph views on top of the underlying RDF graph that can be efficiently queried by LLMs using Cypher. We instantiated this idea on Wikidata and introduced CypherBench, the first benchmark with 11 large-scale, multi-domain property graphs with 7.8 million entities and over 10,000 questions. To achieve this, we tackled several key challenges, including developing an RDF-to-property graph conversion engine, creating a systematic pipeline for text-to-Cypher task generation, and designing new evaluation metrics.

CypherBench：朝向在LLM時代對全尺度現代知識圖進行精確檢索

CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era

摘要

Summary

Support