# MultiRAG: A Knowledge-guided Framework for Mitigating Hallucination in Multi-source Retrieval Augmented Generation Wenlong Wu \( {}^{1} \) , Haofen Wang \( {}^{2} \) , Bohan Li \( {}^{1,3,4\text{ ✉ }} \) , Peixuan Huang \( {}^{1} \) , Xinzhe Zhao \( {}^{1} \) and Lei Liang \( {}^{5} \; {}^{1} \) College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education \( {}^{2} \) College of Design &Innovation, Tongji University \( {}^{3} \) Key Laboratory of Intelligent Decision and Digital Operation, Ministry of Industry and Information Technology \( {}^{4} \) Collaborative Innovation Center of Novel Software Technology and Industrialization \( {}^{5} \) Ant Group Knowledge Graph Team Email: \{wuwenlong, bhli, peixuanh, xinzhe_zhao\}@nuaa.edu.cn carter.whfcarter@gmail.com, leywar.liang@antgroup.com Abstract-Retrieval Augmented Generation (RAG) has emerged as a promising solution to address hallucination issues in Large Language Models (LLMs). However, the integration of multiple retrieval sources, while potentially more informative, introduces new challenges that can paradoxically exacerbate hallucination problems. These challenges manifest primarily in two aspects: the sparse distribution of multi-source data that hinders the capture of logical relationships and the inherent inconsistencies among different sources that lead to information conflicts. To address these challenges, we propose MultiRAG, a novel framework designed to mitigate hallucination in multi-source retrieval-augmented generation through knowledge-guided approaches. Our framework introduces two key innovations: (1) a knowledge construction module that employs multi-source line graphs to efficiently aggregate logical relationships across different knowledge sources, effectively addressing the sparse data distribution issue; and (2) a sophisticated retrieval module that implements a multi-level confidence calculation mechanism, performing both graph-level and node-level assessments to identify and eliminate unreliable information nodes, thereby reducing hallucinations caused by inter-source inconsistencies. Extensive experiments on four multi-domain query datasets and two multi-hop QA datasets demonstrate that MultiRAG significantly enhances the reliability and efficiency of knowledge retrieval in complex multi-source scenarios. Our code is available in https://github.com/wuwenlong123/MultiRAG Index Terms-Retrieval Augmented Generation, Large Language Models, Multi-source Retrieval, Knowledge Graphs, Hallucination Mitigation ## I. INTRODUCTION Large Language Models (LLMs) have achieved remarkable success in handling a variety of natural language processing tasks, attributable to their robust capabilities in understanding and generating language and symbols [1]. In knowledge-intensive retrieval tasks, Retrieval Augmented Generation (RAG) has become a standardized solution paradigm [2]- [4]. Previous works [5]-[11] have made significant strides in addressing the inherent knowledge limitations of LLMs. By introducing external knowledge bases, it has markedly improved the accuracy and fidelity of LLM responses. However, recent studies have highlighted a significant drawback: the retrieval results of RAG are imperfect, including irrelevant, misleading, and even malicious information, ultimately leading to inaccurate LLM responses. To address these limitations, the synergy between LLMs and Knowledge Graphs (KGs) has been proposed to achieve more efficient information retrieval [12]. On one hand, KG can efficiently store data with fixed characteristics (such as temporal KGs, event KGs, etc.), thereby enhancing the processing capabilities of LLMs on specific data [13]-[20]. On the other hand, the collaboration between LLMs and KGs has significantly improved performance in multi-hop and multi-document question answering, including the credibility and interpretability of retrieval [21]. Furthermore, LLM-KG collaborative methods have also provided the latest solutions for knowledge-intensive retrieval tasks [22]-[26], propelling the deep reasoning capabilities of RAG. Nevertheless, existing frameworks still fail to account for the complexity of real-world data. Although RAG can mitigate the generation of hallucinations, these hallucinations often stem from the internal knowledge of LLMs [27]-[29]. Inconsistent information sources and unreliable retrieval methods can still lead to retrieval biases and hallucinations in LLMs. This issue becomes particularly pronounced when dealing with information retrieval tasks that involve multi-source knowledge, where hallucinations are more prominent. Research [30] indicates that approximately 70% of retrieved paragraphs do not directly contain the correct query answers but instead include information indirectly related to the answers, causing misguidance and comprehension bias in LLMs. Building upon the categorization of hallucinations in retrieval [9], we outlines the three most common types of hallucinations encountered in multi-source data retrieval: --- Wenlong Wu and Haofen Wang contributed equally to this work. Bohan Li is the corresponding author. --- ![bo_d6oh96c601uc73e30m7g_1_133_134_762_691_0.jpg](images/bo_d6oh96c601uc73e30m7g_1_133_134_762_691_0.jpg) Fig. 1: Single-source Retrieval & Multi-source Retrieval 1) Inter-source data inconsistency: Discrepancies between different data sources can lead to conflicting information, causing hallucinations in LLMs. 2) Redundancy of similar data: There often exists data that is highly similar and semantically equivalent across multiple data sources, which can impose significant computational overhead on retrieval. 3) Incomplete inference paths: Forming a comprehensive inference path from different data sources is challenging. Existing retrievers often fail to capture the complete logical associations within multiple data sources. Fig. 1 vividly illustrates the differences between single-source and multi-source data retrieval through CA981 flight analysis. The sparse distribution and inconsistency of data are unique issues in multi-source data retrieval, leading to severe hallucination bias in LLMs. Against this backdrop, we focus on addressing the issue of retrieval hallucinations in multi-source data retrieval to empower knowledge-augmented generation. This work primarily explores the following two fundamental challenges: 1) Sparse Distribution of Multi-source Data: Multi-domain queries require fusing structured (SQL tables), semi-structured (JSON logs), and unstructured data (text reports). Due to the variability in data storage formats and sparsity, the connectivity between knowledge elements is low, making it difficult for RAG systems to effectively capture logical associations across sources, thereby affecting the recall rate and quality of retrieval results. 2) Inter-source Data Inconsistency: Conversely, the inherent diversity in knowledge representations across multi-source data often leads to inconsistencies in retrieved fragments. These discrepancies may induce information conflicts during retrieval processes, thereby compromising response accuracy. This challenge becomes particularly pronounced in domain-specific complex reasoning and multi-hop question answering tasks. To address these issues above, we propose MultiRAG, a novel framework designed to mitigate hallucination in multi-source retrieval augmented generation through knowledge-guided approaches. Initially, we introduce multi-source line graphs for rapid aggregation of knowledge sources to tackle issues arising from sparse data distribution. Subsequently, based on these integrated multi-source line graphs, we propose a multi-level confidence calculation method to ensure the reliability of multi-source data queries. This approach not only enhances query efficiency but also strengthens the accuracy of results, providing an effective solution for the multi-source knowledge-guided RAG. The contributions of this paper are summarized as follows: 1) Multi-source Knowledge Aggregation: In the knowledge construction module, we introduce multi-source line graphs as a data structure for rapid aggregation and reconstruction of knowledge structures from multiple query-relevant data sources. This effectively captures inter-source data dependencies within chunk texts, thereby providing a unified and centralized representation of multi-source knowledge. 2) Multi-level Confidence Calculation: In the retrieval module, we perform graph-level and node-level confidence calculations on the extracted knowledge subgraphs. The aim is to filter out and eliminate low-quality subgraphs and inconsist retrieval nodes, ultimately enhancing the quality of text embedded in context to alleviate retrieval hallucinations. 3) Experimental Validation and Performance Comparison: We conducted extensive experiments on existing multi-source retrieval datasets and two complex Q&A datasets, comparing our approach with existing state-of-the-art(SOTA) methods. This demonstrated the robustness and accuracy of our proposed method in retrieval performance. Particularly in multi-source data retrieval tasks, our method significantly outperforms other SOTA methods by more than 10%. ## II. PRELIMINARY In the field of Knowledge-Guided RAG, the primary challenges include efficiently accessing relevant knowledge and achieving reliable retrieval performance. This section introduces the core elements of our approach and precisely defines the problems we address. Let \( Q = \left\{ {{q}_{1},{q}_{2},\ldots ,{q}_{n}}\right\} \) be the set of query instances, where each \( {q}_{i} \) corresponds to a distinct query. Let \( E = \; \left\{ {{e}_{1},{e}_{2},\ldots ,{e}_{m}}\right\} \) be the set of entities in the knowledge graph, where each \( {e}_{j} \) represents an entity. Let \( R = \left\{ {{r}_{1},{r}_{2},\ldots ,{r}_{p}}\right\} \) be the set of relationships in the knowledge graph, where each \( {r}_{k} \) represents a relationship. Let \( D = \left\{ {{d}_{1},{d}_{2},\ldots ,{d}_{t}}\right\} \) be the set of documents, where each \( {d}_{l} \) represents a document. We define the knowledge-guided retrieval enhancement generation problem as follows: \[ \arg \mathop{\max }\limits_{{{d}_{i} \in D}}{LLM}\left( {{q}_{i},{d}_{i}}\right) ,\mathop{\sum }\limits_{{{e}_{j} \in E}}\mathop{\sum }\limits_{{{r}_{k} \in R}}{KG}\left( {{e}_{j},{r}_{k},{d}_{i}}\right) \tag{1} \] where \( \operatorname{LLM}\left( {{q}_{i},{d}_{l}}\right) \) denotes the score of the relevance between query \( {q}_{i} \) and document \( {d}_{l} \) assessed by the LLM, and \( \mathrm{{KG}}\left( {{e}_{j},{r}_{k},{d}_{l}}\right) \) represents the degree of match between entity \( {e}_{j} \) , relationship \( {r}_{k} \) , and document \( {d}_{l} \) . Furthermore, we optimize the knowledge construction and retrieval modules by introducing multi-source line graphs to accelerate knowledge establishment and enhance retrieval robustness. Specifically, the proposed approach is formally defined as follows: Definition 1. Multi-source data fusion. Given a set of sources \( H \) , the data \( D = \{ d, \) name, \( c, \) meta \( \} \) exists, where \( d \) represents the domain of data, \( c \) represents the content of the data file, name represents the file/attribute name, and meta represents the file metadata. Through a multi-source data fusion algorithm, we can obtain normalized data \( \widehat{D} = \; \{ {id}, d, \) name, \( {jsc}, \) meta,(cols_index) \( \} \) . Here, \( {id} \) represents the unique identifier for normalization, \( d \) indicates the domain where the data file is located, name denotes the data file name, meta denotes the file metadata, and jsc denotes the file content stored using JSON-LD. If the stored data is structured data or other data formats that can use a columnar storage model, the column index cols_index of all attributes will also be stored for rapid retrieval and query. Fig. 2 provides an example of JSON-LD format. Definition 2. Multi-source line graph [31]. Given a multi-source knowledge graph \( \mathcal{G} \) and a transformed knowledge graph \( {\mathcal{G}}^{\prime } \) (multi-source line graph, MLG), the MLG satisfies the following characteristics: 1) A node in \( {\mathcal{G}}^{\prime } \) represents a triplet. 2) There is an associated edge between any two nodes in \( {\mathcal{G}}^{\prime } \) if and only if the triples represented by these two nodes share a common node. Based on the definition, it can be inferred that MLG achieves high aggregation of related nodes, which can greatly improve the efficiency of data retrieval and accelerate subsequent retrieval and query algorithms. Definition 3. Multi-source homologous data. For any two nodes \( {v}_{1} \) and \( {v}_{2} \) in \( \mathcal{G} \) , they are defined as multi-source homologous if and only if they belong to the same retrieval candidate set in a single search. Definition 4. Homologous node and homologous subgraph. Given a set of mult-domain homologous data \( {SV} = \; {\left\{ {v}_{i}\right\} }_{i = 1}^{n} \) in the knowledge graph \( \mathcal{G} \) , we define the homologous center node as snode \( = \{ \) name, meta, num, \( C\left( v\right) \} \) , the set of homologous nodes as \( {U}_{\text{ snode }} \) , and the set of homologous edges as \( {E}_{\text{ snode }} \) . Here, name represents the common attribute name, meta denotes the identical file metadata, num indicates the number of homologous data instances, and \( C\left( v\right) \) represents the data confidence. We define the association edge between snode and \( {v}_{i} \) as \( {e}_{i} = {\left\{ {w}_{i}\right\} }_{i = 1}^{n} \) , where \( {w}_{i} \) represents the weight of node \( {v}_{i} \) in the data confidence calculation. Thus, the homologous center node and \( S\mathcal{G} \) together form the homologous subgraph subSG. --- \{ "@context": "https://json-ld.org/contexts/person.jsonld", "@id": "http://dbpedia.org/resource/John_Lennon", "name": "John Lennon", "born": "1940-10-09", "spouse": "http://dbpedia.org/resource/Cynthia_Lennon" \} --- Fig. 2: Data format of JSON-LD Definition 5. Homologous triple line graph. For all homologous subgraphs within the knowledge graph \( \mathcal{G} \) , they collectively constitute the homologous knowledge graph \( S\mathcal{G} \) . By performing a linear graph transformation on the homologous knowledge graph, we obtain the homologous triple line graph \( S{\mathcal{G}}^{\prime } \) . By constructing a homologous triple line graph, multi-source homologous data are aggregated into a single subgraph, centered around homologous nodes, enabling rapid consistency checks and conflict feedback for homologous data. Additionally, the knowledge graph contains a significant number of isolated nodes (i.e., nodes without homologous data), which are also incorporated into the homologous triple line graph. Definition 6. Candidate graph confidence and candidate node confidence. For a query \( Q\left( {q,\mathcal{G}}\right) \) on the knowledge graph \( \mathcal{G} \) , the corresponding Homologous line graph \( S{\mathcal{G}}^{\prime } \) is obtained. The candidate graph confidence is an estimation of the confidence in the candidate Homologous subgraph, assessing the overall credibility of the candidate graph; the candidate node confidence is an assessment of the confidence in individual node to determine the credibility of single attribute node. ## III. METHODOLOGY ## A. Framework of MultiRAG This section elaborates on the implementation approach of MultiRAG. As shown in Fig. 3, the first step involves segmenting and extracting multi-source data to construct the corresponding MLG, achieving preliminary aggregation of multi-source data; the second step requires reconstructing the MLG and performing subgraph extraction to identify candidate homologous subgraphs, ensuring consistent storage of homologous data for subsequent hallucination assessment; the third step involves calculating the graph-level and node-level confidence of the candidate subgraphs, eliminating low-quality nodes to enhance the credibility of the response, and returning the extracted trustworthy subgraphs to the LLM to form the final answer. Finally, integrating the aforementioned steps to form the Multi-source Line Graph Prompting algorithm, MKLGP. ## B. Multi-source Line Graph Construction The MultiRAG method initially employs an adapter structure to integrate multi-source data and standardize its storage format. For practical application scenarios, data is directly obtained from various non-homologous formats and transformed into a unified, normalized representation. Specifically, file names and metadata are parsed, and the domains to which the files belong are categorized. Subsequently, the data content is parsed and stored in JSON-LD format, thereby transforming it into linked data. Finally, unique identifiers are assigned to the data, resulting in normalized datasets. ![bo_d6oh96c601uc73e30m7g_3_208_135_1385_677_0.jpg](images/bo_d6oh96c601uc73e30m7g_3_208_135_1385_677_0.jpg) Fig. 3: Framework of MultiRAG, including three modules. Specifically, a unique adapter is designed for each distinct data format to facilitate data parsing. Although the implementation frameworks of these adapters are largely similar, it is essential to differentiate between the parsing processes for structured, semi-structured, and unstructured data. For structured data, parsing involves storing tabular information in JSON format, where attribute variables within the file are managed using a Decomposition Storage Model (DSM). This approach enables the extraction of all attribute information for consistency checks through the use of column indices. In the case of semi-structured data, parsing corresponds to storing tree-shaped data in JSON format with multi-layer nested structures. This data format lacks column indices and does not support fast retrieval, necessitating the use of tree or graph retrieval algorithms, such as DFS, for efficient searching. Finally, for unstructured data, the focus is currently limited to textual information, which is stored directly. Subsequent steps involve leveraging LLMs for entity and relationship extraction tasks to obtain the relevant information. The final integration of multi-source data can be expressed by the following formula: \[ {D}_{\text{ Fusion }} = \mathop{\bigcup }\limits_{{i = 1}}^{n}{A}_{i}\left( {D}_{i}\right) \tag{2} \] where \( {A}_{i} \in \left\{ {{Ad}{a}_{\text{ stru }},{Ad}{a}_{\text{ semi-s }},{Ad}{a}_{\text{ unstru }}}\right\} \) , representing the adapter parsing functions for structured data, semi-structured data, and unstructured data, respectively. \( {D}_{i} \in \; \left\{ {{D}_{\text{ stru }},{D}_{\text{ semi-s }},{D}_{\text{ unstru }}}\right\} \) represents the original datasets of structured data, semi-structured data, and unstructured data, respectively. Through the parsed data \( {D}_{\text{ Fusion }} = \left\{ {{E}_{\mathrm{q}},{R}_{\mathrm{q}}}\right\} \) , we further extracts key information and links it to the knowledge graph. The knowledge construction process involves three key phases implemented through the OpenSPG framework [26], [32], in which we use the Custom Prompt module \( {}^{2} \) to integrate LLM-based knowledge extraction. For entity recognition, we utilize the ner.py prompts within the kag/builder/prompt/default directory. We first define relevant entity types in the schema. Then, by adjusting the example.input and example.output in the ner.py prompts, we guide the LLM-based SchemaFreeExtractor to identify entities accurately. In relationship extraction, the triple.py prompts play a crucial role. We define relationships in the schema and use the triple_prompt in the SchemaFreeExtractor. The instruction in triple.py ensures that the extracted Subject-Predicate-Object(SPO) triples are related to the entities in the entity_list, enabling effective relationship extraction. Regarding attribute extraction, we rely on the entity standardization prompts in std.py. After entity recognition, the std_prompt in the SchemaFreeExtractor standardizes the entities and helps in extracting their attributes. We modify the example.input, example.named_entities, and example.output in std.py according to our data characteristics to optimize the attribute extraction process. Through these steps of customizing and applying OpenSPG's prompts, we achieve efficient knowledge extraction. The following formula describes the data extraction process: \[ {KB} = \mathop{\sum }\limits_{{D}_{i}}\left( {\left\{ {{e}_{1},{e}_{2},\ldots ,{e}_{m}}\right\} \sqcup \left\{ {{r}_{1},{r}_{2},\ldots ,{r}_{n}}\right\} }\right) \tag{3} \] --- https://github.com/OpenSPG/openspg https://openspg.yuque.com/ --- ![bo_d6oh96c601uc73e30m7g_4_178_142_662_431_0.jpg](images/bo_d6oh96c601uc73e30m7g_4_178_142_662_431_0.jpg) Fig. 4: Example of multi-source line graph transformation ## C. Homologous Subgraph Matching After the preliminary extraction of information, the next step is to identify the multi-source homologous data group set \( \mathcal{{SV}}s \) and the isolated point set \( \mathcal{{LV}}s \) . This process begins by initializing the unvisited node set \( {\mathcal{U}}_{\text{ unvisited }} = \mathcal{V} \) , while setting the homologous data group \( \mathcal{{SV}}s = \varnothing \) and the isolated point set \( \mathcal{L}\mathcal{V}s = \varnothing \) . By traversing all nodes and retrieving node information from various domains, for matched homologous data, construct the homologous node \( s{g}_{i} \) and its corresponding associated edge \( {e}_{i} \) , and add them to the homologous node set \( {\mathcal{U}}_{sg} \) and edge set \( {\mathcal{E}}_{sg} \) , respectively. After the traversal, add \( \left( {{\mathcal{U}}_{sg},{\mathcal{E}}_{sg}}\right) \) to \( \mathcal{S}\mathcal{V}s \) . If no homologous data is obtained after one round of traversal, add the node to the isolated point set \( \mathcal{L}\mathcal{V}s \) . After the traversal is completed, the node will be removed from the \( {\mathcal{U}}_{\text{ unvisited }} \) set. The time complexity of homologous subgraph matching is \( O\left( {n\log n}\right) \) , where \( n \) is the number of nodes in the knowledge graph \( \mathcal{G} \) . For each homologous subgraph in \( \mathcal{{SV}}s \) , homologous linear knowledge subgraph sub \( {\mathcal{{SG}}}^{\prime }{}_{i} \) is constructed by utilizing the homologous node set \( {\mathcal{U}}_{sg} \) and the homologous edge set \( {\mathcal{E}}_{sg} \) . Subsequently, all sub \( {\mathcal{{SG}}}^{\prime }{}_{i} \) and the isolated point set \( \mathcal{{LV}}s \) are aggregated to obtain the homologous linear knowledge graph \( S{\mathcal{G}}^{\prime } \) . It should be noted that \( S{\mathcal{G}}^{\prime } \) is solely used for consistency checks and retrieval queries of homologous data; other types of queries still conducts operations on the original knowledge graph \( \mathcal{G} \) . Here, we provide a simple example of a homologous triple line graph. As shown in Fig. 4, a homologous node is associated with 4 homologous data points. After transformation into a triple line graph, it forms a complete graph of order 4, indicating that the four triples are pairwise homologous. ## D. Multi-level Confidence Computing We define the candidate data from different domains obtained in a single retrieval as multi-source homologous data. These data have been extracted into a homologous line graph for temporary storage. Although targeting the same query object, they often provide inconsistent reference answers. Considering the varying retrieval errors, the multi-level confidence calculation method is adpoted in this framework. First, the confidence of individual homologous line graphs is calculated, followed by the confidence of each candidate node, to determine the final set of answer candidates. 1) Graph-Level Confidence Computing: In the first stage, a confidence calculation method based on mutual information entropy is introduced to assess the confidence of homologous line graphs. The core idea of this method is that if two nodes with the same attributes in a homologous line graph are close in content, their similarity is high, and thus their confidence is also high; conversely, if they are not, their confidence is low. Let \( \mathcal{G} \) be a homologous line graph, and \( \mathcal{N}\left( \mathcal{G}\right) \) be the set of nodes in the graph. For any two nodes \( {v}_{i},{v}_{j} \in \mathcal{N}\left( \mathcal{G}\right) \) with the same attributes, the similarity \( S\left( {{v}_{i},{v}_{j}}\right) \) between them is defined based on the calculation method of mutual information entropy. The mutual information entropy \( I\left( {{v}_{i},{v}_{j}}\right) \) measures the interdependence of the attribute content of the two nodes, and its calculation formula is: \[ I\left( {{v}_{i},{v}_{j}}\right) = \mathop{\sum }\limits_{{x \in {V}_{i}}}\mathop{\sum }\limits_{{y \in {V}_{j}}}p\left( {x, y}\right) \log \left( \frac{p\left( {x, y}\right) }{p\left( x\right) p\left( y\right) }\right) \tag{4} \] where \( {V}_{i} \) and \( {V}_{j} \) are the sets of attribute values for nodes \( {v}_{i} \) and \( {v}_{j} \) , respectively, \( p\left( {x, y}\right) \) is the joint probability distribution of \( {v}_{i} \) taking attribute value \( x \) and \( {v}_{j} \) taking attribute value \( y \) , and \( p\left( x\right) \) and \( p\left( y\right) \) are the marginal probability distributions of \( x \) and \( y \) , respectively. The similarity \( S\left( {{v}_{i},{v}_{j}}\right) \) can be defined as the normalized form of mutual information entropy to ensure that its value lies within the interval \( \left\lbrack {0,1}\right\rbrack \) : \[ S\left( {{v}_{i},{v}_{j}}\right) = \frac{I\left( {{v}_{i},{v}_{j}}\right) }{H\left( {V}_{i}\right) + H\left( {V}_{j}\right) } \tag{5} \] where \( H\left( {V}_{i}\right) \) and \( H\left( {V}_{j}\right) \) are the entropies of the attribute value sets of nodes \( {v}_{i} \) and \( {v}_{j} \) , respectively, calculated as: \[ H\left( V\right) = - \mathop{\sum }\limits_{{x \in V}}p\left( x\right) \log p\left( x\right) \tag{6} \] Subsequently, the confidence \( C\left( \mathcal{G}\right) \) of the homologous line graph \( \mathcal{G} \) can be determined by calculating the average similarity \( S\left( {{v}_{i},{v}_{j}}\right) \) of all node pairs in the graph: \[ C\left( \mathcal{G}\right) = \frac{1}{{\left| \mathcal{N}\left( \mathcal{G}\right) \right| }^{2} - \left| {\mathcal{N}\left( \mathcal{G}\right) }\right| }\mathop{\sum }\limits_{{{v}_{i} \in \mathcal{N}\left( \mathcal{G}\right) }}\mathop{\sum }\limits_{\substack{{{v}_{j} \in \mathcal{N}\left( \mathcal{G}\right) } \\ {j \neq i} }}S\left( {{v}_{i},{v}_{j}}\right) \tag{7} \] where \( \left| {\mathcal{N}\left( \mathcal{G}\right) }\right| \) denotes the number of nodes in the graph. Notably, a homologous line graph exhibiting high confidence demonstrates that its constituent nodes maintain strong attribute-level consistency across their content representations. 2) Node-Level Confidence Computing: In the second phase, the confidence of individual node \( C\left( v\right) \) is calculated, which takes into account the node's consistency, authority, and historical confidence. The following are the detailed calculation methods and formulas. Algorithm 1 Multi-level Confidence Computing Algorithm --- procedure CONFIDENCE_COMPUTING \( \left( {v, D}\right) \) \( {S}_{n}\left( v\right) \leftarrow \) Equation (8) AuthLLM \( \left( v\right) \leftarrow \) Equation (10) Authhist \( \left( v\right) \leftarrow \) Equation (11) \( A\left( v\right) \leftarrow \) Equation (9) \( C\left( v\right) \leftarrow {S}_{n}\left( v\right) + A\left( v\right) \) return \( C\left( v\right) \) end procedure procedure \( \operatorname{MCC}\left( {\mathcal{G}, Q, D}\right) \) \( \mathcal{{SV}}s \leftarrow \varnothing ,\mathcal{{LV}}s \leftarrow \varnothing \) \( {\mathcal{U}}_{\text{ unvisited }} \leftarrow V \) while \( {\mathcal{U}}_{\text{ unvisited }} \neq \varnothing \) do \( v \leftarrow \) pop a node from \( {\mathcal{U}}_{\text{ unvisited }} \) for all \( D \in D \) do if \( v \in \operatorname{Data}\left( {Q,{subS}{G}_{i}}\right) \) then \( C\left( v\right) \leftarrow \) Confidence_Computing \( \left( {v, D}\right) \) if \( C\left( v\right) > \theta \) then \( {\mathcal{U}}_{sg} \leftarrow {\mathcal{U}}_{sg} \cup \{ v\} \) \( {\mathcal{E}}_{sg} \leftarrow {\mathcal{E}}_{sg} \cup \left\{ {e}_{i}\right\} \) else \( \mathcal{{LV}}s \leftarrow \mathcal{{LV}}s \cup \{ v\} \) end if end if end for if \( {\mathcal{U}}_{sg} \neq \varnothing \) then \( \mathcal{{SV}}s \leftarrow \mathcal{{SV}}s \cup \left( {{\mathcal{U}}_{sg},{\mathcal{E}}_{sg}}\right) \) \( {\mathcal{U}}_{sg} \leftarrow \varnothing ,{\mathcal{E}}_{sg} \leftarrow \varnothing \) end if end while return \( \mathcal{{SV}}s,\mathcal{{LV}}s \) end procedure --- a) Node Consistency Score: The node consistency score \( S\left( v\right) \) reflects the consistency of the node across different data sources. We use mutual information entropy to calculate the similarity between node pairs, thereby assessing consistency. For a node \( v \) , its consistency score can be expressed as: \[ {S}_{n}\left( v\right) = \frac{1}{\left| N\left( v\right) \right| }\mathop{\sum }\limits_{{u \in N\left( v\right) }}S\left( {v, u}\right) \tag{8} \] where \( N\left( v\right) \) is the set of nodes with the same attributes as node \( v \) , and \( S\left( {v, u}\right) \) is the similarity between nodes \( v \) and \( u \) as defined in Equation 5 b) Node Authority Score: Authority score is divided into two parts: the node's authority assessed by the LLM and the node's historical authority. This score reflects the importance and authenticity of the node. Additionally, we use an expert LLM to comprehensively evaluate the authority of the node. The node’s authority score \( A\left( v\right) \) can be calculated using the following formula: \[ A\left( v\right) = \alpha \cdot {\text{ Auth }}_{LLM}\left( v\right) + \left( {1 - \alpha }\right) \cdot {\text{ Auth }}_{\text{ hist }}\left( v\right) \tag{9} \] where \( \alpha \) is a weight coefficient that balances the contributions of LLM-assessed authority and historical authority, satisfying \( 0 \leq \alpha \leq 1 \) . Algorithm 2 Multi-source Knowledge Line Graph Prompting --- procedure MKLGP \( \left( q\right) \) \( {E}_{q},{R}_{q} \leftarrow \) Logic Form Generation \( \left( q\right) \) \( {D}_{q} \leftarrow \) Multi Document Extraction \( \left( {V}_{q}\right) \) \( {\mathcal{{SG}}}^{\prime } \leftarrow \operatorname{Prompt}\left( {D}_{q}\right) \) \( \mathcal{{SV}}s,\mathcal{{LV}}s \leftarrow \operatorname{MCC}\left( {\mathcal{S}{\mathcal{G}}^{\prime }, q,{D}_{q}}\right) \) \( {C}_{\text{ nodes }},{\mathcal{G}}_{A} \leftarrow \operatorname{Prompt}\left( {\mathcal{S}\mathcal{V}s,\mathcal{L}\mathcal{V}s}\right) \) Answer \( \leftarrow \) Generating Trustworthy Answers \( \left( {{C}_{\text{ nodes }},{\mathcal{G}}_{A}}\right) \) return Answer end procedure --- Benefiting from the calculation idea of knowledge credibility in the PTCA [33], \( {\operatorname{Auth}}_{\mathrm{{LLM}}}\left( v\right) \) is assessed by the global influence and local connection strength of the node. The LLM can comprehensively calculate the credibility of knowledge by integrating the association strength between entities, entity type information, and multi-step path information. \[ {\operatorname{Autm}}_{LLM}\left( v\right) = \frac{1}{1 + {e}^{-\beta \cdot {C}_{LLM}\left( v\right) }} \tag{10} \] where \( {C}_{\mathrm{{LLM}}}\left( v\right) \) is the authority score provided by the LLM for node \( v \) is the average value of all nodes’ \( {C}_{\mathrm{{LLM}}}\left( v\right) \) , and \( \beta \) is a parameter that controls the steepness of the scoring curve. c) Historical Authority: \( {\operatorname{Auth}}_{\text{ hist }}\left( v\right) \) is an authority score based on the node's historical data. Inspired by Zhu's work [34], we expect to use the credibility of historical data sources and current query-related data for incremental estimation. \[ {\text{ Auth }}_{\text{ hist }}\left( v\right) = \frac{\mathcal{H} \cdot P{r}^{h}\left( D\right) + \mathop{\sum }\limits_{{{v}_{p} \in {D}_{v}\left\lbrack q\right\rbrack }}\Pr \left( {v}_{p}\right) }{\mathcal{H} + \left| {\operatorname{Data}\left( {q,\text{ subS }{\mathcal{G}}^{\prime }{}_{i}}\right) }\right| } \tag{11} \] where \( \mathcal{H} \) is the number of entities provided by data source \( D \) for all historical queries, \( \mathop{\Pr }\limits^{h}\left( D\right) \) is the historical credibility of data source \( D,{D}_{v}\left\lbrack q\right\rbrack \) is the set of correct answers, and Data \( \left( {q,{subS}{\mathcal{G}}^{\prime }{}_{i}}\right) \) is the query-related data obtained from the multi-source line subgraph. Ultimately, we designed the multi-level confidence computing algorithm, MCC, to calculate the credibility of the data sources in the homologous subgraph, ensuring the quality of the knowledge graph embedded in the LLM. The algorithm is shown in Algorithm 1 It should be noted that the MCC algorithm does not directly provide the final graph confidence and node confidence; these values must be obtained through prompt to achieve the ultimate results. ## E. Multi-source knowledge line graph prompting We propose the Multi-source Knowledge Line Graph Prompting (MKLGP) algorithm for multi-source data retrieval. Given a user query \( q \) , LLM is firstly employed to extract the intent, entities, and relationships from \( q \) , and generates the corresponding logical relationships. The dataset then undergoes multi-document filtering to derive text chunks, followed by constructing a Multi-source Line Graph (MLG) for knowledge aggregation. Further, it matches homogeneous subgraphs and utilizes the MCC algorithm to obtain a set of credible query nodes and isolated points \( \mathcal{{SV}}s,\mathcal{{LV}}s \) . Finally, by leveraging the prompt, the graph confidence is obtained, and the node confidence is calculated to enhance the credibility of the answer. The results are then embedded into the context of the LLM to generate a credible retrieval answer. ## IV. EXPERIMENTS This section will conduct experiments and performance analysis on the construction of homologous line graphs and the multi-level confidence calculation modules. Baseline methods will be compared with other SOTA multi-document retrieval QA methods, data fusion methods, and KBQA methods. Extensive experiments will be conducted to assess the robustness and efficiency of MultiRAG, which aims to answer the following questions. - Q1: How does the retrieval recall performance of Multi-RAG compare with other data fusion models and SOTA data retrieval models? - Q2: What are the respective impacts of data sparsity and data inconsistency on the quality of retrieval recall? - Q3: How effective are the two modules of MultiRAG individually? - Q4: How is the performance of MultiRAG in multi-hop Q&A datasets after incorporating multi-level confidence calculation? - Q5: What are the time costs of the various modules in MultiRAG? ## A. Experimental Settings a) Datasets: To validate the efficiency of multi-source line graph construction and its enhancement of retrieval performance, the article conducts multi-source data fusion experiments on four real-world benchmark datasets [35]-[37], as is shown in Table 1 (1) The movie dataset comprises movie data collected from 13 sources. (2) The book dataset includes book data from 10 sources. (3) The flight dataset gathers information on over 1200 flights from 20 sources. (4) The stock dataset collects transaction data for 1000 stock symbols from 20 sources. In the experiments, we issue 100 queries for each of the four datasets to verify their retrieval efficiency. It is noteworthy that the Movies dataset and the Flights dataset are relatively dense, while the Books dataset and the Stocks dataset are relatively sparse, which can impact the model's performance. Additionally, to validate the robustness of the MultiRAG on complex Q&A datasets, we selected two multi-hop question answering datasets, HotpotQA [38] and 2WikiMultiHopQA [39]. Both datasets are constructed based on Wikipedia documents, allowing us to utilize a consistent document corpus and retriever to provide external references for LLMs. Considering the constraints of experimental costs, we conducted a subsample analysis on 300 questions from the validation sets of each experimental dataset. TABLE I: Statistics of the datasets preprocessed
DatasetsData sourceSourcesEntitiesRelationsQueries
MoviesJSON(J)41970145790100
KG(K)5100229264709
CSV(C)470276184657
BooksJSON(J)333922824100
CSV(C)325471812
XML(X)420541509
FlightsCSV(C)1048672100835100
JSON(J)104193989339
StocksCSV(C)10779911169100
JSON(J)10775910619
b) Evaluation Metrics: To assess effectiveness, we adopt the F1 score as the evaluation metric for the data fusion results, following previous experimental metrics [37], [40]-[42]. The F1 score is the harmonic mean of precision (P) and recall (R), calculated as follows: \[ {F1} = 2 \times \frac{P \times R}{P + R} \tag{12} \] Furthermore, to evaluate the retrieval credibility of MKLGP Algorithm, we utilize the recall metric, specifically Recall@K, to assess performance at three distinct stages: before subgraph filtering, before node filtering, and after node filtering. In addition, we employ the query response time \( T \) (measured in seconds) as an evaluative metric to verify the efficiency of knowledge aggregation. c) Hyper-parameter Settings: For all baselines, we carefully adjusted the parameters according to the characteristics of MultiRAG. All methods were implemented in a Python 3.10 and CUDA 11.6 environment. Except for the experiments using GPT-3.5-Turbo for CoT, the rest of the work utilized Llama3-8B-Instruct as the base model. For each different data format, after slicing into Chunks, we stored the slice numbers, data source locations, and transformed triple nodes in the multi-source line graph using JSON-LD format, thereby enabling simple cross-indexing. For hyperparameter settings, the temperature parameter \( \beta \) was set to 0.5 . The number of entities in historical queries was initialized to 50, the initial node confidence threshold was defined as 0.7 , and the graph confidence threshold was set to 0.5. All experiments were conducted on a device equipped with an Intel(R) Core(TM) Ultra 9 185H 2.30GHz and 512GB of memory. d) Baseline Models: To demonstrate the superiority of the MultiRAG method, we compare it with basic data fusion methods and SOTA methods, including the multi-document question-answering methods and knowledge base question-answering methods. Thanks to Zhu's work \( {}^{3} \) [34], we compare with the following baseline methods: --- \( {}^{3} \) https://github.com/JunHao-Zhu/FusionQuery --- TABLE II: Comparison with baseline methods and SOTA methods for multi-source knowledge fusion
DatasetsData sourceData Fusion Methods (Baseline)SOTA MethodsOur Method
TFLTMIR-CoTMDQAChatKBQAFusionQueryMCC
F1/%Time/sF1/%Time/sF1/%Time/sF1/%Time/sF1/%Time/sF1/%Time/sF1/%Time/s
MoviesJ/K37.1971741.4199543.2156746.2158845.1380953.2122.452.698.3
J/C41.9721442.9188445.0139944.5136042.7324652.7183.154.375.1
K/C37.8219941.2157637.6101445.298740.4202742.5141.049.186.0
J/K/C36.61122540.8234641.5255149.8226444.7515153.6137.854.8157
BooksJ/C40.2101742.4195.3s35.2147.655.7124.256.1165.058.522.763.513.66
J/X35.5107035.6277.736.1178.755.1115.654.7200.157.920.663.113.78
C/X43.0103344.1232.642.6184.557.2115.655.6201.460.321.564.213.54
J/C/X37.3230441.0413.240.4342.656.4222.657.1394.159.147.066.827.4
FlightsC/J27.3604979.11478658.3214.076.536076.837674.220.274.980
StocksC/J68.42.3019.2133764.853.365.278.464.088.968.00.3378.612.1
* The F1 score is for Q1 and time is for Q5. * Bold represents the optimal metrics, while underlined text indicates the sub-optimal metrics. The same applies to the following text. 1) TruthFinder(TF) [37]: the classic iterative data fusion method. 2) LTM [42]: the probabilistic data fusion method. 3) CoT [43] is a foundational approach that involves step-by-step reasoning to reach a conclusion, we use GPT- 3.5-Turbo as the base model. 4) Standard RAG [2] is a method that combines the strengths of retrieval and generation models to answer questions. Moreover, we also summerize these SOTA methods below: - IRCoT [44] is an advanced method that refines the reasoning process through iterative retrieval. - ChatKBQA [45] is a conversational interface-based method for knowledge base question answering. - MDQA [46] is a method designed to extract answers from multiple documents effectively. - FusionQuery [34] is a SOTA method based on the efficient on-demand fusion query framework. - RQ-RAG [47] is a method that integrates external documents and optimizes the query process to handle complex queries. - MetaRAG [9] is a method that employs metacognitive strategies to enhance the retrieval process. e) Dataset Preprocessing: To better align the datasets with real-world application scenarios and to demonstrate the applicability of the proposed method to multi-source data, we have split and reconstructed the four datasets into three categories of data formats: tabular data (structured data), nested JSON data (semi-structured data), and XML data (semi-structured data), stored respectively in csv, json, and xml file formats. We also retained some data directly stored in KG format. Table 1 displays the detailed statistics after the dataset division. ## B. Evaluation of Multi-source Knowledge Aggregation (MKA) Q1: How does the retrieval recall performance of Mul-tiRAG compare with other data fusion models and SOTA ## data retrieval models? To validate the effectiveness of the multi-source knowledge aggregation module (MKA) in MultiRAG, we assess it using F1 scores and query times across four multi-source query datasets. By substituting the fusion query algorithm with different baseline models and SOTA models, multiple sets of experimental results are botained to evaluate its performance in multi-domain querying. Table II summarizes the data querying performance of MKLGP and baselines on the four datasets; Q1 focuses solely on the F1 scores of the methods, which includes four data fusion methods and three SOTA methods that support data fusion. Table II demonstrates that the MCC module outperforms all comparative models across four datasets. Experimental results indicate that it achieves an F1 score that is more than 10% higher than the best baseline data fusion model and obtains superior performance compared to other baselines. The MV method performs poorly on all datasets because it can only return a single answer for a query, which fails to accommodate the common scenario where a query has multiple return values. For instance, a movie or a book typically has multiple directors or authors. However, the majority of methods show significantly better performance on the Movies and Flights datasets than on the Books and Stocks datasets. This is because the Movies and Flights datasets are inherently denser, and previous SOTA models can match or outperform our approach in situations where knowledge is abundant, which is acceptable. In contrast, on the more sparse Books and Stocks datasets, our method achieves an average improvement of more than 10% over SOTA methods. ## Q2: What are the respective impacts of data sparsity and data inconsistency on the quality of retrieval recall? MultiRAG demonstrates good robustness in scenarios of varying data sparsity and inconsistency. To validate it, we conducted experiments from the following two perspectives. 1) Sparsity of multi-source data: We applied \( {30}\% ,{50}\% \) , and \( {70}\% \) random relationship masking to four pre-processed datasets, making the connections between data sparser while ensuring that the query answers are still retrievable. 2) Consistency of multi-source data: We added \( {30}\% ,{50}\% \) , and \( {70}\% \) of triple increments (the new triples are copies of the original triples) to the four pre-processed datasets, and completely shuffled the relationship edges of the added triples to disrupt the consistency of multi-source data. Subsequently, we employed MultiRAG to experiment with datasets under both perturbation schemes. TABLE III: Ablation experiments of multi-source knowledge aggregation(MKA) and multi-level confidence computing(MCC)
DatasetsSourceMultiRAGw/o MKAw/o Graph Levelw/o Node Levelw/o MCC
F1/%QT/sPT/sF1/%QT/sPT/sF1/%QT/sPT/sF1/%QT/sPT/sF1/%QT/sPT/s
MoviesJ/K52.625.762.6448.2278362.6445.350.158.238.721.30.3131.625.70.28
J/C54.312.761.3649.1188261.3646.828.957.440.210.50.2930.512.70.29
K/C49.131.664.4045.5423364.4042.765.361.835.928.4-0.2733.131.6-0.29
J/K/C54.839.260.847.5443760.848.175.656.241.535.80.3034.739.20.32
BooksJ/C63.51.192.4757.111.92.4755.24.72.1249.80.920.1843.41.190.22
J/X63.11.222.5659.311.72.6254.75.12.2448.30.890.1942.61.220.22
C/X64.21.162.3855.38.392.3853.93.92.0547.10.850.1641.01.160.17
J/C/X66.81.313.0757.215.83.0859.46.32.8952.71.120.2136.41.310.20
FlightsC/J74.929.8109.972.2NAN109.968.3142.798.561.425.30.8552.129.81.07
StocksC/J78.62.725.3669.6450.85.3672.18.94.1265.31.980.1545.42.720.17
TABLE IV: Performance comparison on HotpotQA and 2WikiMultiHopQA datasets
MethodHotpotQA2WikiMultiHopQA
PrecisionRecall@5PrecisionRecall@5
Standard RAG34.133.525.626.2
GPT-3.5-Turbo+CoT33.947.235.045.1
IRCoT41.641.242.340.9
ChatKBQA47.842.146.543.7
MDQA48.652.544.145.8
RQ-RAG51.649.345.344.6
MetaRAG51.149.950.752.2
MultiRAG59.362.755.761.2
Firstly, to address data sparsity, we conducted experiments on MultiRAG (Ours) and ChatKBQA (SOTA). The experimental results demonstrate that MultiRAG exhibits significant robustness when faced with the challenge of data sparsity. Specifically, after applying 30%, 50%, and 70% relationship masking, the F1 score of MultiRAG on the Books dataset only dropped from 66.8% to 60.0%. On the Stocks dataset, its F1 score decreased from 78.6% to 71.0%, which have been shown in Fig 5b and Fig 5d. These moderate decreases indicate that MultiRAG can effectively maintain its performance even when a substantial number of relationships are masked. In contrast, ChatKBQA's performance decline under the same conditions is more significant. On the Books dataset, ChatKBQA's F1 score dropped from 59.1% to 53.0%, and on the Stocks dataset, its F1 score decreased from 68.0% to 62.0%. This outcome reveals the challenges ChatKBQA faces when dealing with sparse data, especially when a large number of data connections are masked, significantly impacting its performance. Next, we conducted robustness experiments on multi-source data consistency. We perturbed the Books and Stocks datasets to varying degrees to test the performance changes of Mul-tiRAG and ChatKBQA when data consistency is disrupted. The experimental results show that MultiRAG demonstrates excellent robustness in the face of data consistency disruption, while ChatKBQA's performance declines rapidly under perturbation. Specifically, as is shown in Fig. 5a, on the Movies dataset, we added 30%, 50%, and 70% triple increments to the original dataset and randomized the relationship edges of the added triples. The results show that MultiRAG's F1 score slightly decreased from 54.8% to 52.1%, 51.5%, and 49.9%, while ChatKBQA's F1 score significantly dropped from 53.6% to 51.6%, 47.2%, and 40.8%. On the Flights dataset shown in Fig. 5c, we performed the same perturbation operations, and MultiRAG's F1 score slightly decreased from 74.9% to 73.4%, 72.9%, and 71.4%, while ChatKBQA's F1 score substantially dropped from 74.2% to 69.7%, 64.3%, and 55.8%. These results indicate that even when data consistency is severely compromised, MultiRAG can still maintain a high level of performance stability, whereas ChatKBQA's performance is more sensitive to disruptions in data consistency. ## C. Evaluation of Multi-level Confidence Computing Calculating the confidence of subgraphs and nodes to filter trustworthy answers is of significant demand in critical domains such as finance and law. Considering the high temporal and spatial overhead of directly calculating the confidence of all nodes, we draw inspiration from the workflow of recommendation systems, mimicking the process of coarse and fine ranking, and adopt the multi-level confidence computing method to filter credible nodes and enhance retrieval performance. Calculating the credibility of homologous subgraphs allows us to preliminarily determine whether the subgraphs containing answers can generate highly credible answers. For subgraphs with low confidence, more nodes need to be extracted to ensure the robustness of the overall retrieval; for subgraphs with high confidence, only 1-2 nodes are required to generate the correct answer. ![bo_d6oh96c601uc73e30m7g_9_135_136_1525_284_0.jpg](images/bo_d6oh96c601uc73e30m7g_9_135_136_1525_284_0.jpg) Fig. 5: Experimental results of Q2, where (a) and (b) display the multi-source data sparsity experiments, and (c) and (d) display the multi-source data consistency experiments. ![bo_d6oh96c601uc73e30m7g_9_138_530_1525_421_0.jpg](images/bo_d6oh96c601uc73e30m7g_9_138_530_1525_421_0.jpg) Fig. 6: F1 score and Query Time of Movies and Books with corruption level \( 0\% ,{10}\% ,{30}\% ,{50}\% ,{70}\% \) in different sources ## Q3: How effective are the two modules of MultiRAG individually? a) Ablation Study on Component Effectiveness: The MKA module achieves significant efficiency-accuracy synergy through its MLG architecture. As shown in Table III, MLG construction introduces modest preprocessing time (12.7s- 39.2s) while delivering 10-100× query acceleration. Specifically, the flight dataset shows QT reduction from computational infeasibility (marked NAN) to 29.8s through MLG's compact structure. Concurrently, MKA sustains consistent accuracy improvements. Removing MKA causes F1 drops of 7.3% on Movies and 9.6% on Books, demonstrating MLG's effectiveness in connecting fragmented knowledge across sources. The MCC module exhibits more significant effects on performance and hallucination control. Disabling MCC causes drastic F1 degradation of 20.1% on Movies and 33.2% on Stocks, with PT values indicating increased hallucination risks. This validates MCC's critical role in eliminating unreliable information through hierarchical confidence computation. b) Hierarchical Analysis of MCC: Stratified ablation reveals the complementary roles of graph-level and node-level computations. For Movies (J/K/C configuration), removing graph-level filtering reduces F1 to 48.1% (+13.4% vs MCC-disabled) with QT increasing to 75.6s (+93% vs full framework). Conversely, disabling node-level computation yields 41.5% F1 (+6.8% vs baseline), showing graph-level filtering alone cannot resolve local conflicts. The complete MCC framework achieves 54.8% F1 by synergistically combining both layers. Error analysis shows distinct failure patterns: 38.7% errors under graph-level removal (Movies J/K) stem from cross-source inconsistencies, while 52.7% failures with node-level removal (Books J/C/X) originate from local authority issues. This confirms the functional specialization-graph-level ensures global consistency, node-level verifies local credibility. Fig 7 demonstrates that an optimal balance between efficiency and accuracy is achieved at \( \alpha = {0.5} \) , where the hybrid weighting of LLM-assessed authority and historical authority peaks with an F1 score of 67.7% and balanced query time. Specifically, increasing \( \alpha \) towards 1.0, which emphasizes the LLM, reduces query time from 83.2 seconds \( \left( {\alpha = {0.0}}\right) \) to 51.8 seconds \( \left( {\alpha = {1.0}}\right) \) by minimizing historical data validation. Conversely, the F1 score follows a non-monotonic pattern, reaching its maximum at \( \alpha = {0.5} \) before declining as reliance on either the LLM or historical data becomes excessive. This equilibrium leverages the LLM's contextual adaptability (AuthLLM) while maintaining the stability of expert systems (Auth \( {}_{\text{ hist }} \) ), as evidenced by a 62.4% reduction in errors during ablation studies when both components are utilized. By avoiding complete dependence on the LLM \( \left( {\alpha \neq {1.0}}\right) \) and integrating probabilistic LLM inferences with deterministic historical patterns through multi-level confidence computing (Eq.9), the methodology enhances robustness against data sparsity and noise, particularly in the Books and Stocks datasets. Query: "What is the real-time status of Air China flight CA981 from Beijing Capital International Airport (PEK) to New York John F. Kennedy Airport (JFK)?"
Data Sources Structured Semi-structured UnstructuredCA981, PEK, JFK, Delayed, 2024-10-01 14:30
\{"flight": "CA981", "delay_reason": "Weather", "source": "AirChina"\}
"Typhoon Haikui impacts PEK departures after 14:00."
MKA ModuleStructured parsing: Flight attributes mapping
LLM extraction: (CA981, DelayReason, Typhoon) @0.87
MLG Subgraph ![bo_d6oh96c601uc73e30m7g_10_452_427_964_319_0.jpg](images/bo_d6oh96c601uc73e30m7g_10_452_427_964_319_0.jpg)
MCC ModuleWith GCC: Graph confidence=0.71 (Threshold=0.5), Filtered: ForumUser123 (0.47) Without GCC: Unfiltered conflict=2 subgraphs
LLM ContextTrusted: CA981.Status=Delayed (0.89), DelayReason=Typhoon (0.85)
Conflicts: ForumUser123:On-time (0.47), WeatherAPI:Clear (0.52)
Final AnswerCorrect: "CA981 delayed until after 14:30 due to typhoon"
Illucinated: "CA981 on-time with possible delay after 14:30 "
![bo_d6oh96c601uc73e30m7g_10_172_1115_676_387_0.jpg](images/bo_d6oh96c601uc73e30m7g_10_172_1115_676_387_0.jpg) Fig. 7: Influence of hyperparameter \( \alpha \) on multi-source retrieval ## Q4: How is the performance of MultiRAG in multi-hop Q&A datasets after incorporating multi-level confidence calculation? To assess the validity of the multi-level confidence computing method in reducing hallucinations generated by large models and enhancing the credibility of Q&A systems, we compare the Recall@5 scores of different methods on the HotpotQA and 2WikiMultiHopQA datasets. The outcome of Table IV indicates that the multi-level confidence computing method not only demonstrates a higher average Recall@5 score but also maintains a lower standard deviation compared to traditional methods. This suggests that the multi-level confidence computing method is more consistent in its performance across different queries, leading to fewer hallucinations and more reliable Q&A responses. The lower standard deviation is a testament to the robustness of the mechanism in handling the variability in data and the complexity of the queries. Furthermore, we performed a detailed error analysis to identify the types and frequency of hallucinations in the responses generated by the different methods. The results showed that the multi-level confidence computing method significantly reduced the frequency of hallucinations, particularly in the cases where the context was ambiguous or the information was not readily available in the knowledge base. ## Q5: What are the time costs of the two modules in MultiRAG? Intuitively, MLG aggregates homologous data from several sources, ensuring the density of the retrieval subgraphs without the need to traverse and store an excessive number of invalid nodes, thereby significantly reducing the time cost associated with traversing and querying in traditional knowledge graphs. Furthermore, although the SOTA methods are not specifically tailored for low-resource, high-noise data scenarios, they still exhibit considerable robustness and retrieval performance in such environments. Both the MDQA and ChatKBQA models employ LLM-based data retrieval approaches, with the primary temporal and spatial overheads focusing on token consumption and LLM-based searching. In contrast, MultiRAG concentrates its overhead on the construction of the MLG. While in the original context of the MLG, construction times are often within seconds and highly efficient, the introduction of an LLM still incurs additional temporal costs due to text generation, which remains acceptable. Ultimately, these methods all demonstrate satisfactory retrieval performance; however, due to the inherent noise in the datasets, improvements in the accuracy of question-answering are somewhat limited. ## D. Case Study MultiRAG's effectiveness in multi-source integration is demonstrated through a real-world flight status query for "CA981 from Beijing to New York". As detailed in Table V case study exemplifies MultiRAG's unique strength in transforming fragmented, conflicting inputs into trustworthy answers through systematic source weighting and consensus modeling. Firstly, MultiRAG integrated three data formats: structured departure schedules, semi-structured delay codes from airline systems, and unstructured weather alerts. The MKA module extracted key relationships (flight-delay-typhoon) with a confidence score of 0.87. Subsequently, the MCC module resolved conflicts through hierarchical verification by filtering out low-reliability sources, such as user forums (confidence score of 0.47), while prioritizing data from airlines (confidence score of 0.89 ) and weather reports. This dual-layer validation-combining automated threshold checks (graph confidence of 0.71) with LLM-simulated expert reasoning-enabled the precise reconciliation of contradictory departure time claims. Ultimately, the system generated the verified conclusion, "Delayed until after 14:30 due to typhoon," while suppressing the inconsistent "on-time" report. ## E. Restrictive Analysis Lastly but not least, we acknowledge several limitations inherent in our current framework. 1) Lack of optimization of text chunk segmentation. 2) Reliance on LLM-based expert evaluation, which may introduce potential security vulnerabilities. 3) Focuses on eliminating factual hallucinations but lacks handling of symbolic hallucinations. ## V. RELATED WORK ## A. Graph-Structured Approaches for Hallucination Mitigation Recent advancements have demonstrated unique advantages of graph structures in mitigating hallucinations within RAG systems. MetaRAG [9] establishes knowledge association verification through meta-cognitive graph reasoning paths, enhancing self-correction mechanisms in multi-hop QA. Graph-CoT [48] innovatively leverages Graph Neural Networks to establish bidirectional connections between KGs and the latent space of LLMs. In result, it reduces factual inconsistencies by 37% on KGQA benchmarks. Inspired by neurobiology, HippoRAG [23] constructs offline memory graphs with a neural indexing mechanism, decreasing retrieval latency to one-eighth of traditional methods. While ToG 2.0 [25] further advances this field by introducing a graph-context co-retrieval framework that dynamically balances structured and unstructured evidence, resulting in a 29% reduction in hallucination rates compared to unimodal approaches. Unlike prior approaches that primarily focus on unimodal confidence calculations, MultiRAG achieves superior hallucination mitigation through the adaptive filtering of conflicting subgraphs (GCC module) while maintaining multi-domain logical associations via its novel knowledge aggregation mechanism (MKA module). ## B. Heterogeneous Graph Fusion for RAG The fusion of multi-source heterogeneous data relies on advanced graph representation techniques. FusionQuery [34] enhances cross-domain retrieval precision by integrating heterogeneous graphs and computing dynamic credibility evaluations. The Triple Line Graph [31] addresses the challenge of knowledge fragmentation by systematically aggregating cross-domain relationships, leading to Multi-source Line Graph proposed in this paper. Additionally, leveraging the structured representation advantages of KAG [26] in knowledge-guided retrieval, we achieve a unified representation approach for multi-source KGs, underscoring the importance of heterogeneous graph fusion in real-world applications. ## C. Hallucination Benchmark and Confidence-Aware Comput- ing The evaluation of hallucinations in LLMs and associated confidence calculation methods are crucial for mitigating hallucinations. HaluEval [49] offers 5,000 annotated samples across five error categories, but lacks granularity for relational hallucinations. RefChecker [50] implements triple decomposition for fine-grained detection, improving precision by 26.1% over sentence-level methods. RAGTruth [51] contains nearly 18,000 RAG-generated responses with detailed manual annotations including word-level hallucination intensities. However, diverse and complex data sources continue to challenge existing evaluation frameworks. ## VI. CONCLUSION In this work, we introduce MultiRAG, a framework designed to mitigate hallucination in multi-source knowledge-augmented generation. To address hallucinations arising from data sparsity and inconsistency, we propose two key innovations: multi-source knowledge aggregation and multi-level confidence calculation. The introduction of multi-source line graphs enables efficient cross-domain data aggregation, enhancing knowledge connectivity and retrieval performance. Meanwhile, our multi-level confidence computing module adaptively filter out low-quality subgraphs and unreliable nodes. Future work will explore more challenging aspects of hallucination mitigation, particularly in multimodal retrieval and ultra-long text reasoning, to better adapt generative retrieval systems to real-world, open multi-source environments.