Amaral Cejas O., Guo Y., Tang Q.
IEEE Access, vol. 13, pp. 212773-212781, 2025
Recently, Retrieval Augmented Generation (RAG) has emerged as a powerful paradigm for enhancing the capability of Large Language Models (LLMs) by incorporating external, up-to-date, or domain-specific information through retrieval mechanisms. In their implementation, RAG systems usually rely on vector embedding to identify relevant context from an underlying knowledge base, and have gained widespread adoption across tasks such as question answering and summarization due to their superior performance. Surprisingly, the interplay between embedding quality and RAG performance has not received much attention so far, even though it is commonly perceived as the key factor for influencing retrieval accuracy and the trustworthiness of the generated answer. To fill in this gap, in this paper, we systematically investigate the impact of embedding quality on the overall performance of RAG-based applications. Through a comprehensive performance evaluation of 12 embeddings over two datasets (i.e., a public dataset SQuAD and a private Telecom dataset), we reveal quite different retrieval accuracy levels for these embeddings, while following the same trend on both datasets. For the public dataset SQuAD, there is a strong positive correlation between retrieval accuracy and RAG performance, while this relationship is not as strong in the private Telecom dataset. Our findings highlight that embedding quality is a fundamental yet tricky determinant of RAG systems’ ability to generate trustworthy responses and it should be examined deeply on a case-by-case basis. We open source our full implementation at: https://github.com/Yuejun-GUO/RAG-exp
