Designing and Developing an Optimized RAG System with a Comparative Analysis of Vector Stores
Siham Argaw
Department of Computer Science
Faculty Supervisor: Shahrukh Humayoun
The rapid advancement of Large Language Models (LLMs) has led to the development of Retrieval-Augmented Generation (RAG) systems, which enhance LLM performance by integrating external knowledge sources. Vector databases, such as Pinecone and TileDB, play a critical role in RAG systems by enabling efficient storage and retrieval of high-dimensional vectors. However, the lack of systematic comparisons between vector stores makes it challenging for developers to choose the optimal backend for RAG applications. This research addresses this gap by conducting a comparative analysis of Pinecone and TileDB, focusing on their performance in RAG workflows. The research employs a systematic experimental approach to evaluate Pinecone and TileDB with three experiments: 1) Performance across different vector dimensions (384, 768, 1024, 1536) using embedding models like Sentence Transformers and OpenAI’s text-embedding-ada-002. 2) Scalability analysis by testing datasets of 50, 200, 500, 1000, and 2000 pages. 3) Retrieval performance across multiple question formulations to simulate real-world RAG scenarios. These experiments were implemented using Python and LangChain, with results visualized for comparison.