Abstract

RAG-Bot is an open-source project that implements a Retrieval-Augmented Generation (RAG) system with an interactive terminal interface. This project addresses the common challenge of LLM hallucinations by grounding responses in a knowledge base of documents. By combining the power of vector databases with modern language models, RAG-Bot enables accurate, contextual responses to natural language queries across multiple LLM providers including OpenAI, Ollama, Google, and Groq. This blog post details the implementation, architecture, and performance of the RAG-Bot system, demonstrating its effectiveness as a flexible and powerful tool for document-based question answering.

Methodology

System Architecture

RAG-Bot follows a modular architecture with several key components:

1. Document Processing Pipeline: Converts various document formats (starting with JSON) into markdown files that can be chunked and embedded.
2. Vector Database Integration: Uses ChromaDB as the vector store for document embeddings, enabling semantic search capabilities.
3. Embedding Generation: Implements sentence transformers via Huggingface embeddings(specifically all-MiniLM-L6-v2) to create high-quality document embeddings.
4. LLM Provider Interface: Provides a unified interface to multiple LLM providers (OpenAI, Ollama, Google, Groq) with configurable parameters.
5. RAG Query Engine: Combines document retrieval with LLM generation to produce contextually relevant answers.
6. Interactive Terminal: Offers a user-friendly command-line interface for system interaction.

Implementation Details

The implementation uses Python with several key libraries:

• ChromaDB: For vector storage and retrieval
• LangChain: For RAG pipeline components
• Sentence Transformers(huggingface): For document embedding generation
• Various LLM APIs: For text generation capabilities
• ReadyTensor: For some code reference

The document ingestion process follows these steps:

1. Convert JSON documents to markdown format (optional) provided the markdown files included in the data directory
2. Split documents into chunks of appropriate size
3. Generate embeddings for each chunk
4. Store documents and embeddings in ChromaDB

The query process involves:

1. Embedding the user query
2. Retrieving relevant document chunks from ChromaDB
3. Constructing a prompt with retrieved context
4. Sending the prompt to the selected LLM
5. Returning the generated response to the user

Challenges and Solutions

Several technical challenges were addressed during development:

1. ChromaDB File Access Issues: Implemented a robust database manager with proper shutdown procedures and process management to prevent file access conflicts.
2. LLM Provider Abstraction: Designed a flexible provider interface that allows easy switching between different LLM backends without changing the core application logic.

Results

User Experience

The interactive terminal interface provides several advantages:

1. Ease of Use: Simple commands for configuration and querying
2. Flexibility: On-the-fly switching between different LLM providers
3. Transparency: Clear logging of each step in the RAG process

Future Work

Several enhancements are planned for future versions:

1. Web Interface: Developing a browser-based UI for broader accessibility
2. Additional Document Formats: Supporting PDF, HTML, and other formats directly
3. Improved Context Handling: Implementing hierarchical retrieval for better handling of complex queries
4. Memory Features: Adding conversation history to maintain context across multiple queries
5. Evaluation Framework: Building automated testing to measure RAG performance across different domains

Conclusion

RAG-Bot demonstrates the power of combining vector databases with language models to create an effective document querying system. The project provides a solid foundation for building domain-specific assistants that can accurately answer questions based on a corpus of documents. By supporting multiple LLM providers and focusing on modularity, RAG-Bot offers flexibility for various use cases and deployment scenarios.

The open-source nature of this project invites collaboration and extension, potentially leading to more sophisticated RAG implementations in the future. As language models continue to evolve, the RAG approach will remain valuable for grounding responses in factual information and reducing hallucinations.