Abstract

This project presents a production-ready Retrieval-Augmented Generation (RAG) system that addresses fundamental limitations of large language models by grounding responses in custom document collections. By combining vector-based semantic search with multiple state-of-the-art language models, the system delivers accurate, explainable, and conversational answers to user queries. The implementation demonstrates practical solutions for document ingestion, semantic chunking, multi-provider LLM integration, and conversation memory, establishing a foundation for enterprise knowledge management and agentic AI applications.

Introduction

Retrieval-Augmented Generation (RAG) enhances AI assistant capabilities by combining neural language models with context sourced from external knowledge bases. Our project addresses the limitations of “pure” LLMs by enabling responses grounded in uploaded documents (TXT, PDF, etc.), facilitating explainable answers and better multi-turn conversations. The project leverages modular design so it can be extended with memory, tool use, or reasoning workflows.

1. Problem Statement: The Knowledge Cutoff and Context Limitation

The Fundamental Challenge

Large language models (LLMs) like GPT-4, Llama, and Gemini possess impressive general knowledge but face critical limitations in real-world applications:

Knowledge Cutoff Problem: LLMs have fixed training data ending dates (e.g., GPT-4's knowledge cuts off in April 2024). Users need access to current, proprietary, or domain-specific information not available in the model's training data.

Proprietary Information Gap: Organizations cannot share confidential documents with external LLM APIs for fine-tuning, yet need accurate answers based on internal policies, procedures, research, and knowledge bases.

Lack of Explainability: When an LLM generates an answer, users cannot verify its source or accuracy against their actual documents. This is critical in compliance-heavy industries like legal, healthcare, and finance.

Context Window Limitations: Even with extended context windows (100K tokens), manually inserting all relevant documents into every prompt is inefficient and costly.

Real-World Impact

• A financial services firm cannot get accurate answers about their internal lending policies
• A research institution struggles to provide researchers instant access to their 10,000+ paper archive
• A healthcare provider cannot ensure compliance when LLMs hallucinate medical procedures not in their protocols
• Customer support teams waste time training on static FAQs when they could have an AI assistant instantly retrieve relevant answers

2. The RAG Solution: Retrieval-Augmented Generation Explained

How RAG Solves These Problems

Retrieval-Augmented Generation combines three core components:

1. Document Ingestion & Embedding: Your documents are converted into semantic embeddings (numerical representations capturing meaning) and stored in a vector database
2. Semantic Retrieval: When a user asks a question, the system finds the most relevant document chunks based on semantic similarity, not just keyword matching
3. Grounded Generation: The LLM receives both the user's question AND the retrieved relevant context, then generates answers grounded in your actual data

Why This Architecture Works

• Accuracy: Answers are based on your actual documents, eliminating hallucinations from knowledge gaps
• Explainability: Every answer includes traceable sources from your document collection
• Cost Efficiency: You avoid fine-tuning large models; instead, you retrieve only relevant context for each query
• Flexibility: You can update your documents without retraining the model
• Scalability: As your document collection grows, the vector database scales efficiently

3. System Architecture: Components and Data Flow

Core Architecture Overview

The RAG system consists of five integrated components:

3.1 Document Ingestion Pipeline

• Accepts multiple file formats (PDF, TXT, DOCX)
• Extracts raw text while preserving structure and metadata
• Implements error handling for corrupted or unsupported formats
• Stores original document metadata (source, date, author) for retrieval traceability

3.2 Text Chunking Engine

• Breaks documents into semantic chunks optimized for retrieval
• Implements overlap strategy to preserve context at boundaries
• Maintains metadata association with each chunk
• Applies custom chunking logic for different document types

3.3 Embedding & Vector Database

• Converts text chunks into semantic embeddings using Sentence Transformers
• Stores embeddings in ChromaDB with full-text search capability
• Enables similarity search to find relevant contexts
• Supports metadata filtering for advanced retrieval

3.4 Retrieval & Ranking Engine

• Performs semantic similarity search on user queries
• Ranks results by relevance score
• Implements retrieval evaluation metrics
• Supports multi-query expansion for comprehensive coverage

3.5 Generation & Memory System

• Queries selected LLM provider with retrieved context
• Maintains conversation memory for multi-turn interactions
• Generates natural language responses
• Formats output with source citations

4. Technical Architecture Decisions and Rationale

4.1 Embedding Model Selection: Sentence Transformers (all-MiniLM-L6-v2)

Why This Model?

Trade-offs Considered:

• Didn't choose larger models (e.g., 1536D) because the accuracy gain (~2-3%) doesn't justify 4x memory and compute overhead for most use cases
• Didn't choose domain-specific embeddings (e.g., law, medical) because our system supports any document type
• Didn't choose proprietary embeddings (OpenAI's text-embedding-3) because they require API calls, adding latency and cost

4.2 Multi-LLM Provider Architecture

Why Support Multiple Providers?

Different use cases have different requirements:

Architecture Benefit: Users can switch providers based on:

• Budget constraints
• Latency requirements
• Specific task needs
• Regional availability or compliance requirements
  

5. Semantic Chunking Strategy: From Documents to Searchable Segments

5.1 The Chunking Challenge

Raw documents must be split into manageable segments for embedding and retrieval. Naive approaches fail:

• Too small chunks (100 tokens): Lost context, poor semantic understanding
• Too large chunks (2000 tokens): Irrelevant information mixed with relevant, increased embedding costs
• No overlap: Critical information spanning chunk boundaries gets lost

5.2 Our Chunking Implementation

Parameters:

• Chunk Size: 400-600 tokens (~1,500-2,000 characters)
  • Large enough to capture full context for a concept
  • Small enough to minimize noise and retrieval cost

• Overlap: 50-100 tokens (10-20% of chunk size)

  • Ensures information spanning boundaries is captured
  • Prevents context loss at chunk transitions
  • Example: If chunk ends with "CRISPR is a gene editing tool that works by...", next chunk starts with "...by targeting specific DNA sequences"

• Split Strategy: Prioritize semantic boundaries

  • Split at paragraph boundaries, not mid-sentence
  • Respect document structure (headings, lists)
  • Preserve code blocks, tables, and formatted content

Example Visualization:

Original Document:
[===== Chunk 1 (400 tokens) =====]    <- Covers concepts A, B
                           [===== Chunk 2 (400 tokens) =====]  <- Overlaps on B, covers B, C
                                                [===== Chunk 3 (400 tokens) =====] <- Overlaps on C, covers C, D

This overlap ensures that boundary concepts (B, C) are fully captured in multiple chunks

5.3 Chunking Optimization Techniques

1. Metadata Preservation: Associate each chunk with source document, page number, and timestamp
2. Semantic Validity: Chunks represent complete thoughts, not arbitrary text splits
3. Dynamic Adjustment: Analyze chunk retrieval success metrics and adjust chunk size for specific document types
4. Specialized Chunking: Different strategies for different content (code blocks kept intact, tables handled specially)

6. Query Processing and Optimization Pipeline

6.1 Query Understanding Pipeline

User queries vary wildly in complexity and structure. Our system implements multi-step processing:

Step 1: Query Normalization

• Remove special characters and excessive punctuation
• Standardize spacing and whitespace
• Normalize common abbreviations (e.g., "ML" → "machine learning" in context)

Step 2: Query Expansion

• Generate semantic variations of the query
• Example: User asks "How does CRISPR work?"
  • Expanded queries: "CRISPR mechanism", "CRISPR gene editing process", "molecular details of CRISPR"
• Improves retrieval by covering multiple phrasings of the same concept

Step 3: Intent Classification

• Determine query type: Factual, Explanatory, Comparative, or Multi-hop
• Route to appropriate retrieval strategy
• Example: Comparative queries retrieve multiple relevant chunks for side-by-side comparison

Step 4: Context-Aware Retrieval

• Use conversation history to disambiguate follow-up questions
• Example: "How does it work?" understood as "How does CRISPR work?" based on previous turn

6.2 Retrieval Optimization

Multi-Query Retrieval: Retrieve results for both original and expanded queries, then rank by frequency and relevance

Reranking: Use cross-encoder models to rerank initial retrieval results, improving top-K accuracy

Adaptive K: Adjust number of retrieved chunks based on query complexity (simple queries: K=3, complex: K=5-10)

7. Retrieval Evaluation Metrics: Measuring System Performance

7.1 Why Evaluation Matters

Without metrics, you can't distinguish between a working system and one that only seems to work. Metrics quantify:

• Which retrieval strategy works best for your documents
• Whether system improvements actually help users
• Where to focus optimization efforts

7.2 Key Retrieval Metrics

Metric 1: Hit Rate@K

• Definition: Percentage of queries where at least one relevant chunk appears in top K results
• Formula: (Queries with ≥1 relevant result in top K) / Total queries × 100%
• Target: >90% hit rate@10 for production systems
• Interpretation: If hit rate drops, your retrieval is missing relevant information

Metric 2: Precision@K

• Definition: Percentage of retrieved chunks that are actually relevant
• Formula: (Relevant chunks in top K) / K × 100%
• Target: >70% precision@5 is good; >80% is excellent
• Interpretation: How much noise is in your results? High precision = fewer irrelevant chunks

Metric 3: Recall@K

• Definition: Percentage of all relevant chunks successfully retrieved in top K
• Formula: (Relevant chunks retrieved) / (Total relevant chunks available) × 100%
• Target: >80% recall@10
• Interpretation: Are you missing relevant information? Low recall = incomplete answers

Metric 4: Mean Reciprocal Rank (MRR)

• Definition: Average position (1/rank) of the first relevant result
• Formula: (1/rank of first relevant result) averaged across queries
• Target: >0.8 MRR (first relevant result usually in top 1-2 positions)
• Interpretation: How quickly does the user find the answer?

Metric 5: Normalized Discounted Cumulative Gain (NDCG)

• Definition: Weighted ranking quality considering both relevance and position
• Target: >0.75 NDCG@10
• Interpretation: Holistic measure of ranking quality; penalizes relevant results placed low

7.3 Evaluation Framework Implementation

# Pseudocode for evaluation workflow

class RAGEvaluator:
    def evaluate_system(self, test_queries, ground_truth_docs):
        """
        test_queries: List of [query, list of relevant doc_ids]
        ground_truth_docs: Dictionary mapping doc_id to actual relevant docs
        """
        
        hit_rate_at_10 = self.calculate_hit_rate(test_queries, k=10)
        precision_at_5 = self.calculate_precision(test_queries, k=5)
        recall_at_10 = self.calculate_recall(test_queries, k=10)
        mrr = self.calculate_mrr(test_queries)
        ndcg = self.calculate_ndcg(test_queries, k=10)
        
        return {
            'hit_rate@10': hit_rate_at_10,
            'precision@5': precision_at_5,
            'recall@10': recall_at_10,
            'mrr': mrr,
            'ndcg@10': ndcg
        }

7.4 Using Metrics to Improve Performance

1. Low Hit Rate: Your retrieval is missing results. Try:

   • Query expansion strategies
   • Different embedding models
   • Larger K values

2. Low Precision: Too many irrelevant chunks. Try:

   • Stricter similarity thresholds
   • Reranking with cross-encoders
   • Smaller K values

3. Low Recall: Missing relevant information. Try:

   • Better chunking strategy
   • Multi-query retrieval
   • Larger chunk size

8. Conversational Memory: Context Across Turns

8.1 Multi-Turn Interaction Challenge

Users don't ask isolated questions; they follow up with clarifications and related questions:

Turn 1: User: "What is CRISPR?"
        Assistant: "CRISPR-Cas9 is a gene-editing technology..."

Turn 2: User: "How does it work?"
        Assistant: Must understand "it" refers to "CRISPR" from Turn 1
        
Turn 3: User: "What are the risks?"
        Assistant: Must maintain context about CRISPR throughout conversation

8.2 Memory Implementation

ConversationBufferMemory maintains:

• User input history: All previous user messages
• Assistant responses: All previous generated answers
• Context window: Typically last 5-10 turns to balance context and token limits

Integration with RAG:

1. User asks a question
2. System retrieves relevant chunks using combined query + conversation history
3. LLM receives: [retrieved chunks] + [conversation history] + [current question]
4. LLM generates answer using full context
5. New turn is added to memory

8.3 Memory Limitations and Future Work

Current Limitation: Memory resets when application restarts (in-memory buffer)

Future Enhancement: Persistent database storage enabling:

• Long-term conversation analysis
• User behavior learning
• Personalized response styles
• Chat history search and replay

9. Installation and Usage Guide

9.1 Prerequisites

• Python 3.9 or higher
• pip package manager
• 2GB RAM minimum (4GB+ recommended)
• 500MB disk space for dependencies
• Valid API keys for at least one LLM provider:
  • OpenAI: https://platform.openai.com/api-keys
  • Groq: https://console.groq.com/keys
  • Google Gemini: https://ai.google.dev/

9.2 Installation Steps

Step 1: Clone Repository

git clone https://github.com/EphraimMagopa/rag-agentic-ai-project1.git
cd rag-agentic-ai-project1

Step 2: Create Virtual Environment

python -m venv env
source env/bin/activate  # On Windows: env\Scripts\activate

Step 3: Install Dependencies

pip install -r requirements.txt

Step 4: Configure Environment Variables

cp .env_example .env
# Edit .env with your API keys
nano .env  # or use your preferred editor

Example .env configuration:

# LLM Configuration (set at least one)
OPENAI_API_KEY=sk-proj-xxxxx
OPENAI_MODEL=gpt-4o-mini

GROQ_API_KEY=gsk_xxxxx
GROQ_MODEL=llama-3.1-8b-instant

GOOGLE_API_KEY=xxxxx
GOOGLE_MODEL=gemini-2.0-flash

# Vector Database
CHROMADB_COLLECTION_NAME=rag_documents
EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2

Step 5: Prepare Documents

mkdir -p data
# Add your .txt or .pdf files to data/ directory

Step 6: Verify Installation

python -c "import langchain; import chromadb; print('Installation successful!')"

9.3 Usage Examples

Launch Interactive Application

python app.py

Example Session 1: General Knowledge

Enter a question or 'quit' to exit: What is CRISPR gene editing?
<!-- RT_DIVIDER -->
# Experiments
We tested the assistant with a sample corpus of Ready Tensor publications and Wikipedia articles. Each interaction involved:
- Asking an initial factual question covered by the corpus.
- Following up with a related, more specific or context-dependent question.
- Testing edge cases such as queries outside the knowledge base.
Relevant logging and print statements were enabled to verify memory context and retrieval effectiveness.
<!-- RT_DIVIDER -->
# Results
- The assistant provides clear, well-structured answers grounded in the ingested documents.
- Memory integration demonstrated improved handling of follow-up questions (e.g., pronoun reference resolution).
- Example interaction:

:::note{title="Example interaction:"}
User: What is CRISPR gene editing?
Assistant: CRISPR-Cas9 is a revolutionary gene-editing technology...

User: How does it work?
Assistant: CRISPR works by targeting specific DNA sequences...
:::
- Out-of-scope questions receive appropriate fallback responses:
"I am sorry, but I do not have enough information to answer that."
<!-- RT_DIVIDER -->
# Conclusion
This RAG-based AI assistant combines the strengths of vector retrieval and LLM reasoning with the added context of conversational memory. It sets the foundation for more advanced agentic AIs that can integrate broader tool use, intermediate reasoning, and persistent knowledge. Future enhancements may include real tool integration, persistent long-term chat memory, richer document types, and advanced reasoning modules.