Abstract

Despite being a fairly new offering, vector database vendors have sprung up like wild mushrooms after the rain. As of this publication there are more than 20 vector database vendors, each offering different options and prices.
This technical publication combines vector database, RAG and LLM for processing and retrieval of a set of publicly accessible publications related to vector database offerings. The system integrates document processing, vector embedding, and natural language generation to create an intelligent question-answering framework.
The implementation processes Text and PDF documents from a designated directory, segments them into semantic chunks, and creates vector embeddings using HuggingFace's sentence-transformers model. These embeddings are stored in a ChromaDB vector database, enabling efficient similarity search. When presented with a query, the system retrieves the most relevant document fragments and utilizes GROQ's language model to generate responses.

System Architecture

This publication presents implementation of a RAG-based AI Agent - a system designed to answer questions based on document collections through a multi-stage process:

1. Document Processing Pipeline: The system ingests text and PDF documents from a specified directory, processes them through chunking algorithms to create semantically meaningful segments, and prepares them for embedding.
2. Vector Representation and Storage: Document chunks are converted into dense vector embeddings using HuggingFace's sentence-transformers model and stored in ChromaDB - a vector database optimized for similarity search operations.
3. Query Processing: When presented with user questions, the system retrieves the most relevant top 5 document fragments through semantic similarity matching.
4. Context-Aware Response Generation: The retrieved document fragments and user query are passed to GROQ's language model, which generates coherent, contextualized responses.

Core Components

Link to Codebase: https://github.com/rrubis/ready-tensor-assignment-1

1. Document Processing Pipeline

• Document Loading: Implemented insrc/app.py this component scans the configurable document directory (data) for Text and PDF files and loads them using LangChain's TextLoader and PyPDFLoader.

if file.endswith(".txt"):
    file_path = os.path.join(documents_path, file)
    try:
        loader = TextLoader(file_path, autodetect_encoding=True)
        loaded_docs = loader.load()
        documents.extend(loaded_docs)
        logger.info(f"Successfully loaded: {file}")
    except Exception as e:
        logger.warning(f"Error loading {file}: {str(e)}")
elif file.endswith(".pdf"):
        file_path = os.path.join(documents_path, file)
        try:
            loader = PyPDFLoader(file_path)
            loaded_docs = loader.load()
            documents.extend(loaded_docs)
            logger.info(f"Successfully loaded: {file}")
        except Exception as e:
            logger.warning(f"Error loading {file}: {str(e)}")

• Document Chunking: After loading, documents are segmented into smaller chunks using RecursiveCharacterTextSplitter with configurable chunk size (default: 1000 characters) and overlap parameters (default: 200 characters). This chunking strategy preserves semantic coherence while creating appropriately sized segments for embedding.

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=chunk_size,
    chunk_overlap=chunk_overlap,
)
chunks = text_splitter.split_text(text)

2. Vector Database System

The vector database subsystem manages document embeddings and similarity search operations:

• Embedding Generation: Implemented in src/vectorydb.py the module utilizes HuggingFace's sentence transformers using the sentence-transformers/all-MiniLM-L6-v2 model to convert text chunks into database vector representations.

embeddings = self.embedding_model.encode(chunks)
self.collection.add(
    embeddings=embeddings,
    documents=chunks,
 )

• Vector Storage: ChromaDB serves as the persistent vector store, maintaining embeddings and their associated document metadata in the db directory.

self.client = chromadb.PersistentClient(path=db_path, settings=Settings(allow_reset=True))

3. Query Processing and Response Generation

The query processing and response generation system combines information retrieval with language generation:

• LLM Integration: Implemented in the src/app.py the module interfaces with GROQ's API, initializing the language model.

model_name = os.getenv("GROQ_MODEL", "llama-3.1-8b-instant")
llm = ChatGroq(
    api_key=os.getenv("GROQ_API_KEY"), model=model_name, temperature=0.0
)

• RAG Chain Assembly: In the RAG implementation, the system assembles the retrieval-augmented generation chain that connects vector similarity search with the prompt template formatting and language model generation.
  Vector Database Query Execution and Prompt Template Formatting and Language Model Generation:

results = self.collection.query(
    query_embeddings=[query_embedding],
    n_results=n_results,
    include=["documents", "metadatas", "distances"],
)

results = self.vector_db.search(input, n_results)
prompt_value = self.prompt_template.format_messages(context = results["documents"], question = input)
llm_answer = self.llm.invoke(prompt_value).content

4. Configuration Management

The module utilizes configuration parameters and prompt template.
Environment Configuration and Prompt Template:

# Groq API Configuration
GROQ_API_KEY=your_groq_api_key_here
GROQ_MODEL=llama-3.1-8b-instant

# Embedding Configuration
EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2

# Vector Database Configuration
CHROMA_COLLECTION_NAME=rag_documents

template = """
role: |
            A helpful assistant that can answer the users questions given some relevant documents.
style_or_tone:
            - Use clear, concise language with bullet points where appropriate.
instruction: |
            Given the some documents that should be relevant to the user's question, answer the user's question.
output_constraints:
            - Only answer questions based on the provided documents.
            - If the user's question is not related to the documents, then you SHOULD NOT answer the question. Say "The question is not answerable given the documents".
            - Never answer a question from your own knowledge.
output_format:
            - Provide answers in markdown format.
            - Provide concise answers in bullet points when relevant.
Relevant documents: {context}
User's question: {question}
"""

Information Flow

1. Initialization Phase

• The system checks for an existing vector store, ff one exists, it is deleted
• New vector store is created
• Documents in data directory are loaded
• Documents are split into semantic chunks
• Chunks are embedded and stored in ChromaDB

2. Query Phase:

• User submits a question through the CLI interface
• The system retrieves the vector database
• The question is embedded and used to perform similarity search
• Relevant document chunks are retrieved
• Retrieved context and question are formatted into a prompt
• The LLM generates a response based on the provided context
• The answer is returned to the user

3. Integration Points

The src/app.py file serves as the integration point:

• init_vector_db() initializes vector database
• load_documents() loads documents from the data directory and performs chunking
• add_documents() embeds documents and loads them in the vector database
• query() implements the query workflow
• main() provides the CLI user interface

This architecture follows separation of concerns principles, with each module handling a specific aspect of the RAG workflow. The modular design allows for component replacement or enhancement without disrupting the overall system function, making the system adaptable to future improvements in embedding models, language models, or chunking strategies.

Conclusion

This publication has presented a simple implementation of a Retrieval Augmented Generation (RAG) system designed to provide contextually relevant answers to questions based on document collections. Several key conclusions can be drawn.

Key Achievements

The RAG-based AI Agent successfully demonstrates:

1. Effective Document-Grounded Responses: By combining vector-based retrieval with language model generation, the system produces responses that remain faithful to the source material, reducing the hallucination problems common in pure LLM approaches.
2. Modular, Maintainable Architecture: The system's clear separation of concerns across document processing, vector representation, and response generation creates a flexible framework that can be extended and maintained with minimal coupling between components.
3. Practical Implementation Balance: The chosen technologies, particularly the "all-MiniLM-L6-v2" embedding model and "llama-3.1-8b-instant" language model, establish an effective balance between performance, resource efficiency, and response quality.
4. Intelligent Context Retrieval: The semantic search capabilities enabled by the vector database ensure that responses are informed by the most relevant document fragments, even when user queries don't exactly match document terminology.
5. Multiple Document Formats: In addition to Text document processing, the system can also load and process PDF documents.
6. Logging: Implemented logging support utilizing logging library.

Limitations

1. Document Format Restrictions: The system currently only processes plain text (.txt) and PDF files, limiting its application for more complex document formats without additional preprocessing.
2. Fixed Chunking Strategy: The recursive character splitting approach, while effective, uses static chunk size parameters that may not be optimal for all document types or query patterns.
3. Limited Document Understanding: The system treats each document chunk as an independent unit, potentially missing cross-document relationships or broader context that spans multiple chunks.
4. PDF Files are Slower to Load: Loading and processing PDF files is significantly slower than loading similar Text files. Text files also appear to generate more relevant results than their PDF counterparts.

Future Work

Several promising directions for future development emerge from this implementation:

1. Enhanced Document Processing: Expanding support for additional document formats (Word, HTML, etc.) would broaden the system's applicability across different knowledge domains.
2. Dynamic Chunking Strategies: Implementing adaptive chunking that adjusts parameters based on document structure and content density could improve retrieval precision.
3. Evaluation Framework: Developing comprehensive metrics for measuring response accuracy, relevance, and coherence would facilitate systematic improvement of the RAG pipeline.
4. PDF Files Performance and Accuracy: Improving the load process and more accurate result generation of PDF files.

Final Remarks

The RAG-based AI Agent demonstrates how the integration of information retrieval with generative AI can create systems that leverage the strengths of both approaches. By grounding language model outputs in specific document collections, it is possible to achieve a more accurate, trustworthy AI assistants that can effectively serve as knowledge interfaces to organizational documentation.