Building a Domain-Aware RAG Assistant for Private Document Intelligence using ChromaDB

A production-grade Retrieval-Augmented Generation (RAG) system for secure document intelligence, semantic search, and contextual question answering.

Problem Statement

Traditional LLM-based chat systems fail in private and enterprise environments due to hallucinations, lack of grounding, inability to operate on private data, and absence of evaluation mechanisms. Generic chatbots cannot reason over proprietary documents, cannot enforce knowledge boundaries, and cannot guarantee answer traceability.

This project implements a Retrieval-Augmented Generation (RAG) assistant that enables grounded reasoning over private documents with strict knowledge boundaries, semantic retrieval, and controlled generation.

Solution Overview

The system follows a complete RAG pipeline:

Ingestion → Embedding → Vector Storage → Retrieval → Context Fusion → LLM Reasoning → Guardrails → Response Generation

The LLM is constrained to operate only on retrieved document context.

System Architecture

Core Components

1. Document Ingestion Layer
   Loads, normalizes, and prepares user documents for embedding.

2. Embedding Layer
   Converts text into dense semantic vectors for similarity search.

3. Vector Storage Layer (ChromaDB)
   Persistent vector database for fast semantic retrieval.

4. Retrieval Engine
   Performs similarity search and relevance ranking.

5. Context Fusion Layer
   Injects retrieved content into structured prompts.

6. LLM Reasoning Layer
   Generates responses strictly grounded in retrieved context.

Project Structure

RAG-chat-with-docs-chromadb/
│
├── main.py
│   # Entry point of the application.
│   # Loads documents into ChromaDB (based on config)
│   # and invokes the LLM for summarization or Q&A.
│
├── src/
│   └── vector_db.py
│       # Handles vector database creation, embedding,
│       # and interaction with ChromaDB.
│
├── raw_data/
│   # Contains sample input documents for embedding.
│   # Currently supports .txt files only.
│
├── properties/
│   ├── vector-config.yaml
│   # Configuration file for vector database behavior.
│   #
│   # Example:
│   # load_into_vector: true
│
│   └── vector_config_loader.py
│       # Utility to load and parse vector-config.yaml.
│
└── README.md

Configuration Layer

Configuration file:

properties/vector-config.yaml

Control Flag

load_into_vector: true | false

Controls document ingestion, re-indexing, and vector persistence reuse.

Embedding Model

Model:
sentence-transformers/all-MiniLM-L6-v2

Selected for semantic quality, low latency, and production suitability for RAG pipelines.

Query Processing Pipeline

1. Query normalization
2. Semantic vectorization
3. Similarity search
4. Top-k retrieval
5. Context ranking
6. Prompt context injection
7. Controlled LLM generation

Retrieval Strategy

• Semantic similarity search
• Chunk-based indexing
• Top-k retrieval
• Context ranking prior to generation

Guardrails and Safety

Knowledge Boundary Control

LLM restricted to retrieved context only.

Hallucination Prevention

No-context → controlled fallback response.

Data Privacy

Local processing, private vector storage, no document leakage.

Prompt Injection Protection

Context isolation and structured prompts.

Source Grounding

All responses traceable to retrieved documents.

Retrieval and Generation Evaluation

Retrieval Metrics

• Precision@k
• Recall@k
• Semantic relevance scoring

Generation Metrics

• Faithfulness
• Groundedness
• Hallucination rate
• Context adherence

LLM Provider Abstraction

Supported providers:

• OpenAI
• Groq
• Google Gemini

Dynamic provider selection via environment configuration.

Environment Configuration

.env example:

OPENAI_API_KEY=your_openai_api_key_here
OPENAI_MODEL=gpt-4o-mini

GROQ_API_KEY=your_groq_api_key_here
GROQ_MODEL=llama-3.1-8b-instant

GOOGLE_API_KEY=your_google_api_key_here
GOOGLE_MODEL=gemini-pro

Supported Data Formats

Current:

• .txt

Planned:

• PDF
• DOCX
• HTML
• Markdown
• CSV
• JSON

RAG Assistant Scope

Use cases:

• Private document QA
• Internal knowledge assistant
• Research assistant
• Enterprise documentation bot
• Developer knowledge system
• Policy and compliance assistant

Evaluation Readiness

Architecture supports:

• Retrieval benchmarking
• RAG evaluation pipelines
• Automated regression testing
• Dataset-based validation
• Golden-answer verification

Output screenshot

Repository

https://github.com/Raghul-S-Coder/RAG-chat-with-docs-chromadb.git