History-Aware Multimodal PDF Question Answering: Integrating Text, Images, and Query Rewriting

Overview

Reading PDFs that contain figures takes more than standard text splitting. This project handles text and image extraction, uses vision models to summarize pictures, stores everything locally in ChromaDB, and cleans up questions upfront so vector searches actually pull the right pages without altering the user's original query intent during final response generation.

Core Architecture & Components

1. Multimodal PDF Processing (app.py)

   • Parses text page-by-page using PyMuPDF.
   • Extracts embedded images, filters out decorative icons/noise (size thresholding), and converts valid images into base64 strings.
   • Sends each extracted image to a vision-capable LLM (ex: qwen/qwen3.8-27b) to generate a detailed summary/description of what it contains.
   • Merges text chunks and generated image descriptions into a unified vector stream in ChromaDB.

2. Persistent Vector Store & Chunking (vectordb.py)

   • Uses sentence-transformers/all-MiniLM-L6-v2 for local embedding generation.
   • Integrates Chroma with local persistence (./chroma_db) and deterministic MD5-based chunk IDs for idempotent upserts.
   • Applies cosine distance thresholding to filter out low-relevance search hits.

3. Two-Stage Conversational Processing & Query Rewriting

• **Stage 1 (Vector Store Retrieval): Maintains recent conversation history and rolling summaries of older messages. A strictly constrained Query Rewriter function uses this history exclusively to resolve ambiguous pronouns (it, they, that) and transform implicit follow-ups into a clean, standalone search query used for vector database retrieval.
• **Stage 2 (Response Generation): Both the user's original query and the rewritten query (along with the retrieved chunks) are passed to the final generator model. This approach ensures the model retains the precise intent and nuance of the user's original phrasing while leveraging the disambiguated context of the rewritten query for complete clarity.

System Architecture Diagram

   graph TD
    %% Row 1: Ingestion & Summarization
    subgraph Row1 [" "]
        direction LR
        A["PDF Files<br/>in data/"] -->|"PyMuPDF<br/>Text Extract"| B["Text Pages"]
        A -->|"PyMuPDF<br/>Image Extract"| C["Raw Images"]
        
        C -->|"Filter < 150px<br/>& Base64"| D["Vision Model<br/>(Qwen 3.8 27B)"]
        D -->|"Generate<br/>Summaries"| E["Image<br/>Descriptions"]
        
        B --> F["Combined<br/>Documents"]
        E --> F
    end

    %% Row 2: Vector Store Indexing
    subgraph Row2 [" "]
        direction LR
        G["Document<br/>Chunks"] -->|"HuggingFace<br/>Embeddings"| H["Vector<br/>Embeddings"]
        H -->|"Upsert MD5 IDs"| I[("ChromaDB<br/>Vector Store")]
    end

    %% Link Row 1 to Row 2
    F -->|"Recursive Splitter"| G

    %% Remove subgraph borders for a clean look
    style Row1 fill:none,stroke:none;
    style Row2 fill:none,stroke:none;

Tech Stack

• Language Runtime: Python 3.13 (required, as several dependencies rely on this specific version)
• Orchestration: LangChain (0.3.x)
• Vector Database: ChromaDB (Persistent Client)
• PDF & Vision Parsing: PyMuPDF (pymupdf)
• Embeddings: HuggingFace (sentence-transformers/all-MiniLM-L6-v2)
• LLM Providers: Groq / OpenAI / Google Gemini APIs

Configuration Files

The behavior of the RAG system is fully modular and controlled via two YAML files in the config/ directory:

1. config.yaml: Manages global retrieval and conversational window parameters.

   • n_results: Maximum number of document chunks to retrieve per search.
   • n_threshold: Distance score threshold for vector matching (filters out irrelevant hits).
   • max_recent_messages: Sliding window size for retaining active chat history before triggering the background summarization chain.

2. prompt_config.yaml: Houses the structured prompt settings for both the Query Rewriter and the final RAG Generator chain.

   • Defines the role, explicit instruction, output_constraints, style_or_tone, and goal.
   • Configures the query rewriter prompt exclusively to produce standalone queries for vector store retrieval.
   • Configures the final generator prompt to be purely stateless, accepting the user's original and rewritten question alongside the XML-wrapped {context} chunks.
   • Handled by the build_prompt function to dynamically construct system instructions without hard-coding them into Python scripts.

System Limitations

While the multimodal pipeline significantly improves document intelligence over text-only RAG systems, it currently has the following constraints:

• Scope of Image Extraction: The image processing relies on embedded PDF image streams (page.get_images()). Consequently, standalone vector graphics, native charts, and vector-drawn diagrams are not extracted.
• Processing Latency: Processing PDFs with numerous images takes a few minutes. Once an image is extracted, the pipeline converts it to base64, passes it to the vision model to request an image summary, and adds brief delays between requests to stay within API rate limits before saving the chunk into ChromaDB.

Installation & Setup

1. Create a virtual environment:

   python -m venv .venv
   source .venv/bin/activate

2. Install dependencies:

   pip install -r requirements.txt

3. Configure Environment:

   Copy the example environment file to create your .env configuration:

   cp .env.example .env  

   To process embedded images found in PDF files, ensure you specify a vision-capable model (like qwen/qwen3.8-27b) inside .env:

   # Groq API Configuration (alternative to OpenAI)
   GROQ_API_KEY=your_groq_api_key_here 
   VISION_MODEL=qwen/qwen3.8-27b
   GROQ_MODEL=openai/gpt-oss-20b
   
   # Embedding Configuration
   # Optional: HuggingFace model for embeddings (default: sentence-transformers/all-MiniLM-L6-v2)
   EMBEDDING_MODEL=
   
   # Vector Database Configuration
   # Optional: ChromaDB collection name 
   CHROMA_COLLECTION_NAME=
   
   # Hugging Face token
   HF_TOKEN=your_hf_token_here

How to Run

1. Prepare Data: The project includes a sample PDF in the data/ directory covering the basics of dogs and cats. You can use this default file or place any additional PDF documents you want to analyze directly inside the data/ folder.

2. Start the Assistant:

   python app.py

   Note: On startup, the system will automatically parse all PDFs located in the data/ folder, extract text and images, embed them, and ingest them into the local ChromaDB database. Since we use the 'upsert' method when adding documents to the vector store, running the script again will update existing pages and add any new ones.

3. Interact: Once ingestion is complete, type your questions into the terminal prompt to query your documents. Type quit to exit.

Summary

By combining targeted vector search via a rewritten query with the preservation of the user's original query in the final prompt, this setup ensures high retrieval accuracy and maintains the exact intent of the initial user interaction.