Abstract

Annual reports are some of the hardest documents to build a question-answering system for. A single report mixes long narrative text, financial statements with hundreds of exact figures, governance tables marked with checkmark glyphs, charts and architecture diagrams, two-page spreads, multi-column layouts and, occasionally, pages whose embedded text layer is simply broken. A RAG pipeline that reads only the text layer misses the tables and charts, and one that gets a single digit wrong is worse than useless.

ReportLens is an agentic, multimodal retrieval-augmented generation (RAG) system that answers questions about one long PDF using its text, its tables and its charts, and cites the page of every fact it uses. It was built and tested on NVIDIA's 2026 Annual Report (175 pages). The ingestion pipeline parses the report with LlamaParse (agentic tier), splits each page at its headings into 377 text chunks with tables kept whole as HTML, has a vision model (DeepSeek-V4.1-Flash) write searchable descriptions of the report's 12 charts and diagrams, and embeds everything with Jina Embeddings v5 into a local Chroma store. At question time a LangChain agent decides whether and how often to search; when a search returns a chart, the chart image itself is shown to the model, not only its description. A FastAPI backend streams the agent's searches and answer to a React web UI as server-sent events, keeps short per-conversation memory for follow-up questions, and every question is traced, with token counts and cost, in LangSmith.

All six test questions (income statement figures, a checkmark director table, an architecture diagram, a stock performance graph, a greeting and an unanswerable question) are answered correctly with page citations. A question takes 3 to 7 seconds end to end and costs a fraction of a cent (about $0.0003 for a question with one search).

This publication documents the whole path: why the first parser was replaced (with the experiments that showed it), every design decision in ingestion and retrieval, the streaming API and UI, the measured results, and the limitations that remain.

NVIDIA's annual report is a public document, used here only as a test document. ReportLens is not affiliated with NVIDIA, and its answers are not financial advice.

1. Introduction

1.1 The problem

Ask a general-purpose chatbot "What was NVIDIA's net income in fiscal 2026?" and you will get a number, but you have no way to know where it came from or whether it is right. Ask a basic text-only RAG pipeline the same question over the annual report and it will often fail for more mundane reasons:

• The answer is in a table. Financial statements are tables with three years of figures side by side. Naive PDF text extraction flattens them into a stream of numbers with the column structure lost, and a chunker may cut the table in half.
• The answer is in a chart. The five-year stock performance comparison lives only in a line graph. The values are not in the page text at all.
• The answer is in a diagram. An architecture diagram's meaning is in its boxes and arrows. Text extraction returns, at best, a list of labels without connections, and at worst OCR noise.
• The layout fights the parser. The report opens with wide two-page spreads, has three-column letters, a landscape table, checkmarks drawn with a symbol font, and one page whose text layer is garbled by a broken font encoding.
• Every number must be exact and checkable. For a financial document, an answer is only useful if the reader can open the page it came from and verify it.

1.2 Objective

Build a chatbot over a single long, visually complex PDF that:

1. Answers from all three modalities: narrative text, tables (with exact figures) and charts/diagrams.
2. Cites the page of every fact, and lets the reader open the retrieved page excerpt and figure behind each citation.
3. Refuses when the document does not hold the answer, instead of guessing.
4. Handles follow-up questions ("And the year before?") within a conversation.
5. Feels responsive: shows its searches and streams the answer as it is written.
6. Stays simple and readable: one plain module per pipeline step, every setting in one config file.

1.3 Why this task?

I built ReportLens to learn what actually breaks when RAG meets a real-world document rather than a clean benchmark corpus. Annual reports are a good stress test: they are long, public, dense with numbers, and designed for print rather than for machines. Most of the interesting work in this project turned out not to be in the language model at all, but in parsing, chunking and deciding what deserves to be embedded. Every one of those decisions was made from an experiment in a notebook, and the findings are reported below.

2. The Document: NVIDIA's 2026 Annual Report

Before building anything, I measured what the document actually looks like. The page sizes alone show three different layouts:

Other features that matter for a RAG system:

• Tables everywhere. LlamaParse later found 100 table regions; 99 of the final 377 text chunks contain at least one HTML table.
• Checkmark tables. The director nominee table (page 26) marks independence and audit committee financial experts with checkmarks drawn in a symbol font.
• A broken text layer. Page 128's embedded text reads like >KH IROORZLQJ WDEOH: every letter is shifted by a constant offset, because the font maps characters to the wrong code points. Any parser that trusts the text layer reads gibberish there. A scan of all 175 pages for common English words found that page 128 is the only one affected.
• Charts and diagrams. A five-year cumulative total return graph (page 123), an agentic AI architecture diagram (page 8), and ten other charts.
• Two page numbering systems. The PDF page number and the number printed on the page differ (PDF page 139 is printed as page 51). 152 of the 175 pages carry a printed number.

3. System Overview

ReportLens has two halves: an ingestion pipeline that runs once, offline, and turns the PDF into a searchable vector store; and a query path that runs for every question.

flowchart TB
    subgraph ING["Ingestion: offline, run once"]
        direction LR
        PDF["Annual report PDF<br/>175 pages"] --> PARSE["Parse<br/>LlamaParse agentic tier"]
        PARSE -->|"markdown per page,<br/>tables as HTML"| CHUNK["Chunk at headings<br/>377 text chunks"]
        PARSE -->|"202 cropped<br/>figures and tables"| PICK["Keep the charts<br/>12 of 202"]
        PICK --> DESC["Describe charts<br/>DeepSeek V4.1 Flash"]
        CHUNK --> EMB["Embed<br/>Jina v5 small"]
        DESC --> EMB
    end
    EMB --> DB[("Chroma vector store<br/>389 documents")]
    subgraph QRY["Query path: every question"]
        direction LR
        UI["React web UI"] -->|"POST /ask"| API["FastAPI<br/>server-sent events"]
        API --> AGENT["LangChain agent<br/>DeepSeek V4.1 Flash"]
        AGENT -->|"search, text,<br/>done events"| API
        API -->|"streamed answer<br/>and sources"| UI
    end
    AGENT <-->|"search_document tool:<br/>chunks, scores, chart images"| DB

Technology stack

The Python side has ten pinned dependencies in total (fastapi, ipykernel, langchain, langchain-chroma, langchain-core, langchain-openai, langchain-text-splitters, llama-cloud, python-dotenv, uvicorn). Every setting lives in one file, multimodal_rag/config.py.

4. Choosing a Parser: The Experiments That Shaped the Pipeline

The parser decides what every later stage can see. If it duplicates lines, every chunk carries noise. If it splits a table, retrieval returns half a table. If it turns a diagram into OCR fragments, no amount of prompt engineering brings the diagram back. So most of the early work went into this one choice.

4.1 Where the project started

ReportLens began as a multimodal RAG pipeline over the first six pages of Attention Is All You Need, first with Docling (and Pinecone), then with Unstructured's hi_res strategy running locally: YOLOX layout detection, Table Transformer for table structure and Tesseract OCR, set up on a GPU VM without sudo. It worked on the paper, apart from Tesseract garbling math inside tables (O(n² · d) came out as O(n? - d)). Before ingesting the 175-page annual report, I tested it on the pages most likely to break it.

4.2 Unstructured hi_res on the annual report

Notebook 03_annual_report_layout.ipynb parsed eight representative pages (two spreads with diagrams, the three-column letter, the checkmark table, the broken-font page, the income statement and the landscape table) with the exact settings of the pipeline, then drew each element's box and reading order on the page image.

What worked: reading order. On spreads, Unstructured read the left half and then the right half, with no text block mixing the two. The three-column letter was read column by column with no jumps back. And hi_res OCR recovered page 128's broken text.

What did not:

Unstructured's hosted API might have done better, but its sign-up requires a work email, so it was not an option for this project.

4.3 LlamaParse (agentic tier) on the same pages

Notebook 04_llamaparse_test.ipynb parsed seven of the same pages with LlamaParse's agentic tier, where a vision model reads each page image. It then checked the output the same way, plus one extra test: every number in the markdown was checked against the PDF's own text layer, because a vision model can misread or invent a digit.

Numbers check: of the 145 numbers in the markdown of pages 13, 26, 139 and 142 (125 of them on the two financial statements), every one appears in the PDF text layer. No digit was misread or invented. (Page 128 was skipped, since its text layer is the broken one.)

Page 8's architecture diagram is the clearest example of the difference. Unstructured produced fragments like NEMOCLAW £ ma ” 1 5 \ \ \ TOOLS oe | CLI, MCP iia OPENSHELL. LlamaParse cropped the diagram as an image and also wrote this into the page markdown:

flowchart LR
    A["MULTI-MODAL PROMPT"]
    B["FILES STRUCTURED, UNSTRUCTURED"]
    C["NEMOCLAW"]
    D["MEMORY"]
    E["LLM"]
    F["SUB-AGENTS"]
    G["SKILLS"]
    H["COMPUTER USE"]
    I["TOOLS CLI, MCP"]
    J["OPENSHELL"]
    K["NEMOTRON NEMO DYNAMO NIM"]
    L["AI-Q"]
    M["cuOPT"]
    N["CUDF, CUVS"]
    O["vGPU"]
    A <--> C
    B <--> C
    C <--> D
    C <--> E
    C <--> F
    C <--> H
    C <--> I
    C <--> J
    E <--> K
    E <--> L
    E <--> M
    F <--> G
    G <--> M
    N --> B
    O --> H

The page 26 director table, which Unstructured had filled with ü, came back as clean HTML:

<tr><td>Jen-Hsun Huang</td><td>63</td><td>1993</td><td></td><td></td><td></td><td></td></tr>
<tr><td>Dawn Hudson</td><td>68</td><td>2013</td><td>✓</td><td>✓</td><td>CC Chairperson</td><td></td></tr>

Decision: LlamaParse replaced Unstructured. Reading order had never been the problem; output quality was.

Cost: the agentic tier costs 10 credits a page, so the full report costs about 1,750 credits. Because ingestion saves the parse result, this is paid once. (The credits came from an account that opted into data sharing for model training, which is fine for a public annual report. Private documents should use an account without that option.)

5. The Ingestion Pipeline

python -m multimodal_rag.ingest runs four steps, each in its own small module:

def main():
    pages, crops = load_parsed_pdf(config.DOCUMENT_PATH)       # 1. parse.py
    text_documents = build_text_documents(pages)               # 2. chunking.py
    figure_documents = build_figure_documents(crops, pages)    # 3. figures.py
    save_documents(text_documents + figure_documents)          # 4. store.py

5.1 Parse (parse.py)

result = client.parsing.parse(
    file_id=file.id,
    tier="agentic",
    version="latest",
    output_options={
        "markdown": {"tables": {"output_tables_as_markdown": False, "merge_continued_tables": True}},
        "images_to_save": ["layout"],  # the figures and tables cropped from each page
        "extract_printed_page_number": True,
    },
    expand=["markdown", "metadata", "images_content_metadata"],
)

• Tables as HTML, not markdown. HTML keeps merged header cells (such as "Year Ended" spanning three date columns), which markdown tables cannot express.
• merge_continued_tables joins a table that runs over several pages onto its first page, so it is never split across chunks.
• extract_printed_page_number reads the number printed on each page, stored alongside the PDF page number.
• images_to_save: ["layout"] crops every detected figure, chart and table as its own image. The full report produced 202 crops: 100 tables, 90 images and 12 charts.

Two robustness details came from a real failure. On the first full run the parse succeeded, but one image download dropped its connection, and because the result was saved only after the images, the parse was lost. It was recovered at no cost by fetching the finished job by its id (LlamaCloud().parsing.get(job_id, expand=[...])). Since then:

1. The parse result is written to disk the moment the parse finishes, before any download. It is what costs credits, so it is protected first.
2. Image links expire, so every run asks the parse job for fresh links and downloads only the crops not yet on disk. A failed download is fixed by simply running ingestion again.

Later runs reuse the saved parse, so re-running ingestion spends no LlamaParse credits.

5.2 Chunk (chunking.py)

Each page's markdown is split on its own, at its headings, with LangChain's MarkdownHeaderTextSplitter (all six heading levels, headings kept at the top of their section). A section shorter than 500 characters joins the next one:

def merge_small_sections(sections, min_characters):
    """Join each section shorter than min_characters to the next one.

    The last section of a page has no next one, so it joins the one before it.
    """
    chunks = []
    current = ""
    for section in sections:
        current = f"{current}\n\n{section}" if current else section
        if len(current) >= min_characters:
            chunks.append(current)
            current = ""
    if current and chunks:
        chunks[-1] = f"{chunks[-1]}\n\n{current}"
    elif current:
        chunks.append(current)
    return chunks

Each chunk gets a readable id, page139-2 for the second chunk of page 139, and two metadata fields: page_number (the PDF page) and printed_page_number.

The rules behind this design, and the experiment that justified each one:

• One page per chunk, always. A chunk never spans two pages, so every citation points at exactly one page. This removed the need for multi-page citation lists that the Unstructured chunker would have required.
• No maximum size. A page bounds the chunk. Tables are therefore never cut, and a whole financial statement stays in one chunk with its title.
• No lone headings. The first prototype (notebook 05) packed pages into chunks of up to 3,000 characters and left ## NVIDIA Corporation and Subsidiaries (42 characters) as a chunk of its own, because the heading plus its table exceeded the limit. That near-empty chunk ranked first for an unrelated question, and the share-repurchase table it belonged to dropped to third place. Merging short sections forward fixed both.
• Tables are embedded as raw HTML, not as an LLM-written summary. The first prototype had a vision model describe every chunk containing a table, and embedded the description. Notebook 06 tested embedding the raw HTML instead. The right page still ranked first for every table question, with similar scores, and ingestion no longer needs a model call per table:

The full report produced 377 text chunks:

5.3 Describe the charts (figures.py)

A chart cannot be searched by its pixels with a text embedding model, so each chart is stored by a description written by a vision model, and keeps the path of its image so the answering model can be shown the chart itself later.

Which crops deserve a description? The first plan was to describe every figure. Counting the 202 crops on the full report changed that:

• 100 table crops: skipped. The table's HTML is already in the text chunk, so the image would only add tokens.
• 90 image crops: skipped. 66 of them were icons about 20 to 25 points across (skill icons on pages 35 to 41), and most of the rest were photos, logos, signatures and product shots that add little beyond the page text.
• 12 chart crops: described. LlamaParse also files diagrams such as page 8's architecture diagram as charts.

So only charts are described: 12 vision-model calls instead of 102.

The prompt sends the chart image together with its page's full text as context, and asks for four things:

You are writing a searchable description of an image from page {page_number} of a document.
The text of that page is below, for context. Describe the image itself, not the whole page.

PAGE TEXT:
{page_text}

Cover:
1. What kind of image it is (diagram, chart, photo, ...) and what it shows
2. Every label, name, number and connection you can read in it
3. What the image explains, in a few sentences
4. Questions it could answer, and other words people might search for

DESCRIPTION:

Point 2 makes the description carry the chart's actual values; point 4 adds the vocabulary a user might search with, which a chart's own labels rarely contain. An excerpt of the description of the stock performance graph on page 123:

The image is a line graph/stock performance chart [...] titled "Comparison of 5 Year Cumulative Total Return Among NVIDIA Corporation, the S&P 500 Index, and the Nasdaq 100 Index."

Date	NVIDIA Corporation	S&P 500	Nasdaq 100
1/31/2021	100.00	100.00	100.00
1/25/2026	1,448.75	200.33	206.01

[...] Search terms: NVIDIA stock performance graph, cumulative total return chart, S&P 500 Index, Nasdaq 100 Index, NVDA vs S&P 500, five-year shareholder return [...]

Details that matter in practice:

• DeepSeek-V4.1-Flash is a reasoning model, and its reasoning tokens count toward max_tokens. The limit is set to 3,000 so the visible description is never cut off by hidden reasoning.
• Each description is saved to image_descriptions.json as soon as it is written, so a failed run resumes where it stopped, and re-running ingestion makes no model calls at all.
• Descriptions took 8 to 44 seconds each.

5.4 Embed and store (store.py)

Text chunks are embedded as they are (tables included as HTML); charts are embedded by their description.

Jina's embeddings API accepts OpenAI's request format, so LangChain's OpenAIEmbeddings calls it directly. The small wrapper below adds the one thing Jina needs that a generic client does not send: a task, so that documents and questions are embedded asymmetrically (retrieval.passage vs retrieval.query):

class JinaEmbeddings(Embeddings):
    """Jina through its OpenAI-compatible API. It expects task=retrieval.passage for chunks and task=retrieval.query for questions."""

    def _model(self, task):
        return OpenAIEmbeddings(
            model="jina-embeddings-v5-text-small",
            api_key=os.environ["JINA_API_KEY"],
            base_url="https://api.jina.ai/v1",
            check_embedding_ctx_length=False,  # send the text itself, not OpenAI token ids
            max_retries=config.MODEL_RETRY_ATTEMPTS,
            model_kwargs={"extra_body": {"task": task}},
        )

    def embed_documents(self, texts):
        return self._model("retrieval.passage").embed_documents(texts)

    def embed_query(self, text):
        return self._model("retrieval.query").embed_query(text)

LangChain's own community JinaEmbeddings class could not pass a task and sends every text in one request, so it was not used. A free Jina key embeds at most 100,000 tokens a minute, so documents go in batches of 64, one batch a minute. Re-embedding the whole report (389 documents, about 280,000 tokens) takes about 7 minutes. The Chroma collection uses cosine distance, and each ingestion run replaces its contents.

6. The Query Path: An Agent with One Tool

6.1 Why an agent instead of a fixed retrieve-then-answer chain

A classic RAG chain retrieves for every message and then answers. That fails in three common situations:

• "Hi, what can you help me with?" should not trigger a vector search.
• "What were revenue and net income?" benefits from two searches with different phrasings, for example one for the business overview and one for the income statement.
• A follow-up like "And the year before?" cannot be searched as written; it needs to be rewritten as "fiscal 2025 revenue net income" using the conversation so far.

So generation is agentic. LangChain's create_agent gives DeepSeek-V4.1-Flash a single tool, search_document(query), and the model decides whether to search, what to search for, and whether to search again. DeepSeek often issues two or more searches in parallel in one turn.

def build_agent():
    llm = ChatOpenAI(
        model="deepseek-flash",
        base_url="https://api.deepseek.com",
        api_key=os.environ["DEEPSEEK_API_KEY"],
        temperature=0,
        max_retries=config.MODEL_RETRY_ATTEMPTS,
        metadata={"ls_provider": "deepseek"},  # so LangSmith can price the calls (Section 9)
    )
    return create_agent(
        llm,
        [search_document],
        system_prompt=SYSTEM_PROMPT,
        middleware=[show_retrieved_images],
        name="annual_report_agent",
    )

6.2 The search tool

@tool(response_format="content_and_artifact")
def search_document(query: str):
    """Search the document for the passages, tables and figures most relevant to the query."""
    results = vector_store.similarity_search_with_relevance_scores(query, k=config.RETRIEVED_CHUNK_COUNT)
    results = [(document, score) for document, score in results if score >= config.MIN_RELEVANCE_SCORE]
    if not results:
        return "No relevant passages were found in the document for this query.", []
    ...
    # The model reads text_for_model. The chunks are kept on the tool message as its artifact.
    return text_for_model, chunks

• Each search returns the top 3 chunks with a relevance score from 0 to 1.
• Chunks scoring below 0.3 are dropped, so weak matches (and their images) do not fill the prompt.
• The model receives each result as text under a header such as --- Result 2, page 139: Passage --- or --- Result 1, page 8: Figure description (the figure itself follows) ---, so it always knows which page to cite.
• response_format="content_and_artifact" keeps the full structured chunks (id, score, page numbers, text, image path) on the tool message as an artifact, separate from what the model reads. The API returns these to the UI as the answer's sources.

6.3 Showing the chart itself to the model

This is the step that makes the answering stage multimodal rather than "text about images". A tool result can hold only text, so a retrieved chart reaches the model only as its description. Descriptions are good for finding a chart, but the model should read the chart when it answers.

A before_model middleware runs before each model call, looks at every tool result since the model's last call, and, for each retrieved chart, appends a message containing the image itself:

@before_model
def show_retrieved_images(state, runtime):
    """A tool result can only hold text, so the retrieved figures follow in their own message.

    The model can run several searches at once, so every search result since its last call is checked.
    """
    figures = {}
    for message in reversed(state["messages"]):
        if message.type != "tool":
            break
        for chunk in message.artifact or []:
            if chunk["image_path"]:
                figures[chunk["image_path"]] = chunk  # two searches can return the same figure
    if not figures:
        return None

    content = []
    for chunk in figures.values():
        image = base64.b64encode((config.PROJECT_ROOT / chunk["image_path"]).read_bytes()).decode()
        content.append({"type": "text", "text": f"Figure from page {chunk['page_number']}:"})
        content.append({"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image}"}})
    return {"messages": [HumanMessage(content=content)]}

Images are sent only when a chart is actually retrieved. A question about the income statement carries no images at all, which keeps most prompts small.

6.4 Parallel searches and a shared vector store

Parallel tool calls exposed three bugs, all fixed in generate.py:

1. Chroma failed when opened from several threads at once. The agent runs parallel searches in parallel threads, and each opened the store. Now the store is opened once, when the module is imported, and every search shares it.
2. Only the last search's figures were shown. The middleware originally looked only at the latest tool message. It now walks back over every tool result since the model's last call.
3. A chunk found by two searches was listed twice in the sources. The final result now lists each chunk once, by its id.

6.5 The system prompt

You are an assistant that answers questions about NVIDIA's 2026 Annual Report. When asked who you are
or what you can do, say that you answer questions about NVIDIA's 2026 Annual Report, using its text,
financial tables and charts, and that you cite the pages you use.

Use the search_document tool whenever a question needs information from the report. You may search
again with a different query if the first results are not enough.
Answer questions about the report only from what the tool returns, using its text, tables and figures.
Cite the page of each result you use, for example (page 12).
If the results do not contain the answer, say "I don't have enough information to answer that question
based on NVIDIA's 2026 Annual Report."
Reply directly, without searching, to greetings and to questions that are not about the report.
Earlier turns of the conversation help you understand a follow-up question, but search again for any
facts from the report that you need.

The document's title comes from config.DOCUMENT_TITLE. Before the prompt named the report, "Who are you?" got the vague reply "an assistant that answers questions about a document you've provided". Naming it also made the model use the exact refusal sentence more consistently.

6.6 Conversation memory

Earlier turns are passed to the model as plain user and assistant messages, text only: no search results, no figures. Later model calls stay small, and the system prompt tells the model to search again for any facts rather than trust what an earlier answer said. Only the last 3 turns are kept.

6.7 One question, end to end

sequenceDiagram
    participant UI as React UI
    participant API as FastAPI /ask
    participant AG as Agent (DeepSeek Flash)
    participant T as search_document
    participant DB as Chroma + Jina
    UI->>API: question, conversation_id
    API->>AG: last 3 turns + question
    AG->>T: search("consolidated statements of income revenue net income")
    API-->>UI: event: search
    T->>DB: embed query, top 3 by cosine similarity
    DB-->>T: chunks with relevance scores
    T-->>AG: passages as text, chunks as artifact
    Note over AG: middleware attaches the image of any retrieved chart
    AG-->>API: answer, token by token
    API-->>UI: event: text (many)
    API-->>UI: event: done (answer + sources)

7. The Streaming API (FastAPI)

api.py turns the agent into a chatbot backend. The agent is built once at startup.

7.1 Streaming the agent's work

A question that searches takes several seconds, most of it before the first word of the answer. Showing the searches as they start is what makes the wait feel short, so /ask streams three kinds of events:

event: search
data: {"query": "consolidated statements of income revenue net income"}

event: text
data: {"text": "In fiscal year 2026 (ended January 25, 2026), NVIDIA reported"}

event: done
data: {"conversation_id": "…", "question": "…", "answer": "…", "model": "…", "search_queries": [...], "chunks": [...]}

Inside, stream_answer() runs LangGraph's stable agent.stream(..., stream_mode=["messages", "updates", "values"], version="v2") and turns its three streams into these events: messages gives each new token, updates signals that a model call finished (and which searches it asked for), and the last values snapshot is the final state from which the done result is built. LangChain's newer stream_events(version="v3") was tried and worked, but LangGraph still marks it experimental and warns on every run, so the stable API was kept.

Design decisions:

• One code path. answer_question(), used by the test script, simply runs stream_answer() and keeps the done result. There is no second, non-streaming endpoint.
• The done result replaces the streamed text, so the page always ends on exactly what the server saved in the conversation.
• The model's "lead-in" is discarded. DeepSeek sometimes writes "I'll look that up" before asking for a search (in 3 of 7 live test runs). A search event arrives only once that model call has finished, so the UI clears any text before it.
• Errors arrive as an error event. Once streaming starts the HTTP status is already 200, so a DeepSeek or Jina failure after its retries (both are called through the OpenAI client, so both raise openai.APIError) ends the stream with {"error", "service", "status"}. The UI can then say "The answer model (DeepSeek) is rate-limited right now" rather than a generic failure.

7.2 Conversations

• The server keeps the history; the UI keeps only the conversation id, in a JavaScript variable rather than localStorage, so a reload starts a new conversation.
• History lives in a plain dict in memory, not a LangGraph checkpointer: conversation id → its last 3 question/answer pairs. A used conversation moves to the end of the dict, and beyond 1,000 conversations the least recently used one is dropped.
• A missing or unknown id (for example after a server restart) starts a new conversation instead of failing.
• A question that fails, or whose page disconnects, is not saved into the history.
• The server runs with one worker, because conversations are in one process's memory and Chroma's on-disk store is not safe across processes.

8. The Web UI

The UI is a React app (Vite, TypeScript, Tailwind CSS v4, shadcn/ui), designed for reading a report and checking its evidence rather than for chatting. It is laid out as a single left-aligned transcript, not chat bubbles. Answers are set in a serif (Source Serif 4, "the report's voice") and questions and controls in a sans-serif (Public Sans, "the asker's voice"), tables use tabular figures so columns of numbers line up, and a highlighter yellow is reserved for page citations, page tabs and the loading sweep.

While the agent works, the page shows "Reading your question", then the searches the model started ("Searched for …") and "Reading the report" with a highlighter sweep, and then the answer as it streams in:

When the answer is complete, every "page N" in it becomes a citation mark. Under the answer, a Sources row has one tab per retrieved page; opening a tab (or clicking a citation) shows that page's retrieved excerpt with its printed page number and relevance score. Excerpts render LlamaParse's HTML tables, so the reader sees the actual income statement the numbers came from:

When a search retrieves a chart, its thumbnail appears with the sources and opens full size in a dialog:

Implementation notes:

• fetch reads the event stream directly, since the browser's EventSource cannot send a POST.
• Answers and excerpts are rendered with react-markdown, remark-gfm for tables, and rehype-raw followed by rehype-sanitize: raw HTML is needed for LlamaParse's tables, and sanitizing keeps only safe elements.
• A regular expression turns "page 139" and "pages 23, 167" into citation marks. A mark whose page was retrieved opens that page's excerpt; a citation to a page that was not retrieved stays a plain mark.
• A page often contributes several chunks to one answer (page 123 gave three: two text chunks and its chart), so each page's excerpt shows all its retrieved chunks in reading order, using the number in each chunk's id, with figure descriptions last.
• Errors are written for the person asking: a rate limit says which service is rate-limited and to wait a few seconds; a stream cut off before done says the answer was incomplete; each failed question has an "Ask again" button.

In development, Vite's dev server forwards /ask, /health and the image route to the API; in production, FastAPI serves the built UI itself.

9. Observability with LangSmith

I compared LangSmith and Pydantic Logfire on their free plans. Logfire sees the whole server (every HTTP request and every DeepSeek and Jina call) and keeps data longer, but needs its own package and setup code. LangSmith was chosen because it is built for LangChain agents, needs no new package and no code, only three settings in .env (LANGSMITH_TRACING, LANGSMITH_API_KEY, LANGSMITH_PROJECT), and also offers datasets and experiments for evaluation later.

• One question is one trace, named annual_report_agent, holding every model call with its messages, chart images, token counts and cost, and every search_document call with its query and returned chunks.
• Threads. The API passes its conversation id as thread_id metadata, so the questions of one conversation are grouped in the Threads view.
• Cost needed a fix. Traces first showed token counts but no cost. LangSmith's price table lists deepseek-flash under the provider deepseek, but ChatOpenAI labels its calls openai. Passing metadata={"ls_provider": "deepseek"} to the model overrides the label (LangChain applies a model's own metadata after its defaults), and costs appeared.

A sample traced question (one search, two model calls) used 2,273 input tokens, 768 of them served from DeepSeek's cache, and 149 output tokens, for $0.00032. That matches the price table worked out by hand. In the first tracing test, a question and its follow-up "And the year before?" were grouped into one thread; the first question's trace alone held 11 runs (chain, LLM and tool) and 4,669 tokens.

10. Evaluation and Results

10.1 Test questions

generate.py contains six test questions, each chosen to exercise a different part of the system, with its expected answer and page written beside it. All six are answered correctly by the final configuration (DeepSeek-V4.1-Flash for answers, Jina v5 for embeddings), with page citations:

Retrieval: for 4 of the 5 document questions, the expected page ranked first. The exception, the revenue question, retrieved page 23 (the business overview, which states revenue but not net income) first and page 139 second, and the answer combined both.

Follow-ups through the API: a revenue question, then "And the year before?" (the model searched for fiscal 2025 and answered 72,880 million net income, as on page 139), then "Is there a chart comparing that with the stock market?" (it found the page 123 graph, whose image the API served to the UI).

This is a small, hand-checked test set rather than a benchmark. It was designed so that each question can only be answered if a specific capability works: table extraction, checkmark parsing, diagram understanding, chart reading, refusal and conversational follow-up. A proper evaluation suite is listed under future work.

10.2 Relevance scores and the cutoff

The MIN_RELEVANCE_SCORE of 0.3 sits cleanly between noise and real answers, so it removes junk from the prompt. But it cannot detect unanswerable questions: the smartphone question scores 0.66, because the report talks a lot about revenue and devices. Refusal is the system prompt's job, and it worked. A cutoff above about 0.35 would start dropping the director question's correct chunks.

10.3 Latency

Streaming the answer text saves little, because DeepSeek spends most of the time reasoning (its reasoning is not shown) and then writes quickly. Showing the searches is the bigger gain for perceived speed.

10.4 Cost

Because the parse and the descriptions are saved, re-running ingestion costs only the embedding step.

11. The Evaluation Suite

Section 10's six questions show that each capability works once. To measure how well the system works, I built an evaluation suite with deepeval (version 4.3.2). It tests the retriever, the generator and the whole pipeline separately, and adds one behaviour check: can the agent be led off topic? Since this is a personal project, the suite is deliberately small: two golden sets, one run of each metric, every score computed by deepeval, and paid judges used sparingly.

11.1 Design

Golden sets written from the PDF. The main set has 15 questions with short reference answers: 5 on text, 5 on tables, 3 on charts, and 2 whose answer spans two pages.

• The answers come from the PDF itself, not from LlamaParse's output, so a parsing error cannot slip into the references.
• None repeats a Section 10 question.
• Each is worded the way a reader would ask rather than copied from the page, so the retriever cannot simply match words.
• A script checked every number in the text and table answers against the PDF's text layer, and chart values were read off the page images.

A second set holds 8 attempts to lead the agent off topic (Section 11.5).

Judges matched to the job.

Flash judges the retriever because stage 1 has no generated answer for it to grade. The generator's answers, which Flash writes, are graded by a different and larger model, so the answering model never grades its own work. The last two use a decision model (Section 11.4).

Layout.

• evals/goldens/ holds the golden sets.
• evals/notebooks/ holds the notebooks that run them, with the outputs of their latest run.
• evals/findings/ holds one write-up per experiment, committed together with the notebook outputs it describes.

Stage 2 saves the agent's answers and the chunks behind them, so stage 3, or another judge, can score the same answers without running the agent again.

11.2 Stage 1: the retriever alone

Each golden question goes straight to the agent's own search_document tool as a tool call, so it gets exactly the chunks the agent would: the top 3, with scores below 0.3 dropped. No answering model is involved. deepeval then scores the chunks with two metrics:

• Contextual Recall: can each sentence of the reference answer be found in the retrieved chunks? The score is the share that can.
• Contextual Precision: are the chunks that help ranked above those that do not? A relevant chunk alone at rank 1 scores 1.00, at rank 2 0.50, and at rank 3 0.33.

Both metrics require an answer field but never read it, so each test case carries the placeholder "generator not evaluated".

What it showed:

• The expected page was retrieved for 9 of 15 questions, and every miss was a ranking failure. Reading the vector store directly showed each missed answer in a stored chunk, ranked between 4th and 42nd.
  • Four of the six were table chunks, outranked by prose or other tables on the same subject. The director pay table lost to executive pay pages, and the revenue-by-market table lost to pages with growth rates but no dollar figures.
  • One was a wording gap: "buy back its own shares" against the report's "share repurchase authorization".
• It found two parse problems where content runs over a page break.
  • LlamaParse merged page 44's Audit Committee box into page 43's Lead Director table, and the sentence "In Fiscal 2026, the AC met four times" is in no chunk at all.
  • Note 7's table on pages 153 and 154 was not merged. Its second half starts with only a running header and the date "Jan 26, 2025", with nothing saying what the table is.
• Showing the judge the chart images changed no score. The first run gave the judge each chart as its description only, while the answering model also sees the image. A re-run passed the images too, through deepeval's MLLMImage and its LocalModel, since its DeepSeekModel sends text only. Only 2 of the 15 questions retrieved a chart, and both already scored 1.00.
• The judge is not perfectly stable. Between the two runs, one question's precision moved from 0.33 to 0.00 with the same three chunks. One run counted a stockholder proposal's "more than 35,000 people" as relevant to the workforce question; the other did not.

11.3 Stage 2: the generator

Here the real agent answers each of the 15 questions as the chatbot would, writing its own searches. Each test case holds the question, the agent's answer, and every chunk from all of its searches. DeepSeek-V4-Pro reads text only, so charts reached it as their descriptions.

• Faithfulness: the judge lists the answer's claims and the context's facts, then gives each claim a verdict.
• Answer Relevancy: the judge lists the answer's statements and decides whether each one addresses the question.

What it showed:

• A perfect Faithfulness score is weaker than it looks. By default, deepeval fails a claim only when the context directly contradicts it. A claim the context does not support is "borderline", and borderline passes.
  • Five answers contained figures the model worked out itself, and all passed: compute's "83.8%" of Data Center revenue, China revenue "down about 21%", a prior-year total of $43.2 billion, and two more.
  • All five were arithmetically correct, but a wrong one would have passed too. Setting penalize_ambiguous_claims=True makes borderline claims fail.
• The agent retrieves far better than stage 1 suggests. Its own searches reached the expected page for 14 of 15 questions, against 9 of 15 for the raw questions. It rephrases into the report's terms ("share repurchase authorization remaining amount", "non-employee director compensation fiscal 2026") and often searches twice.
• A missing fact sets off a search loop. For the committee question the agent searched 75 times, guessing phrasings such as "AC met six times" for the sentence the parser lost. It then answered correctly that the text does not state it. Nothing limits the number of searches. The same answer cited page 43 for the Audit Committee's chair, where the parser had put page 44's box, so a reader would not find it on that page of the PDF.
• Answer Relevancy marks down extra context, not wrong answers. Each of the six answers below 1.00 added related facts beyond the question, such as the turnover rate in an answer about headcount.
• The run cost about $0.48, far more than any other run. Most of it went to the Pro judge, whose thinking is billed as output. DeepSeek's balance updates several minutes after the calls, so the cost became clear only afterwards. Running one or two questions first and waiting for the bill would have caught it.

11.4 Stage 3: correctness, judged by a decision model

Neither earlier stage checks whether an answer is right. Stage 3 compares each of stage 2's saved answers with its reference answer, so it runs no agent and makes no DeepSeek call.

The judge is Liquid AI's d1.1, a decision model. It writes no text. Given the test case and a set of bounded questions (yes or no, a choice among options, or a score on an ordered scale), it returns a probability for each possible answer, all in one request. deepeval calls it through its JevEval metric and its TypeSafeModel, pointed at Liquid's decision API (https://api.liquid.ai/decisions) through the TypeSafe SDK.

The "Correctness" metric asks three questions:

How the answers become a score:

• A yes-or-no question's value is d1's probability of "yes".
• The score question's value is d1's expected level on the scale, from 0 for "contradicts it" to 1 for "fully matches". So an answer d1 sees as mostly right, with some chance of fully matching, lands a little above 0.67.
• The metric is the weighted mean, so stating something wrong costs twice as much as leaving something out.

What it showed:

• No answer was confidently judged wrong. Nine answers were judged most likely to fully match their reference, and six to be mostly right. The contradiction question passed with 0.81 or more for 13 of the 15.
• Three low scores come from the reference answers. For three table questions, the answer gives exactly what was asked, but the reference holds more: fiscal 2025 figures, every fee line, every country. The references were written for the retriever stage, to cover everything the page says. Without these three, the tables' mean would be 0.93.
• The real gaps trace back to stages 1 and 2:
  • the other executives' pay mix, in a chunk the agent did not retrieve;
  • the Audit Committee's meeting count, lost by the parser;
  • a cash total taken from the balance sheet rather than from Note 7.

d1 judged all 15 answers in 9 seconds.

11.5 Behaviour: does the agent stay on topic?

The system prompt (Section 6.5) tells the agent to refuse anything that is not about the report. Eight tricks tried to get around it. d1.1 judged each answer with its own "Stays on topic" metric:

• a choice question: did the answer decline, give, or partly give the off-topic content?
• a yes-or-no question: is everything in the answer about the report or about what the assistant can do?
• for the two mixed questions, a yes-or-no question on whether it answered the report part.

The metric ran in strict mode, so it scores 1 only when every question passes. The agent was also stopped after 10 searches, so no trick could start a loop like stage 2's.

The agent stayed on topic for all 8. The six outright refusals came without a search, and d1 gave every passing answer a probability of 0.90 or more. The caveat: d1 was never shown an answer that went off topic. So this shows d1 agrees when the agent behaves, not yet that it catches the agent misbehaving.

11.6 How the retriever's score could be improved

Stage 1 is the weakest result. The diagnosis points to ranking, not missing content: every missed answer was in the store, within the top 50 results.

• Rerank a wider candidate list. Fetch the top 30 to 50 chunks and let a cross-encoder reranker (Jina offers one) reorder them before keeping 3. This targets the failure directly. The price is many more Jina tokens per search, since the reranker reads every candidate.
• Give chunks more context. Start each chunk with its section heading or table title, so that page 154's half of Note 7 says what it is. For large tables, a one-line summary would help them compete with prose. This needs a re-embed.
• Add keyword search alongside the vectors (hybrid search with BM25), for exact names such as "PwC", "BlackRock" or "non-employee director".
• Fix the parse where content crosses a page break, such as the page 43 to 44 merge, by re-parsing those pages.

Retrieving 5 chunks instead of 3 would rescue only 2 of the 6 misses, one of them only in part, since the others ranked 13th or lower. And because the agent rewrites its own queries, the chatbot's real retrieval (14 of 15 in stage 2) is already well above stage 1's raw-question score.

11.7 Summary

The suite has clear limits:

• 15 questions and 8 tricks, with one run of each metric;
• judges whose scores move slightly between runs;
• reference answers fuller than some questions ask;
• a text-only judge for stage 2;
• no off-topic answers shown to d1.

Even so, it did what an evaluation should. It found the parse problems at page breaks, the unbounded search loop, a wrong page citation and the weakness of default Faithfulness, none of which the six hand-checked questions had revealed.

12. Challenges, Trade-offs and Lessons

1. The parser is the whole game. Every improvement downstream depended on what the parser could see. The decisive test was not "does it produce text" but specific checks: duplicate lines, split columns, every number against the PDF text layer, checkmarks, diagrams.

2. Not everything deserves to be embedded. Describing all 102 non-table crops would have cost 8.5 times as many vision calls and filled the store with descriptions of icons and signatures. Counting the crops first showed that 66 were icons. The trade-off: two real diagrams filed as image crops are dropped too (page 4's DSX diagram, whose labels are all in the page text, and page 3's "five-layer stack", whose lists appear nowhere else). Keeping one is a one-line exception in pick_charts().

3. Small chunks can hurt more than big ones. A 42-character heading chunk ranked first for an unrelated question. Merging short sections forward, and letting the page bound the chunk instead of a character limit, fixed it without cutting any table.

4. Save the expensive result first. A dropped image download once lost a finished parse, because the result was saved only after the images. The parse is now saved the moment it finishes, and image downloads are resumable.

5. Free tiers shaped the architecture.

• Generation started on Groq, whose free tier allows 7,000 input tokens a minute. A question that retrieves the three largest chunks (about 8,100 tokens, all financial tables) failed with a 413 error. Generation moved to DeepSeek.
• Embeddings started on Nemotron via OpenRouter, whose free models allow only 50 requests a day. They moved to Jina, whose free key allows 100,000 tokens a minute, which in turn is why ingestion embeds one batch of 64 documents a minute.

6. Parallel tool calls are a concurrency problem. Once the model searched in parallel, Chroma failed when several threads opened it at once, and figures from all but the last search went missing. Both are fixed, but they would not have shown up with sequential searches.

7. Reasoning models need room. DeepSeek-V4.1-Flash's hidden reasoning counts toward max_tokens, so the description limit is set generously. And the model's occasional lead-in text before a search had to be handled explicitly in the stream.

8. Observability needs its labels right. LangSmith showed no cost until the provider label matched its price table.

13. Limitations

• One document. The system answers questions about a single, pre-ingested report. Multiple companies, periods or reports are out of scope for now.
• The model adds its own arithmetic. It sometimes adds figures the report does not state, for example "14.5x the initial investment". In one test it called the ending values of 200.33 (on a $100 start) a "~1,449% versus ~200%" cumulative return, which is wrong: the returns are about 1,349% and 100%.
• Refusal wording varies. The meaning has been right every time, but DeepSeek has at least once paraphrased the refusal sentence ("does not report a separate revenue figure for smartphone sales"), so code that looks for the exact sentence could miss it.
• The relevance cutoff cannot detect unanswerable questions (Section 10.2); that depends on the prompt.
• Searches are unbounded. When a fact is missing from the store, the agent keeps searching: 75 searches for one question in the evaluation (Section 11.3).
• Two diagrams are not searchable as images (Section 12, point 2).
• Content that crosses a page break is parsed inconsistently. One box was merged into the previous page's table and a sentence lost, while another table was split into two chunks, the second without a title (Section 11.2).
• Conversations live in memory and are lost when the server restarts; the server must run with one worker.
• Hosted dependencies. Parsing, descriptions, embeddings and answers all depend on external APIs. Jina's free tokens are a one-time grant, and DeepSeek is paid per token, though at about $0.0003 a question.
• Privacy. The LlamaParse account used opted into data sharing for model training; private documents should be parsed with an account without that option.
• A small evaluation. 15 questions and 8 tricks, each metric run once, with judges whose scores vary slightly between runs (Section 11.7).

14. Future Work

1. Better retrieval for tables: a reranker over a wider candidate list, section headings at the start of each chunk, and keyword search alongside the vectors (Section 11.6).
2. A fast "System 1" security layer. A small local classifier in front of the agent that sorts each message into report question, greeting, off-topic or manipulation attempt in one forward pass (tens of milliseconds), answering the last three with fixed replies without calling DeepSeek, and only when its confidence is above a threshold, since wrongly refusing a real question is worse than letting an off-topic one through. Candidates to compare: GLiClass, a DeBERTa zero-shot classifier, a small open LLM read as a single-pass classifier, and dedicated prompt-injection classifiers.
3. A limit on the agent's searches, so that a fact missing from the store cannot start a loop.
4. A stricter evaluation:
   • Faithfulness with penalize_ambiguous_claims=True, so unsupported figures count.
   • Reference answers holding only what each question asks.
   • A control set of off-topic answers, to check that d1 catches failures.
   • Follow-up questions within a conversation.

15. Setup and Usage

Prerequisites

• Python 3.12 or 3.13
• Node.js (developed with Node 22) for the web UI
• API keys for LlamaParse, DeepSeek and Jina (and optionally LangSmith, and Liquid AI for two of the evaluations)

Installation

git clone https://github.com/ChidambaraRaju/reportlens.git
cd reportlens
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

cd frontend
npm install
npm run build
cd ..

Create .env in the project root:

LLAMA_CLOUD_API_KEY=...
DEEPSEEK_API_KEY=...
JINA_API_KEY=...

# Optional: trace every question in LangSmith
LANGSMITH_TRACING=true
LANGSMITH_API_KEY=...
LANGSMITH_PROJECT=multimodal-rag-pipeline

# Optional: the correctness and off-topic evaluations, judged by Liquid AI's d1.1
LIQUID_API_KEY=...

Usage

# 1. Ingest the PDF set in config.py (the first run parses with LlamaParse, about 10 credits a page)
python -m multimodal_rag.ingest

# 2. Try the test questions in the terminal
python -m multimodal_rag.generate

# 3. Start the chatbot, then open http://127.0.0.1:8000/
uvicorn multimodal_rag.api:app

From your own code:

from multimodal_rag.generate import build_agent, stream_answer

agent = build_agent()
for kind, data in stream_answer(agent, "What were NVIDIA's revenue and net income in fiscal year 2026?"):
    if kind == "search":
        print(f"\n[searching: {data['query']}]")
    elif kind == "text":
        print(data["text"], end="", flush=True)

To use another PDF, point DOCUMENT_PATH in config.py at it, set DOCUMENT_TITLE, and update the title, page count and example questions in frontend/src/lib/document.ts.

Configuration

Every setting lives in multimodal_rag/config.py:

16. Project Structure

reportlens/
├── multimodal_rag/
│   ├── config.py        # every setting; loads .env
│   ├── ingest.py        # runs the four ingestion steps in order
│   ├── parse.py         # 1. LlamaParse, saved once, plus the cropped images
│   ├── chunking.py      # 2. split each page at its headings
│   ├── figures.py       # 3. pick the charts and describe them
│   ├── store.py         # 4. Chroma store and Jina embeddings
│   ├── generate.py      # the agent, its search tool and image middleware, streaming
│   └── api.py           # FastAPI backend: /ask (SSE), images, health, the built UI
├── frontend/
│   └── src/
│       ├── App.tsx              # the conversation
│       ├── lib/api.ts           # POST /ask and reading the event stream
│       ├── lib/document.ts      # document title, page count, example questions
│       └── components/          # question box, exchange, sources, markdown with citations
├── notebooks/
│   ├── 03_annual_report_layout.ipynb                  # Unstructured hi_res: the problems
│   ├── 04_llamaparse_test.ipynb                       # LlamaParse on the same pages
│   ├── 05_llamaparse_ingestion.ipynb                  # first prototype: packed chunks, described tables
│   └── 06_heading_chunks_and_image_documents.ipynb    # the approach ingest.py follows
├── evals/
│   ├── goldens/         # 15 report questions with reference answers, and 8 off-topic tricks
│   ├── notebooks/       # retriever, generator, correctness and off-topic evals, with their results
│   └── findings/        # a write-up of each run
├── 2026-Annual-Report-Web.pdf
├── requirements.txt
└── pyproject.toml

17. Conclusion

ReportLens shows that a useful, verifiable chatbot over a complex 175-page PDF does not need an elaborate framework. It needs a parser that sees the page the way a reader does, chunks that respect page and table boundaries, descriptions only for the figures that carry information, and an agent that can search more than once and look at a chart when it finds one. Each of those choices came from a measured experiment: Unstructured against LlamaParse on the hardest pages, packed chunks against heading chunks, described tables against raw HTML, all crops against charts only.

The result answers questions from text, tables, charts and diagrams, cites every page, opens the evidence behind each citation, refuses when the report is silent or the request is off topic, and does so in a few seconds for a fraction of a cent. The evaluation suite measured it. Answers are rarely wrong (correctness 0.80, with no answer confidently judged wrong), the agent resisted every trick to lead it off topic, and the clearest weakness is retrieving tables. That weakness, the unbounded search loop and the model's habit of adding its own arithmetic point to the next steps: reranking and richer chunks for tables, a limit on searches, and a fast classifier in front of the agent.

Repository: https://github.com/ChidambaraRaju/reportlens