FinRAG

1. Overview

This submission presents FinRAG, the first-place winning solution for the 4th AI & Data Science Competition, where we achieved the highest scores in both retrieval and generation tasks. Our system is designed for financial document processing using Retrieval-Augmented Generation (RAG), with a focus on handling structured financial data, including tables, numerical reasoning, and contextual retrieval.

2. Competition Overview

2.1 4th AI & Data Science Competition

The competition aimed to develop a highly effective Retrieval-Augmented Generation (RAG) system capable of processing large-scale financial documents. The competition consisted of two primary tasks:

• Retrieval Task: Rank the most relevant financial documents for each query.
• Generation Task: Generate factually accurate and contextually relevant answers based on the retrieved financial data.

Participants had to build a system that could effectively retrieve and process financial information, including financial statements, earnings reports, and structured tables.

---

2.2 Key Challenges

The competition dataset posed several unique challenges:

Key Technical Challenges:

1. Handling Complex Financial Data: Many queries required processing both textual explanations and numerical data, often found in structured tables.
2. Improving Retrieval Accuracy: The system needed to rank the most relevant financial documents with high precision to maximize nDCG@10.
3. Ensuring High-Quality Answer Generation: The generation model had to understand financial jargon, extract numerical values, and produce precise, fact-based answers.

---

2.3 Results & Leaderboard Rankings

Our system, FinRAG, outperformed all competing teams in both retrieval and generation tasks:

Retrieval Task (Ranked 1st out of all teams)

Generation Task (Ranked 1st out of all teams)

---

3. Solution Architecture: FinRAG

3.1 Overview of FinRAG

FinRAG is an advanced Retrieval-Augmented Generation (RAG) system designed for financial document processing. Our solution incorporates:

• Hybrid Retrieval: Combining Dense Vector Search (Embedding) and Lexical Matching (BM25) for optimal document ranking.
• SQL-Based Table Processing: Extracting structured financial tables and executing automated SQL queries for precise numerical answers.
• LLM-Powered Answer Generation: Utilizing fine-tuned language models to produce factually accurate, context-aware responses.

---

3.2 SQL Preprocessing for Financial Tables

One major challenge in financial document processing was dealing with tables embedded within financial reports. Many financial insights (e.g., revenue trends, debt ratios, stock performance) were hidden within unstructured tabular data rather than plain text.

To address this, we developed a SQL-based preprocessing pipeline:

1. Table Extraction: Convert raw financial tables from PDFs and HTML reports into structured CSV files.
2. SQL Schema Generation: Convert extracted tables into SQL-compatible structures, ensuring correct column formatting.
3. Database Integration: Store preprocessed financial data in a MySQL database for fast structured querying.

This preprocessing enabled precise numerical calculations, reducing LLM hallucination errors when handling financial queries.

---

3.3 Text-to-SQL Query Rewriting

Since financial queries often required numerical operations, we implemented a Text-to-SQL pipeline that translated natural language queries into structured SQL queries.

🔹 Example Query Transformation:

💬 User Query: "When does sales of oil reach the largest value?"

🔄 LLM-Generated SQL Query:

WITH oil_sales AS (
    SELECT 
        year,
        amount_usd_millions
    FROM d8232f0ba
    WHERE product_type = 'oil' AND sales_or_change = 'sales'
),
max_oil_sales AS (
    SELECT 
        year,
        amount_usd_millions
    FROM oil_sales
    WHERE amount_usd_millions = (
        SELECT MAX(amount_usd_millions)
        FROM oil_sales
    )
)
SELECT 
    year,
    amount_usd_millions AS max_oil_sales_amount
FROM max_oil_sales;

✅ Database Result: {'year': 2012, 'max_oil_sales_amount': Decimal('2896.00')}

This approach significantly improved numerical accuracy, reducing calculation errors in financial queries.

---

3.4 Full FinRAG Pipeline

Our final FinRAG pipeline integrates retrieval, SQL processing, and LLM-based generation:

1. Query Processing:
   • Convert user queries into vector representations to query vector database
2. Hybrid Document Retrieval:
   • Use BM25 for keyword matching and Dense Retrieval (embedding search) for semantic understanding.
   • Rank documents using Jina Reranker.
3. SQL Query Execution (If Required):
   • Find Answer Location in Retrieved Documents.
   • If Document Includes Table and/or query requires numerical computations, convert natural language queries into SQL.
   • Execute SQL queries on MySQL database and retrieve results.
4. Final Answer Generation:
   • Use LLM to use original query, ranked documents, and SQL response to formulate well-structured, factually accurate answers.

---

3.5 Web-Based Interactive Chatbot

Video Demonstration

To demonstrate FinRAG’s capabilities, we built a real-time chatbot interface:

• Users submit financial queries via a web interface.
• The system retrieves relevant financial documents and executes SQL queries if necessary.
• The chatbot responds with well-structured financial insights, combining retrieved text + SQL-based numerical calculations.

---

4. Conclusion & Future Work

FinRAG provides a highly effective financial RAG system, demonstrating state-of-the-art retrieval and generation performance. Future work includes:

✅ Expanding to additional financial datasets (e.g., live market data, real-time stock analysis).

✅ Fine-tuning LLMs for financial text comprehension to further reduce hallucinations.

✅ Optimizing SQL parsing techniques to handle even more complex table structures.

---

5. Open-Source Code

The full FinRAG solution is open-source and available on GitHub:

🔗 GitHub Repository