Abstract

Large Language Models (LLMs) have revolutionized the way we interact with textual data. We used to search through long documents but now we can have AI do that for us and just ask a simple query to an LLM. However, they often generate responses without clear attribution, leading to concerns about factual accuracy and trustworthiness. This project presents a Retrieval-Augmented Generation (RAG) model that integrates source citation into LLM responses. Implemented in Streamlit with LangChain, the model dynamically retrieves information from a collection of text files and accurately cites sources. By leveraging vector stores and embedding-based search, our approach ensures more transparent and reliable AI-generated responses. The system is designed to be adaptable, enabling users to query any uploaded text dataset with full source traceability.

Introduction

Language models like GPT-4 and Mistral are powerful tools for text generation, but their lack of explicit source citation makes them less than ideal for fact-based applications like research and legal writing. The goal of this project is to bridge this gap by implementing a custom RAG system that enhances LLM responses with automated source attribution. Our Streamlit-based web application processes large text datasets, enables real-time querying, and retrieves supporting document references to ensure factual grounding.

This project was initially tested with Grimm’s Fairy Tales, but its architecture allows for expansion to any collection of text files. The primary components include:

• Document Processing & Chunking – Converts raw text into searchable vectorized embeddings.
• Vector Database Storage – Enables efficient semantic search using FAISS.
• LLM Query & Response Generation – Uses LangChain to retrieve and generate contextually relevant responses.
• Source Attribution – Extracts the original source file and relevant sections for citation.

Methodology

The workflow of our RAG-based citation system consists of several key steps:

1. Data Preprocessing & Ingestion

• Text documents are loaded dynamically from a data directory.
• Documents are split into chunks (e.g., 500 tokens per chunk) to improve searchability.
• Each chunk is embedded into a vector representation using OpenAI, Hugging Face, or local embedding models.
• The resulting vectors are stored in a FAISS vector database for fast retrieval.

2. Query Processing

• User queries are vectorized in the same embedding space.
• A semantic similarity search retrieves the most relevant document chunks.
• The top N retrieved results are fed into the LLM as context.

3. Response Generation & Source Citation

• The LLM generates an answer using the retrieved document fragments.
• Each response includes an inline citation, listing the source file name and section.
• The system is designed to return grounded and verifiable responses.

4. Web Application (Streamlit)

• A chat-like UI enables users to input queries and receive context-aware responses.
• Message history is displayed for better interaction.
• Users can upload new text files, and the system automatically updates the vector database.

Experiments

To evaluate the effectiveness of our LLM-powered RAG model, we conducted a series of experiments:

1. Baseline Model vs. RAG Model

• A standard LLM without retrieval was tested, generating hallucinated answers.
• The RAG-enhanced model significantly improved response accuracy and citation reliability.

2. Retrieval Effectiveness

• We measured top-k retrieval accuracy using Mean Reciprocal Rank (MRR).
• Results indicated that embedding-based retrieval significantly outperforms simple keyword search.

3. Human Evaluation

• Responses were evaluated by users for factual accuracy, readability, and citation clarity.
• Users preferred RAG responses due to their verifiability.

4. Performance Testing

• Benchmarked response time under varying document sizes. (Depending on the hardware, it was usually less than a minute for recreating the vector store and a few seconds for prompt responses from the model.)
• Optimized query execution using FAISS indexing and batch processing.

Results

The implementation of Retrieval-Augmented Generation (RAG) significantly improved the reliability and transparency of LLM-generated responses. Compared to standard language models, this approach reduced hallucinations by grounding answers in real sources. Users found that responses with citations were more trustworthy and verifiable, as they could trace statements back to the original text.

Key observations include:

• Enhanced Citation Accuracy: Responses included clear references to the correct source file and relevant sections, ensuring transparency.
• Improved User Confidence: Users expressed a higher level of trust in responses when citations were present.
• Effective Information Retrieval: The system was able to retrieve relevant document fragments efficiently, providing contextually appropriate answers.
• Scalability & Performance: Even with multiple documents, retrieval remained fast and responsive, making the system practical for real-world applications.

This project demonstrates that RAG-based models can enhance AI-generated content by making responses more transparent and source-aware, offering a solid foundation for research, education, and fact-based AI applications.

Conclusion

This project successfully integrates source citation into LLM-generated responses using Retrieval-Augmented Generation (RAG). Our Streamlit-based web application enables users to query any document collection, ensuring traceability and trustworthiness in AI responses. The combination of vector-based retrieval and LLM generation significantly reduces hallucinations while maintaining fast query execution.

Future work includes:

• Expanding document types (PDFs, research papers).
• Enhancing citation formatting (e.g., MLA, APA).
• Scaling to multimodal sources (images, videos, and web pages).

By improving transparency in AI-generated content, this project lays the foundation for fact-grounded, citation-aware AI applications.

How to Run the Model

# Download the repo
git clone https://github.com/DomenickD/LLM_Source_Citation

# Change directory
cd LLm_Citation_App

# Install dependencies
pip install -r requirements.txt

# Run the Streamlit app
streamlit run app.py