Korean Scientific Literature RAG

TL;DR

This document presents a Retrieval-Augmented Generation (RAG) framework optimized for scientific and technical information using Korean-language academic datasets. Our system integrates state-of-the-art retrieval and generation techniques to enhance accuracy, reliability, and usability in scientific question answering (QA) tasks.

Key contributions include:

• Hybrid Retriever Architecture: Combining BM25 with Kiwi Tokenizer and Sentence Embedding Retriever into an Ensemble Retriever, achieving higher retrieval accuracy.
• LLM Evaluation for Answer Quality: Incorporating GPT-based evaluation metrics to ensure contextual relevance and factual accuracy.
• Optimization of Chunking Methods: Identifying 500-token chunk size with a 50-token overlap as the best configuration for retrieval and generation.
• LLM Selection & Prompt Optimization: Qwen2-7B-Instruct was chosen for balanced performance across retrieval and answer generation tasks.
• Advanced RAG Enhancements: Reranker and HyDE integration to refine retrieval accuracy, with notable improvements in answer generation.

The developed framework provides a robust and scalable solution for scientific literature QA, offering domain-specific reliability and enhanced retrieval precision.

---

1. Introduction

1.1 Research Motivation & Problem Statement

Large Language Models (LLMs) often hallucinate, relying solely on pre-trained data without incorporating external factual references. This is a critical issue for scientific and technical information retrieval, where accuracy and credibility are paramount. To address this, we develop a RAG system specialized for scientific information, integrating structured retrieval with generative AI to ensure high-fidelity, evidence-backed answers.

1.2 Scope & Contributions

This study explores RAG optimizations for scientific datasets, specifically:

1. Evaluating open-source RAG frameworks for performance on Korean-language scientific corpora.
2. Optimizing retrieval models, implementing an Ensemble Retriever (BM25 + Sentence Embedding).
3. Improving retrieval effectiveness via chunking techniques and document structuring.
4. Selecting optimal LLMs for Korean scientific text generation, balancing accuracy, efficiency, and language-specific considerations.
5. Enhancing RAG with advanced techniques, including Reranking and HyDE, for further refinement.

---

2. Development of the RAG Framework

2.1 Dataset Construction

• Primary Data Source: KISTI AIDA platform
• Dataset Composition:
  • Scientific papers (~480K) from 749 journals
  • QA pairs dataset extracted from academic texts
  • Metadata includes titles, authors, abstracts, keywords, full-text sections
• Preprocessing Steps:
  • Tokenization: Kiwi Tokenizer for BM25 retrieval, default tokenizer for Dense Retriever
  • Chunking: Best performance at 500-token chunks with 50-token overlap

2.2 Comparison of RAG Open-Source Frameworks

We evaluated LangChain, LlamaIndex, and Haystack for RAG implementation. Key findings:

• Performance was comparable across frameworks when tested with identical retrievers and document structures.
• LangChain was chosen due to its ease of use, modular pipeline support, and compatibility with various retrieval techniques.

2.3 Retrieval Optimization

2.3.1 Retriever Selection & Ensemble Strategy

We tested three retrieval approaches:

1. BM25 Retriever (Lexical search with Kiwi Tokenizer)
2. Sentence Embedding Retriever (Dense vector search)
3. Ensemble Retriever (BM25 + Sentence Embedding)

Best results were obtained with Ensemble Retriever, combining BM25's robust lexical matching with dense retrieval’s semantic understanding.

2.3.2 Chunking Optimization

• Tested chunk sizes: 250, 500, 1000, 2000
• Optimal setting: 500-token chunks with 50-token overlap

2.4 Answer Generation & LLM Selection

We evaluated five open-source LLMs:

Qwen2-7B-Instruct was selected for its balanced accuracy and efficiency, avoiding the verbosity of Exaone while maintaining high factual accuracy.

2.5 Prompt Optimization

• Compared Korean vs. English prompts
• English prompts were chosen due to better compatibility with LLM pretraining data

---

3. Advanced RAG Enhancements

3.1 Reranker Integration

• Implemented BAAI/bge-reranker-base
• Improved retrieval ranking, leading to +6% LLM Eval score

3.2 HyDE (Hypothetical Document Embeddings)

• HyDE generated synthetic passages to enhance retrieval context.
• Improved retrieval performance, but increased latency by 4x.
• Optional based on response time constraints.

---

4. Key Results & Findings

4.1 Summary of Best Practices

4.2 Contributions to Scientific Information Retrieval

• Domain-specific RAG optimization for scientific literature
• Hybrid retriever strategy for enhanced accuracy
• LLM-based evaluation framework for better answer quality assessment

---

5. Future Research Directions

• Expanding RAG for other specialized domains (e.g., medicine, legal)
• FineSurE-based evaluation (Faithfulness, Completeness, Conciseness)
• Integration of KONI-Llama models for further Korean-language fine-tuning

---

6. References

• Xiaonan Li, Changtai Zhu, Linyang Li, Zhangyue Yin, Tianxiang Sun, Xipeng Qiu,“LLatrieval: LLM-Verified Retrieval for Verifiable Geneartion”, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2024), 2024.
• Luyu Gao, Xueguang Ma, Jimmy Lin, Jamie Callan, "Precise Zero-Shot Dense Retrieval without Relevance Labels," Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023), 2023.
• Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, Saab Mansour, "FineSurE: Fine-grained Summarization Evaluation using LLMs", Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL2024), 2024.

---

7. GitHub & Open Source Code

🔗 GitHub Repository