This study presents a novel hybrid Retrieval-Augmented Generation (RAG) system integrating AI assistants with local vector storage for enhanced document question answering in industrial environments. The system combines dense semantic search using multilingual embeddings with sparse BM25 keyword retrieval, augmented by temporal filtering for date-aware document retrieval. The architecture employs a privacy-preserving local-cloud approach where sensitive documents are stored locally in a Qdrant vector database while leveraging cloud-based large language models for reasoning. Evaluation on 500 publicly available Indonesian government reports demonstrates that the hybrid approach achieves 95% precision, 89% recall, and 92% F1-score, significantly outperforming single-method baselines (vector-only: 82%, BM25-only: 81%, traditional RAG: 86%). The superior performance is attributed to complementary strengths of semantic understanding and exact keyword matching, combined with strict temporal filtering that eliminates irrelevant results. The temporal intelligence module parses Indonesian and English date expressions with 98% accuracy, enabling precise date-aware retrieval for time-sensitive queries. Concurrent processing achieves 2.9× speedup with 3.2-second average response time and 99.7% uptime over 30 days. This architecture provides an effective solution for intelligent document assistants balancing AI capabilities, data privacy, and temporal awareness, particularly suitable for regulated industries requiring on-premise deployment with multilingual support.
Modern industrial organizations generate large volumes of documents, including daily operational reports, departmental procedures, technical documents, and equipment manuals. The exponential growth of these document repositories creates significant challenges for information retrieval and operational decision-making. Traditional keyword-based search systems often fail to understand semantic context and struggle with temporal queries, resulting in poor retrieval accuracy and user frustration. This challenge is especially acute in industrial environments, where timely access to accurate, date-specific information (such as daily operational reports or the latest maintenance procedures) directly impacts operational efficiency, safety, and regulatory compliance.
The emergence of Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems offers promising solutions for intelligent document question answering. However, existing approaches face three critical limitations: (1) single retrieval methods miss relevant documents, (2) lack of temporal awareness reduces precision for time-sensitive queries, and (3) cloud-only architectures raise data privacy concerns for organizations handling sensitive information.
Classical keyword-based search systems like BM25 (Robertson & Zaragoza, 2009) provide exact term matching but lack semantic understanding. While efficient, they struggle with synonym variations, paraphrasing, and conceptual queries common in natural language questions. Recent advances in dense retrieval using neural embeddings (Reimers & Gurevych, 2019) enable semantic similarity matching through vector representations. However, dense-only approaches may miss documents with exact keyword matches that lack semantic similarity scores.
Lewis et al. (2020) introduced RAG systems that combine retrieval with language generation, significantly improving question-answering accuracy. Subsequent research has explored various retrieval strategies, but most focus on single retrieval methods without temporal awareness. Recent studies suggest that combining dense and sparse retrieval improves coverage and precision. However, existing hybrid systems lack sophisticated temporal filtering mechanisms essential for date-sensitive document collections. Date-aware search remains an understudied area, particularly for multilingual contexts. Most systems rely on simple date matching without intelligent query interpretation or multi-strategy filtering. Cloud-based LLM services raise concerns for organizations with strict data privacy requirements. Hybrid local-cloud architectures offer potential solutions but lack comprehensive evaluation in production environments.
Despite significant progress in RAG systems, four critical gaps remain. First, existing RAG systems lack sophisticated date parsing and filtering mechanisms, leading to irrelevant results when users query time-sensitive information such as "Q1 2024 report" or "latest financial statement." Second, while some systems combine dense and sparse methods, few implement strict temporal filtering across both retrieval paths or provide multi-strategy query analysis. Third, organizations must choose between cloud-based AI capabilities or on-premise data security, without architectures that balance both requirements effectively. Fourth, Indonesian-English mixed document environments require sophisticated date parsing that handles multiple formats, date ranges, and language-specific expressions.
This paper addresses these gaps by developing and evaluating a novel hybrid RAG system with temporal intelligence and privacy-preserving architecture for industrial environments. Specific objectives are: (1) design a hybrid retrieval architecture that combines dense semantic search (Qdrant vector database) with sparse keyword matching (BM25) augmented by strict temporal filtering, (2) develop a multi-strategy date intelligence module that parses Indonesian and English date expressions with high accuracy, (3) implement a privacy-preserving local-cloud architecture where sensitive documents remain on-premise while leveraging cloud-based LLM reasoning, and (4) evaluate system performance on internal company documents (operational reports, departmental procedures, and technical documents) in a real production environment and compare against single-method baselines.
This study makes four significant contributions to the field of intelligent document question answering. First, a comprehensive RAG system integrating dense vector search, sparse BM25 retrieval, and AI assistant reasoning with strict temporal filtering across all retrieval paths. Second, a multi-strategy date parsing system supporting Indonesian and English languages with 98% accuracy across diverse date formats, ranges, and expressions. Third, a local-cloud architecture that maintains data privacy through on-premise vector storage while leveraging cloud LLM capabilities for reasoning, suitable for regulated industries. Fourth, rigorous evaluation on 500 internal company documents (operational reports, departmental procedures, and technical documents) demonstrates 95% precision, 89% recall, and 92% F1-score, significantly outperforming single-method baselines by 6-11% in F1-score with practical deployment validation in industrial production environments.
The remainder of this paper is structured as follows: Section 2 details the system architecture, algorithms, and implementation. Section 3 presents experimental setup, datasets, and evaluation metrics. Section 4 reports comprehensive results including baseline comparisons and ablation studies. Section 5 discusses findings, practical implications, and limitations. Section 6 concludes with key takeaways and future research directions.
2.1 System Architecture
The proposed system uses a modular microservices-based architecture with seven integrated main components. The frontend layer is built using Next.js framework providing a responsive and interactive user interface. The backend API is developed with Python FastAPI framework handling request processing, business logic, and inter-component orchestration. The vector database uses Qdrant deployed in Docker containers to store document embeddings with 1024 dimensions from the BGE-M3 model. The BGE-M3 embedding model (BAAI General Embedding) runs locally through Ollama to generate vector representations of document text and queries, ensuring data privacy by not sending document content to external services. The large language model service uses OpenAI Assistants API with GPT-4o model for reasoning and response generation based on retrieved documents. BM25 index is implemented for sparse keyword-based retrieval as a complement to dense semantic search. PostgreSQL 15 database is used to store document metadata, chat history, user information, and BM25 corpus in JSONL format. This architecture enables clear separation between data storage (on-premise), embedding generation (local), and reasoning (cloud), creating an optimal balance between privacy, performance, and AI capabilities.
Figure 1: Overall System Architecture shows the interaction between components with data flow from user query to response generation through the hybrid retrieval pipeline.
2.2 Document Processing Pipeline
The upload and indexing process is designed to handle various file formats efficiently while maintaining the original document structure and context. The pipeline starts with file validation that checks format, size, and integrity of user-uploaded documents. The system supports six main document formats with specific loaders for each format (Table 1): PDF uses a combination of PyMuPDF and pdfplumber for text extraction with layout preservation and table extraction, DOCX uses UnstructuredWordDocumentLoader to maintain heading structure and formatting, TXT with TextLoader that automatically detects encoding, CSV with CSVLoader for tabular data, JSON with native Python parser for nested structures, and XLSX with pandas for multi-sheet Excel processing.
After successful loading, the system performs automatic metadata extraction including document date and organizational unit from filename or document content using pattern matching and date parsing. The next critical stage is text chunking using RecursiveCharacterTextSplitter with parameters chunk_size 1000 characters and chunk_overlap 200 characters. This chunking strategy ensures each chunk is large enough for meaningful semantic context yet small enough for high-precision retrieval, with 20% overlap to maintain context continuity between chunks.
Each chunk is then processed for embedding generation using the BGE-M3 model that produces 1024-dimensional vectors representing the semantic meaning of the text. Embedding vectors are stored in Qdrant vector database with document metadata as payload for filtering. In parallel, raw text from all chunks is also persisted in JSONL format to build the BM25 corpus that will be loaded into memory at system startup. This pipeline is optimized with concurrent processing using ThreadPoolExecutor to handle multiple file uploads simultaneously (explained in section 2.6).
Figure 2: Upload Processing Flow illustrates the complete sequence diagram from file upload to dual indexing (vector + BM25).
Table 1: Supported Document Formats
Format
Loader
Special Features
PDF
PyMuPDF + pdfplumber
Table extraction, layout preservation
DOCX
UnstructuredWordDocumentLoader
Structure preservation
TXT
TextLoader
Automatic encoding detection
CSV
CSVLoader
Structured data handling
JSON
json.load
Nested structure support
XLSX
pandas
Multi-sheet processing
2.3 Date Intelligence Module
The temporal intelligence module is a key component that distinguishes this system from conventional RAG implementations. This module consists of two main submodules: date parser and query strategy analyzer. The date parser is responsible for extracting temporal information from user queries in Indonesian and English using a multi-layer approach. The first layer uses the dateparser library with locale settings for Indonesian and English that can handle various natural language formats such as "yesterday," "last week," or "minggu lalu." The second layer applies pattern matching with regular expressions for standard formats including Indonesian pattern "(day) (month_name) (year)" like "15 January 2024," English pattern "(month_name) (day), (year)" like "January 15, 2024," and ISO format "YYYY-MM-DD."
The advanced feature of this parser is the ability to handle date ranges and multiple dates. For date ranges, the parser can interpret expressions like "1 to 5 March" or "January 1-5" and expand them into individual date lists. For multiple dates, the parser recognizes enumeration patterns like "1, 2, and 3 March" or "January 1, 2, and 3" then extracts each date separately. All parsed dates are normalized to YYYY-MM-DD format for downstream processing consistency.
The query strategy analyzer classifies user intent into five filtering strategies based on query content and structure (Table 2). The "no_filter" strategy is applied for queries about SOPs, manuals, or procedures that are timeless where revision date is not the main priority. The "latest" strategy is selected when the query contains keywords "latest," "newest," or "most recent" indicating the user only wants the most recent document. The "explicit" strategy is used when the parser successfully extracts specific dates from the query, ensuring only documents with exact date matches are retrieved. The "month_range" strategy is applied for comparative queries mentioning specific months like "compare January and February," the system will retrieve all documents within those month ranges. The "all_available" strategy serves as a fallback for general comparative queries without specific time constraints.
The combination of high date parsing accuracy and intelligent strategy selection enables the system to provide highly relevant retrieval results for time-sensitive queries, eliminating noise from documents of different periods that often becomes a problem in traditional RAG systems.
Figure 3: Date Parsing and Query Analysis Flow shows the decision tree from raw query to selected strategy with concrete examples.
Table 2: Query Strategy Selection
Query Type
Strategy
Date Filter
SOP/Manual
no_filter
None
"Latest/Newest"
latest
Most recent
Specific date
explicit
Parsed dates
Comparative + month
month_range
Full month
Comparative general
all_available
All DB dates
2.4 Hybrid Retrieval System
The core of this system is the hybrid retrieval architecture that combines the complementary strengths of dense semantic search and sparse keyword matching with strict temporal filtering. The retrieval process is triggered when the OpenAI Assistant calls the retrieve_documents function with the user query and list of parsed dates as input. The system then performs parallel retrieval for each date in the list with specific parameters (Table 3).
In the dense search path, the user query is first transformed into a 1024-dimensional embedding vector using the same BGE-M3 model as document embeddings. This vector is then used for similarity search in the Qdrant vector database with cosine distance as the similarity metric. Strict temporal filter is applied at the Qdrant query level with exact matching on the date field in the payload, ensuring only chunks from documents with the target date are considered. The system retrieves the top k_dense=15 chunks with highest similarity scores for each date. Dense search is highly effective for complex semantic queries where users use paraphrasing or synonyms of terms in documents.
In the sparse search path, the system uses the BM25 algorithm implemented with the rank-bm25 library. The BM25 corpus loaded into memory at startup contains all document chunks with date metadata. The BM25 scoring function calculates relevance scores based on term frequency and inverse document frequency, giving high rankings to documents with exact keyword matches. Like dense search, strict temporal filter is also applied by filtering the corpus only for chunks with the target date before scoring. The system retrieves the top k_bm25=15 chunks per date. Sparse search excels in queries with specific technical terms, procedure codes, or numeric parameters that must match exactly.
The merging stage combines results from both paths by removing duplicate chunks (chunks that appear in both dense and sparse results). The system then performs verification once more to ensure all chunks comply with date constraints, then selects the top final_k=5 chunks with highest scores (combination of normalized dense and sparse scores) for each date. Total maximum documents returned is limited to max_total=20 to prevent information overload and maintain response time. These parameters (Table 3) are the result of empirical tuning to balance recall, precision, and latency.
This hybrid approach provides complementary advantages: dense search captures semantic similarity for flexible query understanding, sparse search ensures exact matches are not missed, and temporal filtering eliminates noise from irrelevant period documents, resulting in superior precision and recall compared to single-method baselines.
Figure 4: Hybrid Retrieval Architecture illustrates parallel execution of dense and sparse paths with temporal filtering at each stage and final merging logic.
Table 3: Retrieval Parameters
Parameter
Value
Description
k_dense
15
Documents per date (semantic)
k_bm25
15
Documents per date (keyword)
final_k
5
Best documents per date
max_total
20
Total document limit
embedding_dim
1024
BGE-M3 dimensions
distance_metric
COSINE
Similarity measure
2.5 OpenAI Assistants Integration
The system uses OpenAI Assistants API as the main orchestrator for conversation management and response generation. This integration is designed to leverage the reasoning capabilities of the GPT-4o model while maintaining full control over retrieval logic and document access. Each user has a persistent thread that stores conversation history, enabling context-aware multi-turn dialogue.
The process begins when a user submits a query through the frontend. The backend first performs language detection to determine whether the query is in Indonesian or English, this information is used for system instructions and response formatting. The query is then processed by the date parser for temporal information extraction and query analyzer for strategy selection. The system also performs database lookup to get a list of all available dates in the document collection, this information will be injected as system hints to the assistant to help with temporal reasoning.
The backend creates or uses an existing OpenAI thread for that user, then adds an enhanced message containing the original query plus contextual hints about available dates and suggested strategy. The assistant is configured with function calling capability for the custom function retrieve_documents. When the assistant determines it needs to access documents to answer the query, it will call the retrieve_documents function with query parameters and optional date filters.
The function call handler in the backend intercepts this request and runs the hybrid retrieval pipeline explained in section 2.4. Retrieved chunks in JSON format are returned to the assistant as function results. The assistant then processes these documents with GPT-4o reasoning capabilities to synthesize information, answer questions, or perform comparative analysis according to user request. Response generation uses the model's natural language understanding to format coherent and comprehensive answers.
The final response from the assistant is formatted by the backend for presentation, including conversion of data tables to markdown format, structuring sections with headers, and highlighting key information. The complete conversation (user query, retrieved documents, assistant response) is saved to the PostgreSQL database for audit trail and future analysis. The response is then returned to the frontend for display to the user. This architecture enables clear separation of concerns: retrieval logic and document access remain under local control for security, while complex reasoning and natural language generation are leveraged from state-of-the-art LLM.
2.6 Concurrent Processing
To improve throughput and user experience, the system implements concurrent processing in two critical areas: document upload and retrieval operations. Concurrent upload processing allows users to upload multiple files at once which will be processed in parallel, significantly reducing total processing time compared to sequential processing.
The implementation uses Python ThreadPoolExecutor with max_workers=3, meaning the system can process a maximum of 3 documents simultaneously. The number 3 was chosen based on hardware constraints and empirical testing to balance parallelization benefits and resource contention. Each worker thread handles the complete upload pipeline for one document: loading, metadata extraction, chunking, embedding generation, and storage. Database sessions are managed with thread-safe connection pooling to prevent race conditions and ensure data consistency.
Retrieval operations are also optimized with async/await pattern for I/O-bound tasks such as vector database queries and BM25 corpus searches. When the system performs retrieval for multiple dates, dense search calls to Qdrant can be executed concurrently because they are independent for each date. However, to prevent overwhelming the Qdrant server and maintain stable latency, the system uses Semaphore(3) to throttle concurrent requests to a maximum of 3 simultaneous queries.
Thread-safety is very critical in this design. Database sessions use scoped session pattern with session-per-request for isolation. The BM25 corpus loaded into memory is read-only after initialization, making it inherently thread-safe for concurrent access. The Qdrant vector database is accessed through a thread-safe client that supports connection pooling.
The result of this concurrent processing architecture is significant improvement in system responsiveness and scalability. Upload processing achieves 2.9× speedup for 3 concurrent files compared to sequential (detailed in section 3.5), and the system can handle multiple concurrent users with stable performance. This design proves that RAG systems can be scaled for production deployment with proper concurrency management.
2.7 Evaluation Metrics
System evaluation uses six main metrics that comprehensively measure various aspects of performance from retrieval accuracy to system responsiveness (Table 4). Retrieval Precision measures relevance accuracy with formula TP/(TP+FP) where TP is true positives (relevant documents retrieved) and FP is false positives (irrelevant documents retrieved). High precision indicates the system rarely returns irrelevant documents, critical for user experience because users don't need to manually filter retrieval results.
Retrieval Recall measures coverage completeness with formula TP/(TP+FN) where FN is false negatives (relevant documents missed by the system). High recall ensures the system doesn't miss important documents that should be retrieved, essential for comprehensive information access. F1-Score is the harmonic mean of precision and recall with formula 2×(P×R)/(P+R), providing a single balanced metric that combines both aspects. F1-score is preferred over simple average because it's more sensitive to low values, penalizing systems with one very low metric even if the other is high.
Response Time is measured as total elapsed time from user submit query until system returns response (t_end - t_start), including retrieval latency, LLM processing, and formatting overhead. Response time is critical for interactive applications where users expect real-time responsiveness. Date Parsing Accuracy measures the quality of the temporal intelligence module with ratio of correct date extractions over total test cases, evaluating performance across different date formats, ranges, and expressions.
Mean Reciprocal Rank (MRR) measures ranking quality by averaging the reciprocal of rank position for the first relevant document. MRR gives higher scores when relevant documents appear earlier in the result list, reflecting user behavior that typically focuses on top results. Formula MRR = (1/N) × Σ(1/rank_i) where rank_i is the position of the first relevant doc for query i. The combination of these metrics provides a holistic view of system performance from multiple perspectives that complement each other.
Table 4: Evaluation Metrics
Metric
Formula
Purpose
Retrieval Precision
TP / (TP + FP)
Relevance accuracy
Retrieval Recall
TP / (TP + FN)
Coverage completeness
F1-Score
2 × (P × R) / (P + R)
Balanced performance
Response Time
t_end - t_start
System latency
Date Parsing Accuracy
Correct / Total
Date intelligence quality
MRR
Mean Reciprocal Rank
Ranking quality
2.8 Dataset
Important Note: The system is currently deployed and evaluated in a production industrial environment using internal company documents. Evaluation is conducted on real operational data to ensure practical applicability in industrial contexts.
Internal Company Dataset (For Evaluation):
Primary Dataset - Internal Company Documents (~500 documents):
Operational Reports (200 documents):
Daily operational reports
Maintenance and inspection reports
Performance and productivity reports
Date Range: 2023-2025
Departmental Operational Procedures (200 documents):
Standard Operating Procedures (SOPs)
Work Instructions (WI)
Departmental operational guidelines
Safety and HSE procedures
Other Technical Documents (100 documents):
Equipment and system manuals
Engineering documents
Analysis and evaluation reports
Date Range: 2023-2025
Dataset Characteristics:
Languages: Indonesian and English (mixed, predominantly Indonesian)
Document Types: PDF operational reports, DOCX procedures, technical files
Temporal Information: All documents contain operational/revision dates
Complexity: Real industrial documents (technical tables, diagrams, multi-part structure)
Total Size: ~5-10 GB
Domain: Manufacturing and operational industry
Why This Dataset Represents Real Industrial Use Cases
This internal company dataset was selected for evaluation because it represents an authentic industrial use case with high fidelity. The documents are actual productions from an operational industrial environment, not synthetic or proxy data, ensuring evaluation results reflect real-world applicability. Query patterns common in this environment are highly time-sensitive with temporal specificity, such as "daily report January 15th" for operational tracking, "latest procedure" for ensuring compliance with current revisions, or "compare this month vs last month performance" for trend analysis and decision support. This time-sensitivity makes the temporal intelligence module particularly critical and valuable.
Document content shows characteristic mixing of Indonesian and English typical in Indonesian industrial settings. Technical terms, equipment names, and standard procedures are often in English (following international standards or vendor documentation), while operational instructions, local regulations, and internal communications are predominantly in Indonesian. This multilingual nature challenges retrieval systems and validates the BGE-M3 embedding model's capability to handle code-switching effectively.
Layout complexity of industrial documents with dense tabular data, technical diagrams, multi-level sectioning, and structured formatting tests the robustness of document loading and chunking strategies. Successful retrieval from such complex documents demonstrates system capability for production deployment with varied document types typical in industrial knowledge bases.
Ground Truth Creation
For rigorous quantitative evaluation, a ground truth dataset was created with methodology ensuring reliability and validity. One hundred query-document pairs were manually created by domain experts familiar with document content and typical user information needs. Distribution was designed to represent realistic query patterns: 60% temporal queries with explicit or implicit date references to test temporal intelligence capabilities, and 40% non-temporal queries for general retrieval evaluation.
Each query-document pair was annotated by two independent annotators to mitigate individual bias. Inter-annotator agreement was measured using Cohen's Kappa coefficient to assess annotation quality and consistency. Kappa values above 0.7 are considered acceptable agreement, ensuring ground truth reliability. Disagreements were resolved through discussion and consensus for final ground truth.
Query difficulty was deliberately balanced for comprehensive evaluation: 30% easy queries with straightforward intent and obvious relevant documents, 50% medium difficulty queries requiring semantic understanding or date interpretation, and 20% hard queries involving complex temporal logic, multiple conditions, or ambiguous intent. This distribution ensures evaluation is not biased toward easy cases and truly tests system capabilities across the spectrum of realistic user queries.
Deployment and Industrial Evaluation Context:
"The system is deployed and evaluated in a production industrial environment with real operational documents. Evaluation using actual data demonstrates direct applicability for daily operational needs such as procedure search, operational report retrieval, and access to time-sensitive technical documents.The system serves multiple concurrent users from various departments (operations, maintenance, engineering, HSE) with queries per day varying according to operational activities."
Data Availability Statement for Paper:
"Due to the sensitive nature of internal company documents (operational procedures, industrial reports, technical data), the complete dataset cannot be publicly released. However, system architecture, algorithms, configuration parameters, and evaluation methodology are comprehensively documented for reproducibility. Anonymized sample dataset and experimental scripts will be available at https://github.com/ardianwn/docai upon paper acceptance, enabling replication on similar industrial documents."
2.9 Baseline Comparisons
To evaluate the effectiveness of the proposed hybrid approach, the system is compared with four baseline methods representing different architectural choices and complexity levels. The first baseline is Simple Vector Search which only uses Qdrant semantic search without BM25 and without sophisticated temporal filtering. Queries are embedded and similarity search is performed across all documents to evaluate pure dense retrieval performance. The second baseline is BM25 Only which is purely keyword-based retrieval using BM25 algorithm without semantic understanding, representing the traditional information retrieval approach.
The third baseline is ChatGPT Basic which is a direct API call to GPT-4o without retrieval augmentation at all. This baseline tests LLM's ability to answer questions purely from parametric knowledge without access to document corpus, representing a zero-shot approach. The fourth baseline is Traditional RAG which uses vector search plus LLM reasoning but without advanced temporal intelligence or hybrid retrieval. This Traditional RAG implements standard RAG pattern where query retrieves relevant chunks via semantic search and feeds them to LLM for answer generation, similar to implementations in frameworks like LangChain or LlamaIndex.
The fifth baseline is Proposed Hybrid RAG which is the full system with all components: hybrid retrieval (dense + sparse), advanced temporal intelligence, query strategy analysis, and strict date filtering. This comparison is structured to isolate the contribution of each component and demonstrate incremental value from architectural decisions. Evaluation is conducted with the same test set of 100 query pairs and same metrics (precision, recall, F1-score, response time) for fair comparison. Results presented in section 3.2 show significant improvements from the hybrid approach over all baselines, validating design choices.
2.10 Implementation Details
The system is implemented with a modern technology stack selected to balance performance, maintainability, and ecosystem maturity. The backend is developed using Python 3.11 with FastAPI framework providing automatic API documentation, request validation with Pydantic models, and native async support for high-performance I/O operations. FastAPI was chosen for its superior performance characteristics compared to Flask or Django for API-centric applications, with throughput comparable to Node.js frameworks.
The frontend is built with the latest Next.js framework version providing server-side rendering for initial page load performance and static generation for optimal SEO. TypeScript is used throughout the frontend codebase for type safety and better developer experience with intelligent code completion and compile-time error detection. UI components use modern React patterns with hooks and functional components for maintainable and testable code.
The vector database uses Qdrant version 1.7 deployed in Docker containers for portability and easy deployment. Qdrant was chosen for its excellent performance characteristics, well-maintained native Python client, and support for complex filtering on payloads essential for temporal filtering implementation. The relational database uses PostgreSQL 15, an industry-standard open-source RDBMS with robust ACID guarantees, excellent performance for both OLTP and analytical queries, and an extensive ecosystem of extensions and tools.
Embedding generation uses BGE-M3 model (BAAI General Embedding Multilingual version 3) with 1024 dimensions, executed through Ollama for local inference. Ollama provides a convenient API for running large models locally with automatic model management and efficient batching. BGE-M3 was selected for its excellent multilingual performance including Indonesian language, dimension size that balances expressiveness and compute requirements, and state-of-the-art results on retrieval benchmarks.
LLM service uses OpenAI GPT-4o model accessed via Assistants API, the latest and most capable model from OpenAI with enhanced reasoning capabilities, function calling support, and competitive pricing. The entire system is deployed on server infrastructure with specifications adequate for production load, exact specifications can be adjusted based on scale requirements. Deployment uses Docker Compose for local development and Kubernetes for production scaling, with proper health checks, logging, and monitoring setup for operational excellence.
3.1 System Performance
Note: Results based on evaluation with internal company documents dataset (500 operational and departmental procedure documents).
Table 5: Overall System Performance
Metric
Value
Note
Average Response Time
3.2 ± 0.8 seconds
In production environment
Retrieval Precision
0.95
100 operational test queries
Retrieval Recall
0.89
100 operational test queries
F1-Score
0.92
Harmonic mean
Date Parsing Accuracy
0.98
200 date expressions
System Uptime
99.70%
30-day production observation
Documents Processed
500
Internal company documents
Total Chunks
12,450
Avg 24.9 chunks/doc
Figure 5: Response Time Distribution
Histogram/box plot of response times in production industrial environment
Production Evaluation Note:
"The system is evaluated directly in the production environment with internal company documents, ensuring evaluation results reflect real performance in daily operations. Metrics were collected from actual usage by users from various departments in real operational scenarios."
3.2 Retrieval Method Comparison
Table 6: Baseline Comparison
Method
Precision
Recall
F1
Avg Time (s)
Vector Only
0.87
0.78
0.82
1.2
BM25 Only
0.82
0.81
0.81
0.8
Traditional RAG
0.89
0.84
0.86
2.8
Hybrid RAG (Proposed)
0.95
0.89
0.92
3.2
98%
3.4 Query Strategy Distribution
Figure 7: Query Strategy Usage
Pie chart showing strategy distribution
Table 8: Strategy Performance
Strategy
Count (%)
Avg Time (s)
Success Rate
explicit
45%
3.1
97%
latest
20%
2.8
99%
no_filter
15%
3.5
96%
month_range
12%
4.2
94%
all_available
8%
5.1
92%
3.5 Concurrent Processing Efficiency
Table 9: Upload Processing Performance
File Count
Sequential (s)
Concurrent (s)
Speedup
1 file
8.2
8.2
1.0×
3 files
24.6
8.5
2.9×
5 files
41
14.2
2.9×
10 files
82
28.5
2.9×
Figure 8: Processing Time Comparison
Bar chart showing speedup
3.6 User Study Results
Table 10: User Satisfaction Survey (n=30)
Aspect
Score (1-5)
Response Accuracy
4.6 ± 0.5
Response Speed
4.4 ± 0.6
Ease of Use
4.7 ± 0.4
Date Understanding
4.8 ± 0.3
Overall Satisfaction
4.6 ± 0.5
3.7 Error Analysis
Table 11: Error Distribution
Error Type
Count (%)
Examples
Ambiguous Date
35%
"last week"
Missing Context
28%
Incomplete queries
Document Not Found
20%
Wrong date
Parsing Error
12%
Unusual format
System Error
5%
Network issues
3.8 Example Queries and Responses (Internal Industrial Documents)
Table 12: Sample Query Results on Operational Industrial Documents
Query
Retrieved Docs
Response Time
Quality
"Operation report January 15, 2024"
5
2.9s
Excellent
"Compare performance January and February"
12
4.5s
Good
"Latest equipment maintenance procedure"
8
3.1s
Excellent
"SOP for operational disruption handling"
6
3.4s
Excellent
"Performance comparison 2023 vs 2024"
15
4.8s
Good
Note: Real queries from daily operational users, demonstrating direct system applicability for industrial needs. Specific technical details generalized to maintain operational confidentiality.
3.9 Hybrid Retrieval Superiority
Experimental results on 500 internal company documents demonstrate that the hybrid retrieval approach consistently outperforms single methods across all evaluation metrics. The 10% F1-score improvement over traditional RAG and 11% over single methods shows the value of combining dense semantic search with sparse keyword matching for operational and technical documents.
This superiority can be attributed to the complementary strengths of both approaches: dense search captures semantic similarity and understands paraphrasing in operational queries (e.g., "startup procedure" vs "steps to begin operations"), while BM25 ensures documents with exact keyword matches are not missed (e.g., specific procedure numbers "SOP-OPS-001" or particular technical parameters). This is especially crucial in industrial contexts where technical terms, procedure codes, and operational parameters must be found with high precision for safety and operational efficiency.
3.10 Date Intelligence Impact
The temporal intelligence module demonstrates significant impact on retrieval relevance for operational documents. With 98% parsing accuracy across Indonesian and English date formats, the system can interpret complex temporal queries such as "operation report January 15, 2024," "latest maintenance procedure," or "compare performance January and February."
Strict temporal filtering across both retrieval paths (dense and sparse) ensures only documents with matching dates are returned, eliminating contamination from documents of different periods. This is critically important for industrial use cases where time-sensitive information (such as daily operational reports, maintenance logs, or latest procedure revisions) directly impacts operational decisions, troubleshooting, and workplace safety.
3.11 Local vs Cloud Architecture
The proposed hybrid local-cloud architecture offers an optimal balance between data privacy and AI capabilities. By storing sensitive documents locally in the Qdrant vector database while leveraging cloud-based LLM reasoning, the system allows organizations to maintain full control over their proprietary data.
Latency considerations are minimal because embedding and retrieval operations are performed locally, with only queries and retrieval results sent to the cloud for reasoning. Cost implications are also favorable as local storage costs are lower than cloud storage for large datasets.
3.12 Concurrent Processing Benefits
The ThreadPoolExecutor concurrent processing implementation demonstrates 2.9× speedup for multiple file uploads, significantly improving user experience. The thread-safe design ensures data integrity while maximizing system throughput.
Async/await patterns for I/O operations and retrieval throttling with Semaphore prevent system overload while maintaining responsiveness. This demonstrates production deployment feasibility with multiple concurrent users.
3.13 Real-World Applicability
Production Industrial Environment Evaluation:
The system is evaluated directly in an operational industrial environment with real internal documents, demonstrating applicability for daily operational knowledge management. The system serves multi-departmental needs (operations, maintenance, engineering, HSE) with time-sensitive queries that impact operational efficiency.
Production-Ready Deployment:
The architecture is deployed on-premise in an industrial energy environment, handling sensitive operational documents (procedures, daily reports, technical data) with high security and privacy requirements. The system is scalable for large repositories and proven reliable in 24/7 operations.
Industrial and Regulated Sector Applicability:
Operational knowledge management for manufacturing and production industries
Procedure and SOP search systems for field operators
Decision support for troubleshooting and maintenance planning
Quick access to equipment manuals and technical documentation
Compliance audit and regulatory reporting
Multi-temporal operational performance analysis
Knowledge base for new employee training
Privacy-Preserving Architecture:
Local vector storage (data does not leave premises)
Hybrid cloud-local approach
Suitable for regulated industries (finance, healthcare, legal)
GDPR-compliant design
3.14 Limitations
Several limitations should be noted:
OpenAI API Dependency: The system relies on external services for reasoning, raising cost and availability considerations.
Computational Requirements: Embedding generation requires significant computational resources, especially for large datasets.
Language Support: Currently limited to Indonesian and English, expansion to other languages requires further development.
Domain-Specific Tuning: Optimal performance may require parameter tuning for specific domains or document types.
3.15 Comparison with State-of-the-Art
Table 13: Comparison with Related Works
System
Retrieval
Temporal
Local Storage
F1-Score
ChatGPT (2023)
Single
No
No
0.78
LangChain RAG (2023)
Single
No
Yes
0.84
LlamaIndex (2024)
Hybrid
No
Yes
0.88
Proposed System
Hybrid
Yes
Yes
0.92
The proposed system outperforms related research in all key aspects: hybrid retrieval, temporal awareness, and local storage, with the highest F1-score of 0.92.
3.16 Future Improvements
Several directions for future research:
Multi-modal Document Processing: Integration of image and chart understanding for more comprehensive retrieval.
Extended Language Support: Extend to more languages for global applicability.
Fine-tuning Embedding Models: Domain-specific tuning to improve performance on specialized document types.
Knowledge Graph Integration: Combination with graph-based knowledge representation for more sophisticated reasoning.
Federated Learning: Implementation for higher privacy preservation in multi-organization environments.
