Certifiedunder the RAG-Based AI Assistant module in the Agentic AI Essentials program.Large Language Models (LLMs) can answer many questions, but they may generate information that is not supported by the available documents. This can be a challenge in aviation, where users often need information from technical, operational, and regulatory documents.
This project presents an Aviation Retrieval-Augmented Generation (RAG) Assistant that combines document retrieval with a Large Language Model to answer aviation-related questions. The system processes aviation documents, divides them into smaller text chunks, creates embeddings, stores them in a Chroma vector database, and retrieves relevant information when a user submits a question. The retrieved information is then provided to a Groq-hosted LLM to generate a response.
The implementation uses Python, LangChain, HuggingFace embeddings, ChromaDB, and the Llama 3.3 70B model through Groq. The system is designed to keep responses grounded in the available aviation knowledge base and to avoid answering when sufficient information cannot be retrieved.
Aviation organizations work with a large volume of documents, including technical manuals, operational procedures, reports, policies, and regulatory material. Finding specific information from these documents can be time-consuming when users have to search through them manually.
Large Language Models provide a more natural way of interacting with information, but they can sometimes generate responses that are not supported by a particular document collection. This is commonly referred to as hallucination.
Retrieval-Augmented Generation provides an alternative approach. Instead of asking the language model to answer a question only from its pretrained knowledge, relevant information is first retrieved from a defined knowledge base. The retrieved information is then supplied to the model as context for generating the answer.
The objective of this project was therefore to develop a simple aviation-focused RAG assistant that can retrieve information from aviation documents and provide natural-language answers based on that information.
The project is intended as a practical demonstration of how RAG can be applied to an aviation knowledge base rather than as a replacement for official aviation procedures, regulations, or professional decision-making.
The Aviation RAG Assistant follows a five-stage workflow:
Document Loading → Text Chunking → Embeddings → Vector Retrieval → Response Generation
Figure 1. Aviation RAG Architecture
Suggested methodology image for the publication:
Aviation Documents
│
▼
Document Loading
│
▼
Text Chunking
800 characters
150 overlap
│
▼
HuggingFace Embeddings
│
▼
ChromaDB
Vector Database
│
│
User Question
│
▼
Semantic Retrieval
Top 4 chunks
│
▼
Retrieved Context
│
▼
Groq Llama 3.3
│
▼
Grounded Answer
2.1 Document Loading
The current implementation loads Word documents (.docx) from the project's data directory using UnstructuredWordDocumentLoader.
The extracted document content is then passed to the text-splitting stage.
2.2 Text Chunking
Text chunking is important because entire documents may be too large to pass directly to the language model.
The project uses LangChain's RecursiveCharacterTextSplitter.
The current configuration is:
Parameter Value
Chunk size 800 characters
Chunk overlap 150 characters
Retrieval results Top 4 chunks
The 150-character overlap allows some information from the previous chunk to remain available in the following chunk. This helps reduce the possibility of losing context when an important statement occurs near a chunk boundary.
2.3 Embeddings
After chunking, each text chunk is converted into a numerical representation using the HuggingFace embedding model:
sentence-transformers/all-MiniLM-L6-v2
These embeddings allow the system to compare the meaning of the user's question with the meaning of document chunks.
2.4 Vector Database
The generated embeddings are stored in ChromaDB.
When a user asks a question, the system performs a similarity search against the stored document embeddings and retrieves the four most relevant chunks.
2.5 Response Generation
The retrieved chunks are passed to the Llama 3.3 70B model through Groq.
The system prompt instructs the model to use only the retrieved context. If the required information is not available, the assistant is instructed to respond that there is not enough information in the aviation knowledge base.
This grounding approach is intended to reduce unsupported answers.
The project is available as an open-source repository:
GitHub: https://github.com/Lengerpei/Aviation-RAG-assistant
3.1 Clone the Repository
git clone https://github.com/Lengerpei/Aviation-RAG-assistant.git
cd Aviation-RAG-assistant
3.2 Create a Virtual Environment
For Windows:
python -m venv venv
venv\Scripts\activate
For Linux/macOS:
python3 -m venv venv
source venv/bin/activate
3.3 Install Dependencies
pip install -r requirements.txt
The repository includes a pinned requirements.txt containing the main dependencies used by the application, including LangChain, ChromaDB, Groq, HuggingFace, Sentence Transformers, and related packages.
3.4 Configure the API Key
Create a .env file in the project directory and add:
GROQ_API_KEY=your_api_key_here
The application reads the API key from the environment rather than storing it directly in the source code.
3.5 Prepare the Knowledge Base
Place aviation .docx documents in the project's data directory.
Run the ingestion process:
python src/ingest.py
This loads the documents, creates the text chunks, generates embeddings, and stores them in ChromaDB.
3.6 Start the Assistant
Run:
python src/app.py
The application then provides a simple command-line interface where users can enter aviation-related questions.
Type:
exit to close the application.
The assistant was tested using questions related to the information contained in the aviation knowledge base.
The testing focused on four areas:
Retrieval relevance – whether the system retrieved information related to the user's question.
Answer relevance – whether the generated response addressed the question.
Grounding – whether the answer was based on the retrieved context.
Out-of-scope behaviour – how the assistant responded when the required information was not available.
Example questions included:
What is a Fixed Base Operator?
Explain the role of an airport.
What information is contained in the aviation documents?
Summarize information about a specific aviation topic.
The system was also tested with questions for which the knowledge base did not contain sufficient information.
#Results
The retrieval evaluation produced the following results:
Metric Result
Number of questions 30
Recall@1 83.3%
Recall@3 100%
Recall@5 100%
MRR 0.917
Recall@1 – 83.3%
The correct information was retrieved as the first result for approximately 83% of the evaluation questions.
This indicates that the retrieval system generally ranks relevant information highly.
Recall@3 – 100%
The relevant information was retrieved within the top three results for all 30 questions.
This is an important result because it shows that even when the most relevant chunk was not ranked first, it was still retrieved within the top three results.
Recall@5 – 100%
All evaluation questions had their relevant information retrieved within the top five results.
MRR – 0.917
The MRR score of 0.917 indicates that relevant chunks generally appeared near the top of the retrieval results.
Overall, the evaluation indicates that the current similarity-search approach performs well on the project's test questions.
However, the evaluation dataset contains 30 questions and is based on the current knowledge base. Therefore, the results should be viewed as an initial evaluation rather than a complete measure of system performance.
#Example Retrieval Results
One example from the evaluation was:
Question
What is a Safety Management System in aviation?
The system returned:
Rank 1: chunk_033 | Distance: 0.4037
Rank 2: chunk_032 | Distance: 0.5227
Rank 3: chunk_034 | Distance: 0.8323
The most relevant information was retrieved as the first result.
Another example was:
Question
What are the main components of Air Traffic Management?
The system returned:
Rank 1: chunk_029 | Distance: 0.7171
Rank 2: chunk_031 | Distance: 0.7971
Rank 3: chunk_012 | Distance: 0.8218
This demonstrates how the system identifies relevant sections of the knowledge base based on semantic similarity.
#Discussion
The evaluation demonstrates that the Aviation RAG Assistant can successfully retrieve relevant information from the aviation knowledge base.
The 83.3% Recall@1 result indicates that there is still room to improve the ranking of the first retrieved result.
However, the 100% Recall@3 and Recall@5 results indicate that relevant information was consistently available within the retrieved results.
This suggests that the current embedding and similarity-search approach provides a useful foundation for the application.
The evaluation also highlights an important distinction between retrieval and answer generation. Good retrieval improves the context available to the language model, but it does not by itself guarantee that every generated answer will be correct.
Further evaluation of answer quality would therefore be useful as the project develops.
The Aviation RAG Assistant is intended primarily for aviation knowledge retrieval, learning, and information support.
It should not be treated as a replacement for:
Official aviation regulations
Airport operating procedures
Airline operating manuals
Air traffic control instructions
Safety-critical professional judgement
Current regulatory guidance
Aviation information can change over time, and operational decisions require authoritative and current information.
The RAG architecture reduces the risk of unsupported responses by providing retrieved document context to the language model. However, retrieval does not completely eliminate the possibility of incorrect or incomplete generated responses.
Users should therefore verify important aviation information against authoritative sources before using it for operational or safety-critical decisions.
The project can also be further strengthened with explicit content filtering, answer validation, and stronger safeguards for safety-critical queries.
#Evaluation and Development
The project includes an evaluation component that can be used to assess retrieval performance.
The evaluation workflow includes:
Test Questions
│
▼
Query Embedding
│
▼
Chroma Similarity Search
│
▼
Top-K Retrieved Chunks
│
▼
Compare with Relevant Chunks
│
▼
Calculate Retrieval Metrics
The evaluation dataset can be expanded as the knowledge base grows.
Future evaluation could include:
More questions
Different question types
Questions requiring multiple chunks
Difficult or ambiguous queries
Retrieval precision
Answer faithfulness
Answer relevance
Hallucination analysis
#Maintenance and Support
The knowledge base can be updated by adding new aviation documents to the data directory and rerunning the ingestion process.
The vector database can then be rebuilt using the updated documents.
The evaluation dataset should also be updated when new subject areas are introduced.
Future maintenance and development will focus on:
Expanding the aviation knowledge base
Increasing the evaluation dataset
Improving query processing
Improving retrieval ranking
Testing hybrid search
Adding reranking
Improving answer evaluation
Adding stronger content filtering
Improving source attribution
Monitoring retrieval performance after knowledge-base updates
#Limitations
The current implementation has several limitations.
Limited evaluation dataset
The retrieval evaluation currently uses 30 questions. A larger and more diverse evaluation dataset would provide stronger evidence of general performance.
Knowledge-base dependency
The assistant's responses depend on the content available in the aviation knowledge base.
If the required information is not contained in the documents, retrieval may not provide sufficient context.
Retrieval ranking
Although Recall@3 and Recall@5 reached 100%, Recall@1 was 83.3%. This shows that some relevant chunks are not always ranked first.
Generated response dependency
The quality of the final response depends on both retrieval quality and the language model's ability to use the retrieved context correctly.
Current query processing
The current system primarily relies on semantic similarity search. More advanced query processing and reranking techniques could improve retrieval performance further.
#Future Improvements
Several improvements are planned for future versions.
Hybrid Search
Combine semantic vector search with keyword-based search to improve retrieval for queries containing specific aviation terminology.
Retrieval Reranking
Introduce a reranking stage to improve the ordering of retrieved chunks and potentially improve Recall@1.
Expanded Evaluation
Increase the number of evaluation questions and include more complex queries requiring information from multiple sections.
Answer Evaluation
Evaluate not only whether the correct chunk was retrieved, but also whether the final generated answer is:
Correct
Relevant
Grounded in the source
Complete
Source Citations
Provide users with the source document or relevant chunk used to generate each response.
Query Reformulation
Improve the system's ability to interpret ambiguous or poorly worded questions before retrieval.
Content Filtering
Introduce additional safeguards to identify inappropriate, unsafe, or unsupported requests and reduce the risk of generating potentially harmful responses.
#Project structure.
The repository is organized around the application, knowledge base, ingestion process, and evaluation components.
Aviation-RAG-assistant/
│
├── data/
│ └── Aviation knowledge documents
│
├── src/
│ ├── init.py
│ ├── app.py
│ ├── ingest.py
│ ├── rag_pipeline.py
│ ├── embeddings.py
│ ├── vector_store.py
│ └── evaluation/
│ ├── test_questions.json
│ ├── evaluate_retrieval.py
│ ├── check_chunks.py
│ └── check_metadata.py
│
├── .env.example
├── requirements.txt
├── LICENSE
└── README.md
#Conclusion
The Aviation RAG Assistant demonstrates a practical application of Retrieval-Augmented Generation to aviation knowledge retrieval.
The system processes aviation documents, divides them into searchable chunks, generates embeddings, stores them in ChromaDB, retrieves relevant information through similarity search, and uses the retrieved context to support response generation.
The retrieval evaluation of 30 questions achieved:
Recall@1: 83.3%
Recall@3: 100%
Recall@5: 100%
MRR: 0.917
These results provide an initial indication that the retrieval component is effective for the current aviation knowledge base.
The project also provides a foundation for further development, particularly in retrieval ranking, query processing, evaluation, source attribution, content filtering, and answer-quality assessment.
The goal is not to replace aviation professionals or official aviation information systems, but to demonstrate how RAG can make aviation knowledge easier to search and interact with while keeping responses grounded in a defined document collection.