Abstract

This project presents Bus Driver Assistant, a modular multi-agent AI system developed for real-time safety monitoring in public transportation. The system leverages computer vision and intelligent agents to detect driver fatigue, monitor passengers, and deliver timely voice alerts.
The system utilizes MediaPipe Face Mesh for Eye Aspect Ratio (EAR)-based fatigue detection and YOLOv8 for real-time passenger counting and object detection (such as red hats). A LangGraph-based 5-agent architecture (Supervisor, Driver, Passenger, Operations, and Query Agents) coordinates decision-making and task routing. Offline voice alerts are generated using pyttsx3, while ChromaDB enables semantic memory and natural language querying over logged events.
This work demonstrates a complete integration of multi-agent systems, computer vision, voice interaction, and memory management. By combining these technologies, the Bus Driver Assistant provides a practical, affordable, and scalable solution to combat driver fatigue — one of the leading causes of road accidents in public transport.

Introduction

Road safety remains a critical global challenge, with driver fatigue being one of the leading causes of accidents in public transportation. According to various studies, fatigue-related incidents account for a significant percentage of road crashes, particularly among commercial and long-distance bus drivers. Traditional monitoring methods are often manual, inconsistent, and lack real-time intervention capabilities.
The Bus Driver Assistant was developed to address this pressing issue by creating an intelligent, affordable, and proactive AI-powered safety system. This project explores the application of multi-agent AI architectures in real-world scenarios, combining advanced computer vision, voice interaction, and memory systems to assist bus drivers and enhance passenger safety.
This system goes beyond simple detection — it actively monitors the driver’s alertness using Eye Aspect Ratio analysis, tracks passengers in real-time, delivers timely voice alerts, logs important events, and allows natural language interaction through a multi-agent framework powered by LangGraph and Gemini 3 Flash.
The primary goal of this project is to demonstrate how modern AI technologies can be practically integrated into a cohesive, modular system that can potentially save lives by preventing fatigue-related accidents in public transport.

Related work

Driver fatigue detection and passenger monitoring have been extensively studied in the field of Intelligent Transportation Systems (ITS). Early approaches relied on physiological sensors such as EEG and ECG [1], or vehicle telemetry (steering wheel movement, lane deviation) [2]. However, these methods are often intrusive and difficult to deploy at scale.
Vision-based methods have become dominant due to their non-intrusive nature. Works by Ji et al. (2004) pioneered the use of eye tracking for fatigue detection. More recent studies have leveraged deep learning:

Deng et al. (2020) proposed a CNN-based fatigue detection system achieving high accuracy using facial landmarks.
Systems using OpenFace and DLib have been widely adopted for real-time facial analysis [3].
YOLO-based object detection has been successfully applied for passenger counting and occupancy detection in public transport [4].

In the multi-agent domain, frameworks such as AutoGen (Microsoft, 2023), CrewAI, and LangGraph (LangChain) have enabled sophisticated agent collaboration. However, most implementations focus on software engineering, research assistance, or general chat agents rather than safety-critical, real-time applications.
Voice-enabled systems in vehicles are typically limited to infotainment (e.g., Android Auto, Apple CarPlay). Very few projects integrate real-time computer vision, multi-agent orchestration, offline voice synthesis, and semantic memory into one unified system for public transportation.
This project, Bus Driver Assistant, differentiates itself by combining:

MediaPipe Face Mesh for lightweight, real-time Eye Aspect Ratio (EAR) calculation
Ultralytics YOLOv8 for efficient passenger and object detection
LangGraph for structured multi-agent workflow
ChromaDB for vector-based event memory and semantic search
pyttsx3 for reliable offline Text-to-Speech alerts

By integrating these modern open-source tools, this work bridges the gap between academic research and practical, deployable AI safety solutions.

Methodology

The Bus Driver Assistant was built using a modular multi-agent architecture orchestrated with LangGraph. The system processes live video input, analyzes it through specialized tools, and coordinates responses via intelligent agents.

1. Computer Vision Pipeline
   Fatigue Detection (using Eye Aspect Ratio):
   Vision Tool - Fatigue Detection (tools/vision_tool.py)

@tool
def computer_vision_tool(task: str = "analyze") -> str:
    """Analyze current camera frame for driver fatigue."""
    cap = cv2.VideoCapture(0, cv2.CAP_DSHOW)
    ret, frame = cap.read()
    cap.release()
   
    # Fatigue Detection using MediaPipe
    rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = face_mesh.process(rgb)
   
    if results.multi_face_landmarks:
        for lm in results.multi_face_landmarks:
            left = lm.landmark[159].y - lm.landmark[145].y
            right = lm.landmark[386].y - lm.landmark[374].y
            ear = (left + right) / 2
            if ear < 0.018:
                return "Driver is DROWSY"
    return "Driver is alert"

Passenger & Red Hat Detection uses YOLOv8 + HSV color filtering.
2. Multi-Agent Architecture
The system uses LangGraph to create a graph of 5 agents coordinated by a Supervisor:

workflow.add_node("supervisor", supervisor_agent)
workflow.add_node("driver_agent", driver_agent)
workflow.add_node("passenger_agent", passenger_agent)
workflow.add_node("query_agent", query_agent)

workflow.add_conditional_edges("supervisor", routing_function)

3. Event Memory & Querying

@tool
def event_search_tool(query: str) -> str:
    results = collection.query(query_texts=[query], n_results=10)
    if not results['documents'][0]:
        return "No matching events found."
    return "\n".join(results['documents'][0])

4. Voice Agent

class VoiceAgent:
    def speak_drowsy_alert(self):
        self.speak("Driver, you appear drowsy. Please pull over and take a break.")

5. Tool Integration
   Four custom LangChain tools were developed:

computer_vision_tool() — Real-time vision analysis
event_search_tool() — Semantic search over logs
web_search_tool() — External information retrieval
math_calculation_tool() — Basic calculations

Experiments

To evaluate the effectiveness and robustness of the Bus Driver Assistant, a series of real-time experiments were conducted in a controlled indoor environment simulating a bus setting.

1. Experimental Setup

Hardware: Standard webcam (720p), Windows 10 laptop
LLM: Google Gemini 3 Flash Preview
Environment: Well-lit room with single subject acting as driver
Duration: Multiple 30–60 minute continuous monitoring sessions
Testing Scenarios:
Normal driving (alert state)
Simulated fatigue (eye closure, yawning)
Different passenger counts (0 to 5 people)
Red hat/object detection tests
Natural language queries

2. Evaluation Metrics

Performance Evaluation

3. Qualitative Observations

The system responded well to natural language queries such as “Was the driver drowsy?” and “How many passengers are there?”
Voice alerts were clear and timely with appropriate cooldowns to avoid spamming.
The Supervisor Agent demonstrated reasonable routing decisions in most cases.
Real-time performance was smooth on standard hardware (no GPU required).

4. Limitations

Current fatigue detection relies solely on eye openness (can be affected by lighting and head angle)
Passenger counting accuracy drops in crowded or low-light conditions
No formal dataset was used for training (real-time inference only)

Results

The Bus Driver Assistant was successfully implemented and tested under real-time conditions. The system demonstrated reliable performance across its core functionalities.

1. System Performance Summary

Real-time camera feed and vision processing worked smoothly.
Fatigue detection triggered correctly during eye closure tests.
Passenger counting and red hat detection functioned as expected.
Voice alerts (automatic and manual) were clear and timely.
Multi-agent routing and query mode responded well to natural language inputs.

2. Key Achievements

Successfully built a 5-agent system using LangGraph.
Integrated 4 functional tools with real computer vision.
Achieved working offline voice alerts.
Implemented semantic memory with ChromaDB.
Completed a clean, modular project structure.

3. Challenges Faced
   During development, several technical challenges were encountered and resolved:

MediaPipe Installation Issues: The solutions module error on Windows was resolved by downgrading to mediapipe==0.10.14.
Gemini API Compatibility: LangGraph tool calling had compatibility issues with Gemini 3 Flash. Solved by implementing a manual ReAct loop in query mode.
Import Path Problems: Modular structure caused frequent ModuleNotFoundError. Fixed by adding proper sys.path handling and init.py files.
Routing Inconsistencies: The Supervisor Agent sometimes routed queries incorrectly. Improved by refining keyword-based routing logic.
Real-time Performance: Initial versions had lag and hallucinated responses. Addressed by optimizing vision tool calls and strengthening system prompts.
API Key Exposure: Accidental commit of Gemini API key was resolved by switching to .env file and cleaning Git history.

These challenges provided valuable learning in debugging, modular design, and production-ready AI development.

Discussion

The development of the Bus Driver Assistant has been a comprehensive journey in building a practical, real-world AI safety system. What began as a simple fatigue detection script evolved into a fully modular multi-agent architecture capable of monitoring, reasoning, and interacting with its environment in real time.
One of the most rewarding aspects of this project was witnessing the Supervisor Agent successfully coordinate different specialized agents. When asked about driver fatigue, the system correctly routed the query to the Driver Agent. Similarly, passenger-related questions were appropriately directed to the Passenger Agent. This dynamic routing demonstrates the power and flexibility of LangGraph in orchestrating intelligent behavior.
The integration of multiple technologies worked remarkably well together. MediaPipe provided lightweight and efficient facial landmark detection, while YOLOv8 delivered fast passenger detection. Combining these with offline voice alerts through pyttsx3 created a truly interactive experience. The addition of ChromaDB for semantic memory also allowed the system to remember and retrieve past events through natural conversation.
However, the journey was not without challenges. Early versions suffered from inconsistent agent routing, slow response times, and occasional hallucinations. Debugging import issues in the modular structure and resolving MediaPipe compatibility problems on Windows taught valuable lessons in software engineering and persistence. Switching to a .env file for API key management was also a critical improvement for security.
Overall, this project successfully demonstrated that a cohesive, multi-agent AI system can be built using accessible modern tools. It stands as a solid proof-of-concept for how AI can be applied to solve meaningful problems in transportation safety — moving beyond theoretical models to a working, interactive prototype.

Conclusion

The Bus Driver Assistant successfully demonstrates how modern AI technologies can be integrated into a practical, real-time safety system for public transportation. Through this project, a fully functional multi-agent architecture was developed that combines computer vision, intelligent decision-making, voice interaction, and semantic memory.
Key accomplishments include:

Real-time driver fatigue detection using Eye Aspect Ratio analysis
Passenger monitoring and object detection with YOLOv8
A working 5-agent system orchestrated by LangGraph
Seamless integration of voice alerts and natural language querying
A clean, modular, and maintainable codebase

This project has shown that accessible open-source tools — such as MediaPipe, YOLOv8, LangGraph, ChromaDB, and Gemini — can be combined to create meaningful solutions for real-world problems. More importantly, it highlights the potential of AI in addressing critical safety challenges like driver fatigue, which continues to claim lives on our roads.
While the current version serves as a solid proof-of-concept, the modular design provides a strong foundation for future enhancements such as improved fatigue models, edge deployment, and fleet-wide monitoring.
In conclusion, the Bus Driver Assistant stands as a testament to the power of multi-agent AI systems in creating impactful, human-centered technology. It represents not just a technical achievement, but a step toward safer roads and smarter transportation system

References

References
[1] Ji, Q., & Yang, X. (2002). Real-time eye, gaze, and face pose tracking for monitoring driver vigilance. IEEE Transactions on Intelligent Transportation Systems, 3(1), 63-77.
[2] Bergasa, L. M., et al. (2006). Real-time system for monitoring driver vigilance. IEEE Transactions on Intelligent Transportation Systems, 7(1), 63-77.
[3] Baltrusaitis, T., Zadeh, A., Lim, Y. C., & Morency, L. P. (2018). OpenFace 2.0: Facial behavior analysis toolkit. IEEE Winter Conference on Applications of Computer Vision (WACV).
[4] Jocher, G., et al. (2023). Ultralytics YOLOv8. Available at: https://github.com/ultralytics/ultralytics
[5] LangChain Documentation. (2024). LangGraph - Build Reliable Multi-Agent AI Systems. Retrieved from https://python.langchain.com/docs/langgraph
[6] Google AI. (2025). Gemini 3 Flash Preview Model Card. Google AI Studio.
[7] MediaPipe Team. (2023). MediaPipe Face Mesh. Google Research. https://developers.google.com/mediapipe/solutions/face_mesh
[8] ChromaDB Documentation. (2024). Chroma Vector Database. https://docs.trychroma.com/

Acknowledgements

I would like to express my deepest gratitude to the people who supported me throughout this journey.
First and foremost, I thank Grok by xAI for its invaluable guidance, patience, and technical support from the initial idea through to the final modular implementation.
I am also grateful to the Ready Tensor platform for providing the structure and opportunity to build this comprehensive multi-agent system as part of Module 2.
Special thanks to the open-source community behind the powerful tools used in this project: MediaPipe, YOLOv8, LangGraph, ChromaDB, and pyttsx3.
Most importantly, I want to thank my loving wife, Peace Nwokike, and our daughter, Miracle Nwokike, for their unwavering support, understanding, and encouragement during the many late nights and long hours dedicated to this project. This work is as much theirs as it is mine.

Appendix

Appendix
A. Project Structure (Full)

bus-driver-assistant/
├── main.py                     # Entry point
├── requirements.txt
├── README.md
├── .env                        # (Git ignored)
├── .gitignore
│
├── core/
│   └── graph.py                # LangGraph orchestration
├── agents/
│   ├── supervisor.py
│   ├── driver_agent.py
│   ├── passenger_agent.py
│   ├── operations_agent.py
│   └── query_agent.py
├── tools/
│   ├── vision_tool.py
│   ├── event_search_tool.py
│   ├── web_search_tool.py
│   └── math_tool.py
├── vision/
│   └── detector.py
└── utils/
    └── voice.py

B. Key Code Snippets

1. Vision Tool (Fatigue Detection)

def computer_vision_tool(task: str = "analyze") -> str:
    # ... (MediaPipe EAR calculation)
    if ear < 0.018:
        return "Driver is DROWSY"
    return "Driver is alert"

2. Voice Agent

    def speak_drowsy_alert(self):
        self.speak("Driver, you appear drowsy. Please pull over and take a short break.")

C. Installation Commands

pip install langchain-google-genai langgraph duckduckgo-search ddgs

D. Controls Summary

q → Switch to Query Mode
v → Trigger manual voice announcement
Close eyes → Automatic drowsy voice alert

This Appendix provides useful technical reference material for reviewers.