MEDIBOT — Multi‑Agent AI Medical Assistant with Streamlit UI

Abstract

In the age of intelligent healthcare, the demand for secure, explainable, and privacy-centric AI systems has grown exponentially. Traditional medical chatbots rely heavily on cloud-based APIs, raising concerns about data privacy, limited transparency, and inconsistent factual grounding. MEDIBOT redefines this paradigm through a multi-agent, retrieval-augmented, and domain-controlled medical conversational framework that ensures privacy, reliability, and clinical relevance.

The MEDIBOT Multi-Agent System integrates LangChain, FAISS, and Google Gemini LLMs to build a retrieval-augmented medical dialogue engine capable of dynamic reasoning, factual validation, and multimodal processing. By orchestrating specialized AI agents—spanning Diagnosis, Drug Information, Lifestyle Guidance, Medical Research Summarization, and Image Analysis—MEDIBOT provides context-aware, evidence-based, and ethically aligned healthcare assistance for professionals, students, and patients.

This project is an updated version of the following project:

https://app.readytensor.ai/publications/hIEm1nLm0QNG

The new version is an updated, production-ready release featuring a Streamlit web UI and several key enhancements:

• Streamlit UI: Interactive dashboard with live agent progress, chat interface, and one-click response downloads.

• Browse Files & Images: Upload PDFs, DOCX, or images for analysis using EasyOCR (text extraction) and BLIP (image captioning).

• Enhanced Privacy: Runs locally with strict domain enforcement and no third-party API dependency.

• Improved Orchestration: Structured agent flow (Diagnosis → Drug Info → Lifestyle → Research → Image) with guardrails and retries.

• Retrieval-Augmented Generation (RAG): Contextual retrieval using FAISS and semantic similarity scoring.

• Long-Term Memory: Conversation summarization for consistent and coherent dialogue.

• Safety & Evaluation Layers: Domain filters, factual accuracy checks, and ethical alignment.

• Usability Paths: CLI, Python API, and Streamlit—use in any workflow.

• Artifacts & Logs: Organized outputs with audit-friendly metadata and logging.

Highlights

🩺 Medical-grade RAG pipeline with factual retrieval
🧠 Multi-agent orchestration across Diagnosis, Drug, Research, Lifestyle & Imaging
🔍 Semantic evaluation for accuracy and ethical alignment
💬 Conversational memory for context continuity
🖼️ File and image understanding with EasyOCR + BLIP
💡 Hybrid multimodal outputs with medical diagrams & text
🔐 Privacy-preserving, audit-ready architecture

---

🔁 Feature Matrix: Previous vs Streamlit Edition

---

🧬 Introduction

In modern healthcare, artificial intelligence is rapidly evolving from passive information retrieval to interactive medical reasoning systems. However, many existing medical chatbots depend on external cloud APIs and unverified data sources, leading to potential privacy violations, latency, and unreliable medical responses.
To address these challenges, MEDIBOT introduces a multi-agent, retrieval-augmented, and privacy-focused medical conversational framework that operates under strict domain control and ethical safety constraints.

At its core, MEDIBOT is an intelligent healthcare assistant built using LangChain, FAISS, and Google Gemini large language models (LLMs). It employs a multi-agent orchestration pipeline to deliver accurate, explainable, and context-aware medical insights. Each agent is designed to handle a specific clinical sub-domain—ranging from diagnosis, drug information, and lifestyle guidance to medical research summarization and image-based interpretation.

---

⚙️ System Overview

The MEDIBOT system is divided into three major layers:

User Interaction Layer (Streamlit UI)

• Provides a clean, interactive chat interface for both professionals and students.

• Supports file uploads (PDF, DOCX) and image uploads (X-ray, prescription scans) through EasyOCR and BLIP, allowing the chatbot to analyze documents and images directly.

• Displays real-time agent reasoning, chat history, and semantic evaluation feedback.

• Agent Orchestration Layer (LangChain Multi-Agent Framework)

Implements domain-specific agents, including:

• 🧠 Diagnosis Agent – Identifies possible conditions or symptoms from patient input.

• 💊 Drug Information Agent – Retrieves dosage, side effects, and interactions from medical knowledge bases.

• 🧘 Lifestyle Agent – Provides personalized recommendations for diet, fitness, and wellness.

• 📚 Research Agent – Summarizes academic studies and medical literature.

• 🖼️ Image Agent – Interprets uploaded medical images via BLIP or visual models.

The controller agent manages conversation flow, delegates tasks, and ensures responses stay within medical boundaries.

Retrieval & Memory Layer (FAISS + Semantic Evaluation)

• Uses FAISS vector storage to retrieve factual medical information and prior conversation context.

• Embeddings are generated using HuggingFace Sentence Transformers (MiniLM-L6-v2) for efficient semantic search.

• Implements semantic similarity scoring to validate factual accuracy and improve response relevance.

• Maintains short- and long-term memory for coherent multi-turn dialogue.

---

🔐 Privacy and Data Handling

Unlike conventional chatbots, MEDIBOT is designed to run either locally or with controlled API calls, ensuring that sensitive medical data never leaves the user’s environment.

• All uploaded files, embeddings, and logs are stored in local directories (./vectorstore/, ./outputs/).

• API keys are loaded from a .env file and never logged or transmitted externally.

• A domain enforcement layer filters out unrelated or unsafe prompts before processing.

This architecture enables offline or semi-offline operation, making it compliant with healthcare privacy standards and ideal for educational, diagnostic, and research contexts.

---

💡 Core Technologies

---

🌍 Purpose and Vision

The overarching vision behind MEDIBOT is to make medical AI assistance both ethical and accessible.
By running locally and maintaining full transparency in its reasoning process, MEDIBOT ensures that healthcare AI remains trustworthy, explainable, and privacy-respecting.
Its modular design also allows it to scale to new medical domains or integrate with electronic health systems in the future.

“MEDIBOT transforms medical dialogue into a secure partnership between AI intelligence and human expertise—bridging data ethics with digital care.”

---

Methodology

🏗 System Architecture Overview

MEDIBOT is built on a layered, modular design that combines retrieval-based intelligence, domain-specific agent workflows, and memory-aware conversation control.

The system starts with a user query, which is first screened through a Medical Domain Filter. Based on the detected intent (e.g., symptoms, drugs, diet, research), the query is routed to the corresponding specialized agent. Each agent performs its own reasoning, retrieval, or generation task and returns a structured medical response.

At its core, the LangChain ConversationalRetrievalChain powers MEDIBOT’s reasoning loop, supported by:

FAISS Vector Store for semantic medical knowledge retrieval.

Google Gemini 2.0 Flash for controlled, evidence-based natural language generation.

ConversationBufferMemory and summary mechanisms for context retention.

---

🧩 Agents and Roles

---

Detailed Responsibilities

Diagnosis Agent

1. Performs symptom extraction using regex and heuristic matching.

2. Searches PubMed via E-utilities and fetches recent abstracts.

3. Creates an on-the-fly FAISS vector store for symptom-specific retrieval.

4. Generates differential diagnoses, suggested tests, and clinical reasoning.

Drug Information Agent

1. Calls OpenFDA endpoints for official drug labeling data.

2. Displays indications, dosage, contraindications, and warnings.

3. Presents information in an easy-to-read, structured format.

BMI Agent

1. Extracts height and weight values from text.

2. Calculates BMI and classifies user into WHO categories.

3, Returns actionable diet and health advice.

Lifestyle & Prevention Agent

1. Uses Gemini to detect intent (exercise, diet, or mixed).

2. Fetches real exercises and food suggestions from WGER.

3. Generates motivating, structured wellness plans with emojis and personalization.

Medical Research Agent

1. Retrieves top scientific articles from Europe PMC API.

2. Summarizes study highlights, authors, journals, and publication years.

3. Generates a readable summary or detailed synthesis based on query type.

Image Agent

1. Creates labeled medical diagrams via Gemini image generation.

2. Falls back to Stable Diffusion XL if Gemini API quota is exceeded.

3. Saves outputs with timestamps for reproducibility and visual education.

🧠 Memory & Semantic Evaluation

1. Conversation Memory: Tracks 5 most recent dialogue turns using LangChain’s ConversationBufferMemory.

2. Summary Memory: Compresses older chats into a brief context string for continuity.

3. Semantic Evaluation: Measures factual alignment between retrieved context and generated answers using cosine similarity on HuggingFace embeddings.

4. Non-Medical Filter: Blocks unrelated, political, or humorous inputs to maintain medical domain integrity.

---

🛠 Tool Integration

---

The Architecture at a Glance

To help visualize how the system works internally, consider the following architecture diagram:

---

🔧 Project Structure Snapshot

---

Experiments

📄 Output (Diagnosis Example)

User: “I have fever, sore throat, and body ache.”
Agent Activated: Diagnosis Agent
Output:

Possible Diagnoses: Influenza, Streptococcal Pharyngitis, Viral Infection

Recommended Tests: Throat swab, CBC, PCR

Reasoning: Based on symptom cluster, most likely viral etiology with supportive management.

Sources: PubMed abstracts from 2022–2024

🧬 Output (Drug Information Example)

User: “Tell me about Insulin”
Agent Activated: Drug Info Agent
Output Summary:

Brand & Generic: Tylenol | Paracetamol

Indications: Fever, mild to moderate pain relief

Dosage: 500 mg every 6 hours (max 4g/day)

Contraindications: Liver disease, alcohol use

Source: FDA Drug Label Database

🩻 Output (Image Generation Example

User: “Generate a diagram of the human heart.”
Agent Activated: Image Agent
Output:

Generated via Gemini 2.5 Flash Preview

Backup available via Stable Diffusion XL

---

🔧 Installation & Quick-Start Guide

Follow these steps to get the MEDIBOT Multi-Agent System running on your local machine.

Clone the repository

git clone https://github.com/pamuarun/MEDIBOT-Multi-Agent-System.git
cd MEDIBOT-Multi-Agent-System

Create a virtual environment (optional but recommended)

python -m venv .venv
source .venv/bin/activate # on macOS/Linux
.venv\Scripts\activate # on Windows

Install Python dependencies

pip install -r requirements.txt

Add your API keys in the .env file
Example:
GOOGLE_API_KEY=your_gemini_api_key
HF_TOKEN=your_huggingface_token

Run the Streamlit app

streamlit run app.py

🩺 Conclusion

MEDIBOT stands as a testament to how cutting-edge AI innovation can coexist with medical ethics, privacy, and transparency.
Through the seamless fusion of LangChain’s RAG pipeline, multi-agent orchestration, and an intuitive Streamlit interface, it demonstrates that healthcare AI can be both powerful and principled.

This architecture not only ensures factual precision and domain safety but also redefines the standard for responsible AI in medicine—where every response respects the sanctity of data and the trust of its users.
By bridging intelligent automation with human-centered design, MEDIBOT lays the groundwork for a new era of ethical, explainable, and privacy-preserving digital healthcare.

"The future of healthcare isn’t about replacing doctors with AI it's about empowering them with intelligence that respects privacy, ethics, and humanity."

---

🧭 Significance & Implications

By orchestrating specialised medical agents under strict domain control, MEDIBOT demonstrates that clinically relevant, multimodal, and context-aware healthcare dialogue can be achieved without exposing data to external APIs or sacrificing patient privacy.
The system validates that retrieval-augmented reasoning and domain-filtered AI workflows can coexist in a secure, local, and explainable architecture—bridging the gap between AI innovation and medical data ethics.

---

🛡️ Security, Privacy & Safety

Threat Model & Trust Boundaries

• Local Execution: All inference runs locally through Gemini API or local vector retrieval; no third-party data transmission by default.

• Data Isolation: Uploaded medical files, chat logs, and embeddings are stored only within the project workspace (./vectorstore/ and ./outputs/).

• Cost Efficiency: No recurring API usage fees for embeddings or retrieval; local FAISS ensures offline querying.

• Explicit Trust Zones: UI ↔ Orchestrator ↔ Agents ↔ Vectorstore ↔ Storage — clear, auditable separation of responsibilities.

---

Controls

• Input Hardening: Uploaded files validated by type (PDF, DOCX, image) and size before parsing.

• Prompt Safety: Medical domain enforcement filters reject unrelated or sensitive non-medical queries.

• Tool Allow-Listing: Only authorised tools (EasyOCR, BLIP, FAISS retriever, Gemini LLM) are permitted; each runs with timeouts and exception guards.

• Output Filtering: Removes PHI/PII patterns, profanity, and non-medical outputs.

• Secrets Handling: .env variables (API keys) loaded once at runtime; never stored in memory dumps or logs.

• Logging & Audit: Structured logs include timestamps, run IDs, and agent traces; medical responses can be traced back to vector citations.

• Dependency Security: Periodic pip-audit scans; minimal trusted library set.

• Sandboxing: Streamlit session isolated; no shell command execution; file I/O constrained to app directories.

---

⚖️ Compliance Alignment

• Privacy: Adheres to data-minimization and purpose-limitation principles—inputs processed locally, no telemetry sent externally.

• Healthcare Ethics: Implements domain-specific content control and factual validation aligned with clinical communication norms.

• GDPR & HIPAA Readiness: Local storage only, optional data-purge (./outputs/ cleanup), and transparent logging for audit trails.

• Transparency: Each response includes optional reference sources from FAISS retrieval for user verification.

---

🔐 Security Test Cases (in Continuous Validation)

---

📜 Licensing & Usage Rights

This project is released under the GNU General Public License v3.0 (GPL-3.0)

✔️ Commercial & private use
✔️ Distribution & modification
✔️ Patent use

Permissions of this strong copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights.

A full license text is available in the accompanying LICENSE file.
When redistributing, please retain copyright
and attribution notices.

Model License:

External LLMs and APIs (e.g., Google Gemini, Hugging Face, WGER, OpenFDA) are governed by their respective creators’ licenses.
Ensure compliance when integrating or extending third-party components.

---

🌐 Access to Technical Assets

---

Results

---