TL;DR This production-ready evolution of the **LangGraph-Orchestrated Research Assistant** builds on the initial Ready Tensor prototype with major upgrades for safety, reliability, and usability. Key enhancements include: * **Guardrails:** Input/output validation, prompt protection, and structured responses. * **Observability:** Advanced logging, feedback capture, and performance monitoring. * **Deployment:** Streamlit app, Docker support, and cloud hosting readiness. * **Documentation:** Detailed guides, walkthroughs, limitations, and safety notes. ## Tool Overview & Architecture ### System Architecture Goals The upgraded system transforms the original LangGraph-Orchestrated Research Assistant prototype into a **production-grade, reliable, and user-friendly** multi-agent AI application. Built on **LangChain** and **LangGraph**, it combines structured agent orchestration, strong safety measures, and robust deployment capabilities to handle real-world usage at scale. The architecture is driven by four primary production-focused goals: --- ### 1. Safety & Reliability Guardrails * **Input/Output Validation:** Ensures only valid, well-structured data is processed. * **Prompt Protection:** Shields agents from prompt injection or adversarial input. * **Structured Response Constraints:** Enforces consistent JSON or table-based outputs for downstream processing. --- ### 2. Observability & Monitoring * **Enhanced Logging:** Tracks agent decisions, request/response cycles, and error events. * **Feedback Capture:** Allows users to submit corrections or improvements. * **Instrumentation:** Collects performance metrics to optimize execution time and accuracy. --- ### 3. Scalable Deployment & Accessibility * **Streamlit Web App:** Delivers an interactive and user-friendly interface. * **Docker Support:** Ensures reproducible builds and containerized deployment. * **Cloud Hosting Options:** Supports multiple deployment environments for scalability. --- ### 4. Comprehensive Documentation & Onboarding * **Usage Guides:** Clear instructions for running and interacting with the system. * **Step-by-Step Walkthroughs:** Detailed workflows for key tasks. * **Limitations & Safety Considerations:** Transparent disclosure of known boundaries. --- ### Workflow The production-ready workflow mirrors the original LangGraph orchestration but with added **guardrails and observability hooks** at each stage: 1. **User Input** is validated and preprocessed. 2. **LangGraph State Machine** routes requests to specialized agents (extraction, comparison, enrichment, fact-checking). 3. **Guardrails** filter and constrain outputs before passing them downstream. 4. **Streamlit UI** displays results alongside orchestration diagrams generated via **Mermaid** and **Graphviz**. 5. **Logs & Metrics** are captured for monitoring and future tuning. --- If you’d like, I can now **merge this with the previous “TL;DR” architecture** so you have a single, unified document for your research assistant. --- ## Productionizing the Multi-Agent System ### Guardrails [Guardrails-AI](https://hub.guardrailsai.com/) enforces structured, validated outputs from LLMs. #### Comparison: Without vs. With Guardrails-AI | Feature | Without Guardrails-AI ❌ | With Guardrails-AI ✅ | |----------------------|------------------------------|-----------------------------------| | Output format | Free-form, unstructured text | Schema-validated JSON | | Input flexibility | High | Moderate (requires schema) | | Retry on invalid | No | Yes | | Production-safety | Risk of LLM drift | Robust | | Dev speed | Fast | Requires setup effort | #### Integration Steps 1. Install `guardrails-ai`. 2. Create a `.rail` schema for publication profiling. 3. Update analysis functions (e.g., `PublicationExplorer.analyze_pub1()`, `PublicationExplorer.analyze_pub2()` in `src/explorer.py`). 4. Validate outputs before updating state. **Validated output includes:** - `tools`: Frameworks/libraries extracted from publication - `evaluation_methods`: Metrics or strategies described - `datasets`: Benchmarks used - `task_types`: Types of tasks addressed - `results`: Key findings All fields must be present for validation. Structured, machine-readable JSON enables downstream analytics and reproducibility. --- ### Observability Observability is essential for monitoring, tracing, and debugging during real-world deployment. Logging and monitoring use Python’s logging module and node tracing. #### Key Observability Features | Feature | Tool/Technique | Purpose | | --------------- | ------------------------------------ | ---------------------------------------- | | Structured Logs | Python’s `logging` + JSON formatting | Centralized log collection & readability | | Tracing | `uuid` session ID + node info | Track LLM invocation paths & failures | #### Implementation Steps 1. Install logging tools. 2. Create `logger.py`. **Enhancements Include:** - Structured logging using loguru - Session traceability via uuid - Centralized logs to console and file - Logging across all major graph nodes --- ### Deployment The application runs locally via [Streamlit](https://streamlit.io), or can be deployed to cloud platforms (Streamlit Cloud, [Docker](https://www.docker.com/), [Hugging Face Spaces](https://hf.co/spaces)). --- # Improvements Over the Prototype | Feature | Before (Prototype) | After (Production) | |----------------------|-------------------------------|-----------------------------------------------| | Guardrails | ❌ None | ✅ .rail schema with fallback | | Logging | ⚠️ Basic | ✅ Structured logs with timestamps | | Output Separation | ❌ Mixed outputs | ✅ Separate folders: outputs/profiles/, outputs/comparison/ | | Validation | ❌ Ad-hoc | ✅ Schema-driven, strict validation | | Resilience | ❌ Fragile | ✅ Robust with fallbacks and logs | | Deployment | ❌ Prototype | ✅ Docker & cloud ready | | Interface | ⚠️ Exposed API key | ✅ Clean sidebar info, no key exposure | | Documentation | ⚠️ Incomplete | ✅ README, deployment.md, code docstrings | > _Note: "Before" refers to the workflow in the original publication._ --- ## Repository Structure ```text /Agentic_AI_Developer_Certification_Project3-main ├── LICENSE ├── README.md # Project overview and instructions ├── deployment.md # Deployment guide for various environments ├── Dockerfile # Instructions for building a Docker image ├── requirements-test.txt # Development and test dependencies ├── requirements.txt # Project dependencies ├── .gitignore # Ignored files and folders ├── .env.example # Example environment file for API keys ├── data/ │ ├── project_1_publications.json # Sample Ready Tensor dataset │ ├── sample_publications/ # Input publication `.txt` files ├── docs/ │ ├── Untitled diagram _ Mermaid Chart-2025-07-09-115351.png │ ├── langgraph_flowchart.mmd │ ├── publication_flowchart.png ├── examples_screens/ # Streamlit interface screenshots ├── logs/ # Runtime log output │ └── pipeline.log ├── outputs/ # Validated profiles and comparison results │ ├── profiles/ │ └── comparisons/ ├── src/ # Source code │ ├── app.py # Main Streamlit App │ ├── explorer.py # LLM-based publication comparison engine │ ├── generate_flowchart_graphviz.py │ ├── generate_flowchart_mermaid.py │ ├── loader.py # Converts JSON into individual .txt files │ ├── paths.py # Centralized path definitions │ ├── utils.py # Helper functions │ ├── logger.py # Centralized log configuration │ ├── docs/ │ │ ├── langgraph_flowchart.mmd │ │ ├── publication_flowchart.png │ ├── rails/ # Guardrails XML schemas │ │ ├── profile_extraction.rail ``` ## Installation Instructions --- ## Prerequisites - Python 3.10+ - [OpenAI API key](https://platform.openai.com/account/api-keys) - [Tavily API key](https://www.tavily.com/) - Set as `OPENAI_API_KEY` and `TAVILY_API_KEY` environment variables --- ## Installation 1. **Clone the repository** ```bash git clone https://github.com/micag2025/Agentic_AI_Developer_Certification_Project3 cd Agentic_AI_Developer_Certification_Project3 ``` 2. **Install dependencies** ```bash pip install -r requirements.txt # For testing: pip install -r requirements-test.txt ``` > _Note: Test dependencies are separated from runtime dependencies._ 3. **Set up environment variables** - Copy `.env.example` to `.env` and add your OpenAI and Tavily API keys. 4. **(Recommended) Use a virtual environment** ```bash python3 -m venv .venv source .venv/bin/activate # Linux/macOS .\.venv\Scripts\activate # Windows ``` --- ## Running the Application 1. Ensure `project_1_publications.json` is present in `data/`. > _The sample dataset is available in the "Datasets" section of the related publication._ 2. Launch the Streamlit app: ```bash streamlit run src/app.py ``` 3. Open your browser to the local Streamlit URL (usually http://localhost:8501). You can now interact with the LangGraph-Orchestrated Research Assistant for Ready Tensor! --- **Output Locations** - Validated Profiles: `outputs/profiles/*.json` - Comparison Reports: `outputs/comparisons/*.json` and `.html` - Log Files: `logs/*.log` Download the latest validated profile and log file directly from the Streamlit interface. --- **Debugging Guardrails Integration** Run the app and monitor the terminal for raw vs. validated outputs. The pipeline will: - Trigger the PublicationExplorer - Invoke `analyze_pub1` and `analyze_pub2` - Print both raw and validated outputs in the terminal --- # Usage Examples ### Example: Validated Profile (Guardrails-AI) Sample output in `outputs/`: ```json { "tools": ["LangGraph", "Microsoft AutoGen"], "evaluation_methods": [], "datasets": [], "task_types": ["AI Agent", "Autonomous Agents", "Multi-Agent System"], "results": [] } ``` If `evaluation_methods` is empty, this means the publication lacks evaluation details or the model did not extract any. Guardrails validated the output using the `.rail` schema. --- ### Example: Observability (Logging, Tracing, Monitoring) Logs can be found in `/logs`. System logs like a production-grade application, capturing: - Function, line, module, process, thread, timestamps - Informative tags (📊, 📈, 📝, 🔍, 🤖) - Both model responses and validated outputs --- ### Example: Deployment (Streamlit Cloud, Docker, CLI) - Ensure `outputs/profiles/`, `outputs/comparisons/`, and `logs/` directories exist and are writable. - Validated profiles and logs are downloadable via the UI. - For Docker, map local volumes as needed. --- ### Streamlit App Example 1. Launch the UI: ```bash streamlit run src/app.py ``` 2. Navigate to http://localhost:8501 to access the Research Assistant. ![1_Screenshot_improved_initial_StreamLit_interface.jpeg](1_Screenshot_improved_initial_StreamLit_interface.jpeg) 3. Choose two publications from the dropdown menus. Now, the two publications are selected from the Ready Tensor dataset and enter a Query E.g., “Evaluation Methods” or “Dataset comparisons”. And the publications are analyzed independently to extract relevant metadata. ![3_Screenshot_improved_Streamlit_example_usage1_processing.jpeg](3_Screenshot_improved_Streamlit_example_usage1_processing.jpeg) 5. Overall final output Finally, we get a structured response ready for user presentation off of the workflow ![5_Screenshot_improved_message_results_saved_example_usage1.jpeg](5_Screenshot_improved_message_results_saved_example_usage1.jpeg) ## References - [LangGraph](https://www.langchain.com/langgraph) - [LangChain](https://www.langchain.com/langchain) - [Openai API](https://huggingface.co/learn/agents-course/unit2/langgraph/introduction?utm_source=chatgpt.com) - [Streamlit](https://docs.streamlit.io/) - [Tavily](https://www.tavily.com/) - [Mermaid](https://mermaid.js.org/) -[Guardrails](https://www.guardrailsai.com/) - [Ready Tensor Certifications](https://app.readytensor.ai/hubs/ready_tensor_certifications) - [Technical Evaluation Rubric](https://app.readytensor.ai/publications/WsaE5uxLBqnH) - [Engage and Inspire: Best Practices for Publishing on Ready Tensor](https://app.readytensor.ai/publications/engage_and_inspire_best_practices_for_publishing_on_ready_tensor_SBgkOyUsP8qQ) ## Contributing We welcome enhancements and issue reports: 1. Fork the repo 2. Create a feature branch (git checkout -b feature-name) 3. Commit & Push your changes 4. Submit a Pull Request Please adhere to existing code style and update documentation as needed. # Future Implementations We encourage and welcome community contributions to help make this assistant even more robust and versatile. Here are several potential areas for future improvement, along with example contributions for each: --- ### **Integrate Additional LLM Providers** Expand the system to support multiple large language model backends (e.g., OpenAI, Anthropic Claude, Google Gemini), allowing users to select or compare across different models. * **Example:** Add support for Anthropic Claude by creating a new API wrapper in `src/utils.py` and updating `explorer.py` to enable backend selection. --- ### **Extend the UI for Richer User Interaction** Enhance the user interface to provide more flexibility and usability. * **Example:** Add a sidebar widget for filtering comparison results by publication topic, or implement functionality for uploading new publication files directly via the Streamlit interface. --- ### **Design New Guardrail Schemas for Various Research Domains** Adapt and extend validation schemas to better handle domain-specific needs. * **Example:** Create a new `.rail` schema for extracting information from specific types of publications, and update the validation logic in `explorer.py` to accommodate new domain-specific fields. --- ### **Integrate Fine-Tuned or Domain-Adapted LLMs** Improve accuracy in extraction, summarization, and reasoning tasks by supporting fine-tuned or specialized LLMs. * **Example:** Add configuration options for loading and selecting fine-tuned models, and develop benchmarking tools to evaluate their performance in the publication profiling pipeline. --- ### **Investigate and Address Missing Evaluation Methods** Enhance reliability by investigating why the system sometimes fails to extract evaluation methods from publications. * **Example:** * **LLM Extraction Limitations:** The LLM may miss evaluation methods due to vague or implicit phrasing in the source text. * **Preprocessing (Truncation):** If publication text is truncated (e.g., `MAX_CHARS = 12000`), key evaluation sections may be omitted from analysis. Improving data preprocessing or LLM prompts could mitigate this issue. ## License Licensed under the [MIT License](https://opensource.org/license/mit) ## Contact For questions or feedback, please contact the authors: * chibueze.k.muoneke@gmail.com * michelaagostini73@gmail.com * nyajuaya.j.a@gmail.com ## Acknowledgments This work is part of the Agentic AI Developer Certification program by [ReadyTensor](https://readytensor.ai/) Special thanks to the Ready Tensor developer community for guidance and feedback.