Certifiedunder the Multi-Agent System module in the Mastering AI Agents program.Certifiedunder the Multi-Agent System module in the Mastering AI Agents program.The Multi-Agent System for Insurance Cancellation is designed to automate the process to cancel insurance policy for customers. The system is built using Python and the LangGraph orchestration framework. Using mock data stored in CSV files, multiple specialized agents collaborate to validate policy details, evaluate cancellation eligibility based on predefined business rules, calculate refundable amounts, log refund records, and generate a professional cancellation notice in PDF format. This system demonstrates key concepts including agent orchestration, tool integration, shared state management, and human-in-the-loop interactions, while remaining simple and reproducible without reliance on external databases. The system incorporates structured error handling, tool validation, retry mechanisms, and fault-tolerant orchestration to ensure production-grade robustness.
Insurance policy cancellation is common but operationally sensitive process that requires accurate data validation, strict rule enforcement, and clear communication with customers. The goal of this project is to model the cancellation process using a multi-agent AI system while keeping the implementation lightweight, transparent, and suitable for learning purposes.
This project implements a multi-agent insurance policy cancellation system using the LangGraph framework in Python. Instead of connecting to a live database, the systems operates on mock customer and policy data stored in a CSV file, allowing the full workflow to be simulated in a controlled environment. Each agent in the system is responsible for a clear defined task, such as customer intake, policy analysis, refund calculation, refund record logging, and professional notice generation.
Large language models are used selectively where natural language understanding and generation add value, such as interacting with users via a command-line interface and generating formal cancellation notices. In contrast, critical business rules—such as eligibility checks and refund calculations—are implemented using deterministic logic to ensure correctness and auditability. Agents communicate through a shared state managed by LangGraph, enabling structured coordination and clear progression through the cancellation lifecycle.
By combining agent-based orchestration, custom tool integration, and LLM-powered communication, this project demonstrates how AI agents can be applied to enterprise-style workflows in a safe, explainable, and modular manner. The resulting system serves as both a practical learning artifact and a foundation for extending into more advanced features such as email delivery, monitoring, or web-based user interfaces.
The workflow integrates:
The workflow consists of multiple specialised agents that collaboratively process cancellation requests:
The Multi-Agent insurance cancellation system is implemented as a multi-agent workflow using the LangGraph framework, where each agent operates on a shared state object, allowing structured communication and progressive enrichment of information throughout the workflow.
The workflow begins with a command-line interface (CLI) interaction and proceeds through a sequence of specialized agents, terminating once a final cancellation and refund notice has been generated and stored.
The system consists of multiple agents, each with a clearly defined responsibility.
This agent is to collect the insurance policy number from the user, validates its existence in the mock dataset, and confirms customer details before proceeding. It interacts directly with the user via the CLI and validates the existence of the policy by querying a mock CSV dataset using a custom data lookup tool. This agent uses a large language model (LLM) to guide conversational interaction, ask clarification questions, and present retrieved policy details for confirmation.
If the policy number provided can be found in the mock dataset, the agent will retrieve the policy details for the customer and confirm the customer if policy details are correct or not. If the policy number can't be found, the agent will request the customer to enter the correct policy number.
The Analysis Agent evaluates whether a policy is eligible for cancellation based on predefined business rules. These rules include checking whether the policy is active, whether payment has been made, and whether the current date is before the policy end date. This agent does not use an LLM; instead, it relies on deterministic Python logic implemented as a custom cancellation_rules tool to ensure correctness, transparency, and auditability.
After eligibility evaluation, control is passed to a human review step for eligibility approval.
The Refund Agent calculates the refundable amount for eligible policies. Refunds are computed using a pro-rata approach based on the remaining policy duration and the total payment amount. This agent uses a custom refund calculation tool and does not rely on an LLM, ensuring deterministic financial computation.
After refund calculation, the workflow transitions to a second human review step.
The Logger Agent persists approved refund decisions into a CSV file stored in the output directory. Logged data includes the policy number, refund eligibility, refund amount, decision date, and relevant customer information. This agent uses a custom CSV logging tool and is executed only after human approval of the refund.
Logs include:
The Summary Agent generates a professional insurance cancellation or refund notice. It uses an LLM to produce well-structured, formal language suitable for customer communication. The generated notice is then exported as a PDF file using a custom PDF generation tool. This agent represents the final customer-facing output of the system.
Outputs:
Human-in-the-loop interactions are intentionally placed at key decision boundaries within the workflow to enhance safety, accountability, and trust. Two HITL checkpoints are implemented:
Following the Analysis Agent, a human-in-the-loop (HITL) checkpoint is introduced. At this stage, a human reviewer examines the eligibility decision and explicitly approves or rejects the policy cancellation. Only policies approved at this step are allowed to proceed to refund calculation.
Outcomes:
The refund amount calculated by the Refund Agent is reviewed by a human reviewer. This step ensures that financial outcomes are verified before any permanent record is created. Only when the refund is approved does the workflow proceed to refund logging.
Outcomes:
Evaluation uses DeepEval, with test cases for five dimensions (see code/evaluation/deepeval_eval.py):
This project uses a CSV file named insurance_policies.csv which is stored under /data folder to mock customer and policy data.
A couple of customer tools are implemented to handle data lookup (data_lookup.py), cancellation rules definition (cancellation_rules.py), refund calculation (refund_calculator.py), refund logging (refund_logger.py), and PDF generation (notice_generator.py).
Large language models are used only where natural language interaction or generation is required, such as user communication and notice creation. All business-critical logic, including eligibility rules and refund calculations, is implemented using deterministic Python code to ensure correctness and explainability.
Policy cancellation eligibility is determined using the following rules:
1. The policy status must be active.
2. The policy payment must be marked as paid.
3. The current date must be earlier than the policy end date.
Only policies that satisfy all rules are eligible for cancellation and refund processing.
To evaluate the system, multiple test scenarios were executed using the CLI interface and mock CSV data. These scenarios were designed to simulate realistic customer interactions and edge cases.
The following test cases were evaluated:
Valid and Eligible Policy: Active policy, payment completed, and end date in the future.
Inactive Policy: Policy already cancelled.
Unpaid Policy: Active policy with payment not completed.
Expired Policy: Policy end date earlier than the current date.
Invalid Policy Number: Policy number not present in the dataset.
Incorrect Customer Confirmation: User rejects the retrieved policy information and re-enters the policy number.
Human Review Rejection: Human rejects the request and the workflow ends.
Human Review Approval: Human approves the request and the workflow proceeds to the next phase.
Each scenario was executed multiple times to confirm consistent agent behavior and state transitions.
git clone https://github.com/jingozuo/AAIDC_Project2_JZ.git
pip install -r requirements.txt
//Linux/macOS cp .env.example .env or //Windows copy .env.example .env
https://github.com/jingozuo/AAIDC_Project2_JZ.git
Mock policy dataset located under:
/data/insurance_policies.csv
https://docs.langchain.com/docs/langgraph
Documents located under:
outputs/refund_log.csv outputs/Cancellation_Notice_<policy>.pdf
class InsuranceCancellationState(TypedDict, total=False): """State for the insurance cancellation graph.""" phase: Phase policy_details: PolicyDetails user_input: str messages: Annotated[list[AnyMessage], add_messages] output: str invalid_policy_attempts: int #HITL fields human_decision: Optional[Literal["approve", "reject"]] human_decision_reason: Optional[str] hitl_checkpoint: Optional[str]
In this project, Phase should be also defined. Phases include:
"ask_policy", "awaiting_policy", "confirm_customer", "ready_for_analysis", "human_eligibility_check", "ready_for_refund", "human_refund_check", "ready_for_summary", "summary_complete", "end"
def lookup_policy_in_csv(policy_number: str): """ Look up a policy in the CSV file by policy number. Returns a dict with keys like "First Name", "Policy Number", etc., or None if not found. """ with open(DATA_FILE_PATH, "r", encoding="utf-8") as f: reader = csv.DictReader(f) for row in reader: if row.get("Policy_Number", "").strip() == policy_number: return {CSV_KEY_MAP.get(k, k): v.strip() if isinstance(v, str) else v for k, v in row.items()} return None
def check_cancellation_eligibility(policy_details: Dict[str, Any]) -> Tuple[bool, str]: """ Check the cancellation rules for the policy. Returns: - True if the policy is eligible for cancellation - False if the policy is not eligible for cancellation - Reason for the eligibility or non-eligibility """ xxxxx return True, "Policy is eligible for cancellation"
def calculate_refund_amount(policy_details: Dict[str, Any]) -> Tuple[bool, str, float]: """ Calculate the refund amount for the policy. Returns: - True if the refund amount is calculated successfully - False if the refund amount is not calculated successfully - Reason for the calculation or non-calculation """ xxxxx return True, "Refund amount calculated successfully.", round(refund_amount, 2)
def log_refund_record(policy_details: Dict[str, Any], refund_amount: float, refund_reason: str): """ Log the refund details to the refund log file. """ xxxxx return True
def generate_notice_pdf(policy_details: Dict[str, Any], refund_amount: float, refund_reason: str, notice_text: str) -> str: """ Generate a notice as a PDF with title, description, then notice details. """ xxxxx return file_path
Each node is just a function that takes in the state and returns updates. Here're some example below:
def make_intake_agent_node(llm_model:str, llm_intake_prompt_config: Dict[str, Any]) -> Callable[[Dict[str, Any]], Dict[str, Any]]: """ Ask the user for their policy number and confirm if they want to proceed with cancelling the policy. """ llm = llm_model def run_llm(messages): response = llm.invoke(messages) return response def intake_node(state: InsuranceCancellationState) -> Dict[str, Any]: print("\n") print(" → Intake node running (phase=%s)" % state.get("phase", "ask_policy")) xxxxx xxxxx return intake_node def make_analysis_agent_node(): """ Analyze the policy and determine if it can be cancelled. """ def analysis_node(state: InsuranceCancellationState) -> Dict[str, Any]: print("\n") print(" → Analysis node running (phase=%s)" % state.get("phase", "ready_for_analysis")) xxxxx xxxxx return analysis_node def make_refund_agent_node(): """ Calculate the refund amount for the policy. """ def refund_node(state: InsuranceCancellationState) -> Dict[str, Any]: print("\n") print(" → Refund node running (phase=%s)" % state.get("phase", "ready_for_refund")) xxxxx xxxxx return refund_node def make_log_refund_node(): """ Log the refund record after human has approved the refund. """ def log_refund_node(state: InsuranceCancellationState) -> Dict[str, Any]: xxxxx xxxxx return log_refund_node def make_summary_agent_node(llm_model:str, llm_summary_prompt_config: Dict[str, Any]) -> Callable[[Dict[str, Any]], Dict[str, Any]]: """ Generate a summary of the notice. """ llm = llm_model def summary_node(state: InsuranceCancellationState) -> Dict[str, Any]: print("\n") print(" → Summary node running (phase=%s)" % state.get("phase", "ready_for_summary")) xxxxx xxxxx return summary_node def make_hitl_node(checkpoint_name: str): """ Make a human in the loop agent to approve or reject the refund. """ def hitl_node(state: InsuranceCancellationState) -> Dict[str, Any]: xxxxx xxxxx return hitl_node
from nodes import ( make_intake_agent_node, make_analysis_agent_node, make_refund_agent_node, make_log_refund_node, make_summary_agent_node, make_hitl_node ) xxxxx def build_insurance_cancellation_graph(llm_model: str, prompt_config: Dict[str, Any]) -> StateGraph: """ Build the insurance cancellation graph. """ #Create the graph graph = StateGraph(InsuranceCancellationState) #Add the nodes xxxxx graph.add_node("intake", intake_node) graph.add_node("analysis", analysis_node) graph.add_node("eligibility_hitl", eligibility_hitl_node) graph.add_node("refund", refund_node) graph.add_node("refund_hitl", refund_hitl_node) graph.add_node("log_refund", log_refund_node) graph.add_node("summary", summary_node) #Add the edges graph.add_edge(START, "intake") #Intake routes graph.add_conditional_edges( "intake", route_from_intake, { "intake": "intake", "analysis": "analysis", "end": END, }, ) #Analysis routes -> eligibility HITL graph.add_conditional_edges( "analysis", route_from_analysis, { "eligibility_hitl": "eligibility_hitl", "end": END, }, ) #Eligibility HITL routes -> Refund or End graph.add_conditional_edges( "eligibility_hitl", route_after_human, { "refund": "refund", "end": END, }, ) #Refund routes -> refund HITL graph.add_conditional_edges( "refund", route_from_refund, { "refund_hitl": "refund_hitl", "end": END, }, ) #Refund HITL routes -> Log refund (after approval) or End graph.add_conditional_edges( "refund_hitl", route_after_human, { "log_refund": "log_refund", "end": END, }, ) graph.add_edge("log_refund", "summary") #Summary routes -> End graph.add_edge("summary", END) xxxxx
print("🔍 Running graph...") graph = build_insurance_cancellation_graph(llm_model, prompt_config)
""" DeepEval evaluation for the insurance cancellation workflow. Evaluates five dimensions via GEval: 1. Eligibility correctness 2. Refund calculation correctness 3. Workflow sequencing 4. Summary notice quality 5. Agent boundary enforcement Use llama-3.3-70b-versatile (Groq) by default; set GROQ_API_KEY in .env. Groq does not support logprobs, so we use a wrapper that omits them (see GroqLiteLLMModel). """ xxxxx def import_deepeval(): try: from deepeval.test_case import LLMTestCase, LLMTestCaseParams from deepeval.metrics import GEval return LLMTestCase, LLMTestCaseParams, GEval except ImportError as e: raise ImportError("Evaluation requires: pip install deepeval") from e xxxxx def run_evaluation( samples: List[Dict[str, Any]], llm_model: str = "llama-3.3-70b-versatile", ) -> Dict[str, Any]: xxxxx return result
xxxxx llm_model = os.environ.get("EVAL_LLM_MODEL", "llama-3.3-70b-versatile") print("DeepEval (5 dimensions). Model:", llm_model) if llm_model == "llama-3.3-70b-versatile" and not os.environ.get("GROQ_API_KEY"): print("Warning: GROQ_API_KEY not set. Set it in .env or environment for Groq/Llama.") print() results = run_evaluation(MOCK_SAMPLES, llm_model=llm_model) print("Eligibility correctness: ", results.get("eligibility_correctness")) print("Refund correctness: ", results.get("refund_correctness")) print("Workflow sequencing: ", results.get("workflow_sequencing")) print("Summary notice quality: ", results.get("summary_notice_quality")) print("Agent boundary enforcement:", results.get("agent_boundary_enforcement")) print("Overall: ", results.get("overall")) print("Done.") xxxxx
// From the project root: python code/main.py or // From the code directory: cd code && python main.py
python -m code.run_evaluation
The Intake Agent Validate:
If invalid input is detected, the system re-prompts the user until valid input is provided or retry limits are reached.
The Analysis Agent enforces strict cancellation rules:
Rule evaluation uses deterministic logic rather than LLM inference to ensure financial correctness and traceability.
Two approval checkpoints are enforced:
System transitions only occur when approvals are explicitly granted.
Refund logging provides:
This ensures reproducibility and compliance.
The system successfully processed all tested scenarios with deterministic and explainable outcomes.
Scenario 1: Human review approval for refund
Happy Path - Human decides to approve the refund, then this refund record will be logged in and a notice PDF file will be generated.
Scenario 2: Entering invalid policy number
When a user enters an invalid policy number, Intake agent says "Policy number not found. Please provide a correct policy number".
If the user fails 3 attempts, Intake agent says "Maximum attempts (2) reached. Please try again later", and the workflow ends.
Scenario 3: User confirms the policy details are NOT correct
When a user enters a valid policy number and confirms the policy details are NOT correct, Intake agent says "The information was not correct. Please enter the correct policy number:" so the user needs to provide a correct policy number
Scenario 4: Policy is not active
When a user enters a valid policy number and confirms the policy details are correct, Analysis agent begins to check the cancellation eligibility. If the policy status is not active, Analysis agent says "Policy is not eligible for cancellation. Reason: Policy is not active", then the workflow ends.
Scenario 5: Payment has not beed made
When a user enters a valid policy number and confirms the policy details are correct, Analysis agent begins to check the cancellation eligibility. If the user haven't made the payment for the policy, Analysis agent says "Policy is not eligible for cancellation. Reason: Payment has not been made", then the workflow ends.
Scenario 6: Policy is eligible for cancellation
When a user enters a valid policy number and confirms the policy details are correct, Analysis agent begins to check the cancellation eligibility. If the policy is eligible for cancellation, it goes to Human Review for Eligibility.
Scenario 7: Human review rejection for eligibility
When Human Review rejects the eligibility, it shows "Human decides to reject eligibility, stopping the workflow", and the workflow ends.
Scenario 8: Human review approval for eligibility
When Human Review approves the eligibility, it shows "Human decides to approves eligibility, proceeding", and it goes to Refund agent to calculate the refund amount.
Scenario 9: Human review rejection for refund
When Human Review rejects the refund, it shows "Human decides to reject refund, stopping the workflow", and the workflow ends.
This project demonstrates the design and implementation of a multi-agent insurance cancellation system using Python and the LangGraph framework. By decomposing the workflow into specialized agents and combining deterministic business logic with selective LLM usage, the system achieves both reliability and flexibility.
The use of mock CSV data allows the system to remain lightweight and reproducible while still modeling realistic insurance operations. Custom tools for data lookup, refund calculation, audit logging, and PDF generation further extend the system beyond basic conversational AI capabilities.
The project successfully showcases agent orchestration, tool integration, shared state management, and human-in-the-loop interaction. The resulting system provides a strong foundation for future enhancements, such as email delivery, web-based interfaces, monitoring, and integration with real databases or external APIs.