Abstract

AI can now generate code faster than humans can review it. This creates a dangerous bottleneck where security risks and architectural flaws can hide inside large volumes of machine-generated software.

To address this problem, I built the Automaton Auditor (Emerald Suite v2.0) — a multi-agent LangGraph forensic swarm designed to govern code rather than generate it.

The system introduces a Digital Courtroom architecture where specialized AI agents analyze repositories, argue opposing interpretations, and synthesize a final verdict based on verifiable evidence.

By combining AST-based code analysis, strict Pydantic state contracts, and adversarial multi-agent reasoning, the Emerald Suite transforms manual code review into a scalable forensic service.

Introduction: The Scaling Paradox

Modern development has entered the era of “vibe coding.”

Developers can describe a system and AI generates thousands of lines of code instantly. The problem is that human review cannot scale at the same speed.
This leads to what I call Orchestration Fraud — when a system claims to do one thing but the underlying code tells another story.
The Emerald Suite addresses this by shifting the engineer’s role:
from writing code → to governing code.

My mission was to shift from being a "bricklayer" who writes code to an "architect" who governs it.

1. I built the Emerald Suite as a production-ready solution.
It is a "Glass Box" system. Every decision in our courtroom is explicit, traceable, and anchored in hard evidence rather than guesses.

2. Evaluation: Why a Digital Courtroom?
Traditional AI grading is broken. If you ask an LLM to rate code on a scale of 1 to 10, you get inconsistent "vibe" scores.
Instead of a single AI judge, the system creates structured disagreement between agents.
The Courtroom Model works better because LLMs are excellent at arguing specific positions. We use this to bridge the "Judicial Gap"—the space between seeing a file exists and judging its actual quality.

The Courtroom Roles:
The Prosecutor (Pessimistic): Philosophy is "Trust No One." Its mission is to find gaps, security flaws, and "Hallucination Liabilities."

The Defense (Optimistic): Focuses on the "Spirit of the Law." It identifies engineering intent and creative workarounds visible in the Git history.

The Tech Lead (Realistic): The pragmatic anchor. It evaluates code maintainability and whether the system is built to scale.

Methodology:

Architecture: Hierarchical multi-agent system (“Digital Courtroom”) with Detectives (RepoInvestigator, DocAnalyst, VisionInspector), Judges (Prosecutor, Defense, Tech Lead), and Chief Justice synthesizing final verdicts. Detectives collect structured evidence, Judges analyze it independently, Chief Justice resolves conflicts and produces the audit report.

Tools & Frameworks: Python, LangGraph for multi-agent orchestration, Pydantic for typed state, AST parsing for code analysis, RAG-lite PDF parsing for reports, sandboxed git clone using tempfile.

Installation:

git clone https://github.com/hydropython/forensic-swarm-auditor.git
cd forensic-swarm-auditor
uv install
cp .env.example .env

Running the Auditor: Provide a GitHub repo URL and a PDF report. The system outputs a structured audit report (Markdown/PDF) with scores, judge opinions, and remediation steps.

Workflow / Architecture Flow

Start & Sandbox: The process begins by initializing a forensic sandbox where the repository and related files are isolated for safe analysis. The AgentState Contract maintains typed, structured state throughout the workflow.

Dispatcher & Detectives: A dispatcher assigns tasks to multiple forensic detectives in parallel:

• Repo Detective: Examines the repository structure and code.

• Doc Analyst: Reviews associated documentation.

• Vision Inspector: Processes visual evidence or images.

• Aggregation: All detective outputs feed into the Clerk Aggregator, which applies min-max logic to evaluate whether evidence is complete or needs further review.

• Judicial Review: When evidence meets thresholds, the case moves to the Judicial Courtroom for parallel evaluation by the Prosecutor, Defense, and Tech Lead, each forming independent judgments.

• Chief Justice Synthesis: The Chief Justice deterministically synthesizes all inputs, resolves conflicts, and decides the final audit outcome.

• Report Generation: The workflow ends with the Report Generator, producing a structured audit report summarizing findings, scores, and recommendations.

Experiments

::youtube[Title]{#https://www.youtube.com/watch?v=3MjdDmIcL3M&t=1s}

Ouput of the experment:

• I put the auditor to work on the current forensic-swarm-auditor.git repository. The system analyzed 51 historical commits and 14 core source files.
• Global Verdict: Emerald Tier: The system rendered an Aggregated Score of 4.12 / 5.00. This reflects a highly stable system that successfully passed the "bulk-upload" threshold.
• Findings & Judicial Conflict Engineering Chronology (5.0): The REPO agent verified an elite, atomic progression of work across 51 commits.
• Security Hygiene (1.2): Critical Breach Detected. The Prosecutor identified a path-injection risk due to raw string concatenation.
• Judicial Dialectic: Deep friction was recorded. The Prosecutor argued the security gap was a total failure. The Defense successfully argued that the iteration proved high sovereign state-tracking intent. The Chief Justice used deterministic rules to cap the score at 4.12.

Dialectic Friction (Trial Transcript)

The beauty of the courtroom is that it turns disagreement into data. For the Engineering Chronology and Swarm Resilience criteria, I saw a fascinating clash:

• The Prosecutor’s Charge: "The commit trace is undeniable, but the security gap is a fail." The Prosecutor looked at my path management and saw red. It identified raw string concatenations in src/core/graph.py as an "Orchestration Fraud" liability, docking the score by 2.0 points.
• The Defense’s Rebuttal: "Iteration proves intent. High state-tracking sovereignty." The Defense fought back by pointing to the 51 verified commits. It argued that a developer who iterates that much isn't being lazy; they are building a "Master-class" chronology, which should outweigh a localized path error.
• The Tech Lead’s Ruling: "AST sharding implementation is architecturally sound." The Tech Lead acted as the realistic anchor. It acknowledged the Prosecutor's security fear but confirmed that the Pydantic schema and AST sharding were world-class, providing the final weight to keep the score at an Emerald level.( for more check out the sample report output of the auidtor)

From Conflict to Remediation

This friction is what makes the Remediation Plan so clear. Because the Prosecutor focused so heavily on the "fail" of my path strings, the system produced a specific instruction: “Replace all raw string path concatenations with pathlib.Path objects.” This isn't just a generic suggestion; it’s a fix born from a trial.

Conclusion

This project proves that reproducibility is the price of scale. We have transformed a manual bottleneck into an automated service. By forcing the AI to argue against itself, we eliminated the "agreement trap" and surfaced a critical security flaw I had missed. The Emerald Suite now "thinks about thinking" to ensure code is structurally verified and secure.

Market leadership in the AI era will be defined by the integrity of the systems that govern code, not the volume of code produced.

Future Work: The 10x Scale-Up

I asked myself:
What happens if this system has to work 10x faster and 10x larger by tomorrow?

Three parts of the foundation will crack first. Our future work focuses on these limits.

• From File Dumps to Intelligent Sharding
  At 10x scale, sending a full repository dump to an agent causes Context Saturation. Precision drops. I will implement Smarter Sharding. This means using AST pre-scanners to send only "relevant code clusters" to specific agents, keeping context windows lean and accuracy high.

• From Local Tempfiles to MicroVM Orchestration
  Cloning 10x more repos concurrently on a local machine creates resource contention and security risks. Our current tempfile strategy is fast but has no resource limits. We will migrate to MicroVM isolation (e.g., Firecracker or gVisor). This provides a "Dedicated Clean Room" for every audit with hard caps on CPU and RAM.

• From Multimodal Vision to 2M-Token Gating
  Complex system diagrams for large architectures can exceed 1M-token windows when combined with full repository maps. I plan to upgrade the VisionInspector to Gemini 2.0 Pro. This will enable a Deep-Gating mechanism, allowing the auditor to analyze 1,000-page technical manuals alongside full AST dumps in a single inference pass.(Different and optimized use of llm models at diffrent nodes)

• Automated Constitution Updates
  Currently, the human lead is the bottleneck for updating the audit rubric. We will explore Recursive Governance. Agents will monitor real-world attack patterns and peer-audit results to propose new "Statutes" for the courtroom, ensuring the auditor evolves as fast as the code it governs.