Program: Agentic AI in Production Certification — Multi-Agent System Capstone
Category: AI Systems → Multi-Agent Systems (Developer Tooling)
Author: Joram Kirubi
Code: https://github.com/joramkirubi/Publication-assistant

Abstract

Problem. Preparing an AI/ML project for public sharing requires judgment calls a project's own author is poorly positioned to make: is the title accurate, does the summary undersell the work, which standard documentation sections are missing, and what tags would actually help someone find it. Manually checking a repository against a documentation standard is repetitive and easy to skip under deadline pressure.

What was built. Publication Assistant is a four-agent system, orchestrated with LangGraph, that takes a public GitHub repository URL and produces a markdown report: a suggested title and summary, 5-8 recommended tags, and a list of documentation sections present or missing relative to Ready Tensor's Open Source Repository Guide (Essential and Professional tiers). A human review checkpoint sits between suggestion generation and final report synthesis, so the system proposes rather than dictates. It ships with two interchangeable front ends (a CLI and a Streamlit web UI) built on identical backend logic, a guardrails layer for input validation and output redaction, a resilience layer for retry/timeout handling on every outbound call, and a health-check utility for deployment verification.

How it works. A Repo Analyzer agent fetches the README, file structure, and repository metadata via the GitHub REST API. Two agents — Metadata Recommender and Content Improver — run in parallel, since neither depends on the other's output: one extracts and refines tags, the other drafts a title/summary grounded in the README and optional web search context. The graph then pauses at a human-in-the-loop checkpoint before a Reviewer/Critic agent checks documentation structure and synthesizes the final report, incorporating any edits or feedback the human provided.

Key results. The system was validated with 78 automated tests (97% statement coverage on the src/ package), all running offline against mocked external services. A manual end-to-end run against an independent public repository produced a correctly structured report identifying missing documentation sections and a materially improved title/summary. Two concurrency-related defects were found and fixed during development: an ambiguous state-update error when applying human edits to a parallel-write graph, and a CLI input-validation gap that silently treated any keystroke as approval.

Keywords: multi-agent systems, LangGraph, human-in-the-loop, agent orchestration, documentation quality assessment, Groq, guardrails, resilience engineering, Streamlit

System Overview and Pipeline

Purpose

Publication Assistant answers a narrow, well-defined question: given a public GitHub repository, what should change before this project is published? It does not generate documentation from nothing — every claim it produces is intended to be traceable back to the repository's own README, file structure, or metadata. The Reviewer/Critic agent is explicitly instructed to ground its output in retrieved content rather than invent details.

Pipeline

Component overview

Why this shape

Four agents instead of one large prompt. Repo analysis, metadata suggestion, content drafting, and review are different tasks with different failure modes. Fetching a README can fail on a 404; drafting a summary can fail on an empty README; structure checking can fail on an unusual heading style. Splitting these into separate agents means one failure is isolated and recorded (in the errors field of shared state) rather than corrupting the entire run.

Parallel fan-out for Metadata Recommender and Content Improver. Both agents depend only on Repo Analyzer's output, not on each other's. Running them as parallel branches of the graph reduces wall-clock latency without changing the result, since they write to disjoint fields of shared state.

Repo Analyzer has no LLM call. It is a thin wrapper over three deterministic HTTP calls. This is a deliberate choice: making the first step of the pipeline non-probabilistic means a failure here is unambiguous (a 404, a rate limit, a network error) rather than an LLM misinterpreting a tool result.

Interface and User Experience

Two front ends, one backend

Publication Assistant exposes the same pipeline through a command-line interface and a Streamlit web application. Both call the exact same two functions in src/graph.py — start_pipeline() and resume_pipeline() — so there is no duplicated business logic between them, and a guardrail or bug fix applies to both automatically.

CLI (main.py)

python main.py --repo [--description TEXT] [--output PATH]
[--no-save] [--auto-approve] [--health-check]

The interactive review prompt (Approve as-is? [Y/n/e=edit]:) is a real blocking checkpoint: the process waits for a Y, n, or e response, validating input in a loop rather than accepting the first keystroke — this specific validation was a defect found and fixed during development.

Web UI (app.py, Streamlit)

The UI implements the identical three-stage flow as a session-state machine (stage: "input" -> "review" -> "done"):

1. Input — a form collecting the repository URL and optional description. Invalid URLs are rejected by the same guardrail function the CLI uses, with the error shown inline rather than as a stack trace.
2. Review — the paused pipeline state is rendered as editable fields (title, summary, tags, feedback). Approving or rejecting resumes the graph with the edited values.
3. Report — the rendered markdown report with a "Download report (.md)" button and a "Start over" action.

The sidebar exposes a "Run health check" button (calls the same src/health.py logic as --health-check) and a "Start over" button that resets all session state.

Why a Streamlit UI specifically

Gradio and Streamlit were both viable; Streamlit was chosen because its session-state model maps directly onto the pipeline's own pause/resume semantics — the paused LangGraph state and the Streamlit session state can be held as the same object across a rerun, with no separate state-synchronization layer needed. This kept the UI a genuinely thin wrapper rather than requiring its own state machine independent from the graph's.

Memory and Reasoning

What "memory" means in this system

Publication Assistant has no vector database, no long-term memory across runs, and no conversation history in the chatbot sense. Its memory is the LangGraph shared state object plus a checkpointer that persists a single run's state across a human-in-the-loop pause. This section covers exactly how that state is structured, how each agent reasons over it, how concurrent writes are reconciled, how the pause/resume mechanism works at the implementation level, and two real defects this design produced during development.

The shared state object

PublicationAssistantState is a TypedDict(total=False) defined in src/state.py. Every agent reads from and writes to this single object; there is no message-passing between agents directly.

class PublicationAssistantState(TypedDict, total=False):
    repo_url: str
    user_description: str
    owner: str
    repo: str
    readme_text: str
    file_structure: list[str]
    repo_metadata: dict
    suggested_keywords: list[str]
    suggested_tags: list[str]
    suggested_title: str
    suggested_summary: str
    positioning_notes: str
    structure_report: dict
    review_notes: list[str]
    final_report: str
    human_approved: bool
    human_feedback: str
    errors: Annotated[list[str], operator.add]

Two design choices here carry real weight:

total=False. Every field is optional in the type system because at any given point in the graph's execution, most fields are genuinely unset — suggested_title does not exist until Content Improver runs, final_report does not exist until Reviewer/Critic completes. Making the schema total=True would require populating every field with placeholder values at every step, which would make partial/failed states harder to distinguish from complete ones.

readme_text truncation. Repo Analyzer truncates the fetched README to settings.max_readme_chars (12,000 characters) before writing it to state. This bounds the token cost of every downstream LLM call that reads readme_text, and it protects against a pathological README (a generated file, a vendored dependency's README committed by mistake) from silently blowing up the context window of three separate LLM calls.

The reducer problem: concurrent writes to errors

Metadata Recommender and Content Improver run as parallel branches. Both are wrapped in try/except blocks that append to state["errors"] on failure. LangGraph's default behavior for a state key written by two nodes in the same superstep is to raise an error unless a reducer is declared — because without one, the framework cannot know whether the second write should overwrite the first or merge with it.

The errors field is annotated Annotated[list[str], operator.add], which tells LangGraph to concatenate lists returned by concurrent writers rather than treating a second write as a conflict. This was not a hypothetical concern — it was discovered as a real InvalidUpdateError during development when both parallel agents failed in the same run (a repo with a very short README caused both agents to log a low-confidence warning simultaneously), and is now covered by a dedicated regression test that intentionally fails both parallel agents in the same graph invocation and asserts both error messages are present in the final errors list, in the order the nodes completed.

No other field needed a reducer, because no other field is written by more than one node.

Reasoning per agent

Repo Analyzer performs no LLM reasoning. It calls github_repo_reader.invoke({"repo_url": ...}), which internally makes three sequential GitHub REST API calls (README content, top-level tree, repository metadata) and writes their results directly into state. This is intentional: the first step of the pipeline should fail in predictable, mechanical ways (404, rate limit, network error) rather than in the ambiguous ways an LLM call can fail (a plausible-sounding but wrong interpretation of a tool result).

Metadata Recommender reasons in two stages. First, keyword_extractor (a non-LLM tool using term-frequency-style extraction over readme_text and repo_metadata.topics) produces a ranked candidate list, written to suggested_keywords. Second, an LLM call is prompted with those candidates plus the README and asked to select and refine 5-8 final tags, written to suggested_tags. Splitting extraction from refinement means the LLM's job is narrow (choose and phrase, not invent from nothing), which reduces the chance of a hallucinated tag unrelated to the actual repository content.

Content Improver reads readme_text, repo_metadata, and user_description, optionally calls web_search (Tavily) for positioning context about similar projects, then makes a single LLM call to draft suggested_title, suggested_summary, and positioning_notes. The prompt explicitly instructs the model to ground the summary in what the README actually describes rather than in what a typical project of that type might contain — this instruction exists because an earlier prompt draft, without it, occasionally invented features implied by the tech stack but not present in the actual README.

Reviewer / Critic is the only agent that reads the full accumulated state rather than a subset of it. It runs readme_structure_checker (a deterministic tool comparing README headings against Ready Tensor's Essential/Professional section lists) to produce structure_report, then makes one LLM call that receives: the structure report, the suggested title/summary/tags (as edited by the human, if edited), any human_feedback, and the original README. This call synthesizes final_report. Because it runs after the human checkpoint, its prompt includes an explicit instruction to treat human_feedback as a directive to incorporate, not as content to merely acknowledge — verified by a dedicated test that inspects the literal message payload sent to the LLM after a human edit, confirming the edited text (not the original agent-generated text) is what the model actually receives.

The human-in-the-loop checkpoint: mechanics

The graph is compiled with interrupt_before=["reviewer_critic"] and an InMemorySaver checkpointer:

checkpointer = InMemorySaver()
app = graph.compile(checkpointer=checkpointer, interrupt_before=["reviewer_critic"])

start_pipeline() runs the graph with a fresh thread_id (a uuid4) stored in config["configurable"]["thread_id"]. Execution proceeds through Repo Analyzer, then the parallel Metadata Recommender / Content Improver branch, then genuinely stops — not by returning early, but because LangGraph's checkpointer persists the full state at that point, keyed by thread_id, and control returns to the caller without running Reviewer/Critic.

resume_pipeline(app, config, edits=...) does two things in sequence. First, if edits is provided, it calls app.update_state(config, edits, as_node="content_improver") — writing the human's edited title/summary/tags/feedback into the same state object under that thread_id. Second, it calls app.invoke(None, config), which resumes execution from exactly where it paused, using the now-edited state as input to Reviewer/Critic.

A real defect: InvalidUpdateError: Ambiguous update, specify as_node

The first implementation of resume_pipeline() called app.update_state(config, edits) without the as_node argument. This worked in every test that only exercised one branch, but failed on any real run, because Metadata Recommender and Content Improver both wrote to state in the same graph superstep — LangGraph could not determine which of the two simultaneous writers the human's edit should be attributed to, and raised InvalidUpdateError. The fix was to pass as_node="content_improver" explicitly, since the fields being edited (suggested_title, suggested_summary, suggested_tags) are semantically closer to Content Improver's and Metadata Recommender's outputs than to any other node's. This is documented as a design decision rather than an arbitrary fix: as_node in LangGraph does not have to match which node "actually" produced a field in a deep sense, only which node's identity the update should be recorded under for the checkpointer's bookkeeping — but choosing the semantically closest node keeps the state history interpretable if inspected later.

A related defect: silent approval on invalid CLI input

Independent of the state model itself, the original CLI review prompt read one line of input and treated anything that wasn't exactly "n" or "e" as approval — including a typo, an empty line from an accidental Enter press, or unrelated pasted text. This meant a user could unintentionally approve suggestions they intended to reject or edit. The fix replaced the single read with a validation loop that re-prompts until the input matches y, n, e (case-insensitive) or is empty (which defaults to yes, matching the [Y/n/e] prompt convention), with every other input rejected and re-prompted rather than silently accepted. A dedicated test exercises this with a sequence of invalid inputs followed by a valid one, asserting the prompt loop does not proceed until valid input arrives.

Resilience as a memory-adjacent concern

Every LLM call that participates in the reasoning described above goes through invoke_llm() in src/llm.py, which wraps llm.invoke() with an application-level retry (2 attempts, 1.5s base delay, exponential backoff) and a hard 45-second timeout (thread-based, not signal-based, because this project's primary target environment is Windows, which has no SIGALRM). This matters for the state model specifically: a hung or transiently-failing LLM call inside a parallel branch would otherwise block the fan-in at the human checkpoint indefinitely. The retry/timeout wrapper converts "the whole pipeline hangs" into "this agent's errors entry says why it degraded," which is the same graceful-degradation pattern that governs every other failure mode in the system.

Deployment Guide

Prerequisites

Local install

git clone https://github.com/joramkirubi/Publication-assistant.git
cd Publication-assistant
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env

Technical Implementation

Stack

Module structure

Edit .env and set GROQ_API_KEY (and optionally TAVILY_API_KEY, GITHUB_TOKEN).

Verify before running

Checks config presence and GitHub connectivity in under a second, without spending any LLM tokens. Run this after any environment change.

Run it

python main.py --repo https://github.com/owner/repo
streamlit run app.py

Containerized deployment

A Dockerfile, docker-compose.yml, and .dockerignore ship with the repository. Secrets are passed at container run time, never baked into the image.

Containerized deployment

A Dockerfile, docker-compose.yml, and .dockerignore ship with the repository. Secrets are passed at container run time, never baked into the image.

docker compose up
or equivalently:
docker build -t publication-assistant .
docker run -e GROQ_API_KEY=... -p 8501

publication-assistant

The image's HEALTHCHECK calls python main.py --health-check internally, so a container that's "up" but misconfigured (missing API key) is distinguishable from one that's genuinely healthy.

Continuous integration

A GitHub Actions workflow (.github/workflows/tests.yml) runs the full test suite with a 70% coverage floor on every push and pull request to main, across three Python versions. All 78 tests mock external services, so CI runs with no real API keys configured.

Logging and maintenance

Both entry points configure Python's standard logging at INFO level. Every agent logs a full traceback on failure before degrading gracefully. Reports are timestamped and never overwritten (reports/<owner>_<repo>_<timestamp>.md), so old runs are retained for audit — prune periodically if disk usage matters for your deployment.

API and Interface Specification

This system has no REST API. Its interface is the shared state schema plus two front ends built on identical backend calls.

Programmatic interface (src/graph.py)

start_pipeline raises GuardrailViolation (a ValueError subclass) if repo_url isn't a valid https://github.com// URL.

State schema (src/state.py)

CLI interface

python main.py --repo [--description TEXT] [--output PATH] [--no-save] [--auto-approve] [--health-check]

Exit codes: 0 success, 1 invalid input or unexpected pipeline failure.

Web UI interface

streamlit run app.py exposes the same three stages as screens: input form, human review form, final report with download. The UI holds no pipeline logic of its own — every field and button maps directly to a src/graph.py call or a PublicationAssistantState field listed above.

Troubleshooting Guide

First step for any issue

python main.py --health-check

Distinguishes a configuration problem from a real failure in under a second, without spending LLM tokens.

Common issues

Resilience behavior to know

Every outbound call (GitHub API, Groq, Tavily) retries transient failures automatically with exponential backoff before giving up, and every LLM call has a hard 45-second timeout. If something fails after retries, the report's warnings section explains why rather than the process crashing silently.

FAQ

Does this work with private repos? Only with a GITHUB_TOKEN scoped to that repo; the tool is designed and tested against public repos.

What happens without TAVILY_API_KEY? Content Improver still runs, drafting from the README alone rather than failing — a deliberate graceful-degradation path, not a bug.

Why does the final report sometimes redact text that looks like an API key? src/guardrails.py runs a last-resort regex redaction pass on every report in case a README or search result echoed something resembling a credential. It is a safety net, not a sign of an actual leak.

Why Groq

Groq was selected for LLM inference speed (relevant given three of the four agents make sequential-dependent LLM calls within a single run) and free-tier availability during development. This creates a single point of provider coupling by design: src/llm.py is the only module that imports ChatGroq, so switching providers means changing one file, not four agent implementations.

def get_llm(temperature: float | None = None) -> ChatGroq:
    require_groq_key()
    return ChatGroq(
        model=settings.model_name,
        temperature=temperature if temperature is not None else settings.model_temperature,
        api_key=settings.groq_api_key,
        timeout=60,
        max_retries=2,
    )

The resilience layer

src/resilience.py provides two decorators used across the codebase rather than a heavier retry framework, since only two behaviors were needed:

def with_retry(max_attempts=3, base_delay=1.0, backoff_factor=2.0, exceptions=(Exception,)):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(1, max_attempts + 1):
                try:
                    return func(*args, **kwargs)
                except exceptions as exc:
                    if attempt == max_attempts:
                        raise
                    time.sleep(base_delay * (backoff_factor ** (attempt - 1)))
        return wrapper
    return decorator

with_timeout(seconds) runs the wrapped function in a ThreadPoolExecutor(max_workers=1) and raises ResilienceTimeoutError if future.result(timeout=seconds) expires. This is thread-based rather than signal.alarm-based specifically because the project's target environment is Windows/PowerShell, which does not support SIGALRM. The underlying call is not forcibly killed on timeout — Python has no safe mechanism for that — it continues running in the background thread while control returns to the caller with an error.

These decorators are applied at two call sites: the three GitHub API fetch functions in github_repo_tool.py (3 attempts, 1.0s base delay, retried only on ConnectionError, Timeout, and HTTPError — not on 404s, which are handled explicitly and not retried, since retrying a request that is wrong will not make it right), and invoke_llm() in llm.py (2 attempts, 1.5s base delay, 45s hard timeout).

Guardrails implementation

Input validation accepts only https://github.com//-shaped URLs, rejecting other hosts (closing off SSRF-style redirection to internal hosts), non-https schemes, extra path segments, control characters, and oversized input. The optional free-text description is sanitized (control characters stripped, whitespace collapsed, capped at 500 characters) before it reaches any LLM prompt. Output filtering runs a regex-based redaction pass over the final report for anything resembling a leaked API key (Groq, GitHub, Tavily, AWS, generic sk- style keys) as a last-resort safety net against a README or web-search snippet echoing a real credential back through the LLM.

Health checks

src/health.py performs five fast, side-effect-free checks: GROQ_API_KEY presence, GitHub API connectivity (one lightweight GET request), TAVILY_API_KEY presence (optional), GITHUB_TOKEN presence (optional), and reports/ directory writability. Exposed via python main.py --health-check and a sidebar button in the Streamlit UI. No LLM tokens are spent running it.

Testing and Evaluation

Test suite composition

Total: 78 tests, all running offline against mocked GitHub, Groq, and Tavily calls — no real API keys are required to run the suite.

Coverage

97% statement coverage on src/, against a project requirement of 70%. The uncovered lines are concentrated in narrow exception branches (unreachable-in-practice fallback paths) rather than untested feature logic.

Reproducing these results

pytest tests/ -v
pytest tests/ --cov=src --cov-report=term-missing

Manual end-to-end validation

Beyond the mocked test suite, the full pipeline was run against an independent public repository not used during development (joramkirubi/medical-rag-assistant) with a real Groq API key. The run correctly identified five missing documentation sections (Overview/Description, Usage, Configuration, Testing, Contributing) against the two present (Title, Installation, License), and produced a title/summary that named the actual retrieval architecture (RAG with ReAct reasoning) described in the README rather than a generic restatement of the repository name.

Two defects found only through this manual, interactive testing

Both defects described in the Memory and Reasoning section — the InvalidUpdateError from applying human edits to a parallel-write graph, and the silent-approval gap in the CLI's input handling — were invisible to the mocked test suite at the time they were introduced, because the tests exercised each code path independently rather than through a real interactive session. Both are now covered by dedicated regression tests, but their discovery is a specific argument for running a real, interactive, non-mocked pass before considering a human-in-the-loop feature complete: mocked unit tests validate that each function behaves correctly in isolation, not that two features compose correctly when exercised together in the order a real user would trigger them.

Discussion and Conclusion

What this system does well

It gives an objective, repeatable first pass on documentation quality that does not depend on the author's own blind spots about their project. The four-agent split with a mechanical first step (Repo Analyzer) and an explicit human checkpoint means failures are localized and suggestions are never applied without a human seeing them first. The guardrails and resilience layers were built as thin, dependency-light wrappers rather than adopting a heavier framework, which kept them auditable in isolation — src/guardrails.py and src/resilience.py can each be read start to finish in a few minutes.

Known limitations

Conclusion

Publication Assistant demonstrates that a documentation-review task can be decomposed into agents with genuinely different failure modes, coordinated through a single shared state object with an explicit human checkpoint, and hardened for production use without changing that core architecture. The two real defects surfaced during development — both concurrency-related, both invisible to isolated unit tests — are documented here not as footnotes but as evidence for why interactive, end-to-end testing remains necessary even when a mocked test suite reports high coverage.