AI/ML projects are increasingly built around rapidly evolving frameworks, architectures, retrieval systems, models, and evaluation practices. However, understanding trends across multiple projects usually requires manually inspecting repositories, README files, dependency manifests, configuration files, and implementation details.
The Cross-Publication Insight Assistant addresses this problem by providing a multi-agent system that analyzes public GitHub repositories and Ready Tensor publications and extracts structured project information, enabling users to ask cross-project questions such as:
The system combines LangGraph-based multi-agent orchestration, repository ingestion, semantic retrieval, deterministic aggregation, fact-checking, and natural-language summarization. A FastAPI backend exposes the system through REST endpoints, while a Streamlit interface provides an interactive user experience.
The primary goal is not simply to summarize individual projects, but to enable grounded reasoning across multiple projects while maintaining traceability between generated insights and their underlying evidence.
AI/ML repositories contain a large amount of information about technologies, architectures, tools, models, databases, and implementation patterns. However, this information is distributed across README files, source code, dependency files, configuration files, and documentation.
A user interested in cross-project questions must typically inspect each repository individually and manually compare the findings.
For example, answering:
"What percentage of these projects use LangGraph?"
requires identifying all relevant projects, determining whether each project actually uses LangGraph, counting the matches, calculating the percentage, and presenting the result with supporting evidence.
A conventional LLM chatbot can generate an answer, but it may hallucinate project characteristics or make arithmetic mistakes.
The Cross-Publication Insight Assistant automates this process through a multi-agent architecture.
The system accepts public project URLs and natural-language questions. Projects are ingested, parsed, analyzed, and indexed so that both structured metadata and semantic evidence are available for downstream analysis.
A LangGraph workflow then routes each question according to its intent. The supported execution branches are aggregate analysis, semantic retrieval, comparison, and unsupported-query handling.
The overall workflow is:
Input → Ingestion/Analysis → Query Routing → Retrieval/Computation/Comparison → Fact Checking → Summarization → Structured Answer + Evidence
This architecture separates deterministic operations and verification from language-model generation.
The Cross-Publication Insight Assistant consists of an ingestion pipeline and a stateful LangGraph query workflow.
The ingestion pipeline retrieves project information from GitHub and supported publications. Repository material is passed through a parser and code extractor before being divided into semantic chunks. The chunks are stored in ChromaDB for vector retrieval, while the Project Analyzer Agent extracts structured project metadata for database storage.
The query workflow begins when a user submits a natural-language question through the FastAPI backend.
The Query/Router Agent classifies the question into one of four branches:
Aggregate questions are processed by a deterministic mathematics node. Retrieval questions are processed through vector search. Comparison questions are handled by the comparison node. Unsupported questions are sent to an explicit unsupported-query handler.
The aggregate, retrieval, and comparison branches converge on the Fact-Checker Agent. Verified findings are then passed to the Summarizer Agent, which produces the final structured answer and supporting evidence.
The system contains four specialized agents:
The Project Analyzer uses GPT-4o through OpenRouter to analyze project structures, package manifests, and code snippets and extract metadata such as frameworks, LLMs, storage technologies, architectures, and confidence scores.
The first stage of the system transforms heterogeneous project repositories into information that can be analyzed consistently.
The ingestion layer extracts repository trees, README files, requirements files, configuration manifests, and relevant source-code files.
The extracted material is passed to the Project Analyzer Agent. Instead of generating an unrestricted textual summary, the analyzer extracts structured technical characteristics.
The resulting project representation can include:
This structured representation enables deterministic cross-project analysis.
At the same time, repository content is transformed into semantic chunks and stored in ChromaDB. This gives the system two complementary representations of a project:
The repository identifies the ingestion, parsing, project analysis, database, vector storage, and retrieval components as separate implementation phases.
LangGraph serves as the orchestration layer for the system.
The workflow maintains query state and routes questions according to their intended operation rather than treating every question as a conventional RAG query.
The Project Analyzer Agent analyzes repository structures, package manifests, and code snippets.
It extracts structured metadata describing technical characteristics of each project.
The analyzer uses GPT-4o through OpenRouter and produces information including frameworks, LLMs, storage systems, architectures, and confidence scores.
The Query/Router Agent operates at the question level.
It classifies natural-language questions into:
aggregateretrievecompareunsupportedIt also extracts targeted search terms that downstream components can use.
The Fact-Checker Agent verifies generated claims against source evidence and mathematical calculations.
Its verification statuses are:
verifiedpartialunverifiedThis provides a dedicated verification layer between analysis and final response generation.
The Summarizer Agent receives verified findings, metrics, and citations and converts them into natural-language answers.
The summarizer is therefore separated from the responsibility of determining whether a claim is factually supported.
This produces a clear separation between:
Reasoning → Computation → Verification → Communication
The system supports three primary analytical query types and an explicit unsupported-query path.
Aggregate questions require counting and mathematical operations across multiple analyzed projects.
For example:
"What percentage of projects use LangGraph?"
The system determines which projects satisfy the requested condition and passes the relevant values to the deterministic aggregation component.
The numerical operation is therefore performed independently of LLM generation.
The evaluation framework also treats deterministic aggregation as a separate component, measuring the correctness of counts, ratios, and percentages across variable datasets.
Retrieval questions use semantic search over indexed project content.
For example:
"Which projects use vector databases?"
The query is routed to the RAG vector-search node.
ChromaDB stores the semantic representations of repository content, while retrieval can use project-level filtering and semantic similarity to identify relevant evidence.
Comparison questions allow users to examine differences between groups of projects.
For example:
"How do projects using LangChain differ from projects using CrewAI?"
The comparison branch can use available structured project information to compare technical attributes across groups.
Questions that do not match the supported query types are routed to an explicit unsupported-query handler.
This prevents the system from forcing unrelated questions through an analysis path that was not designed to answer them.
Semantic retrieval is implemented using ChromaDB and OpenAI embeddings through OpenRouter.
Repository content is divided into semantic chunks before being indexed.
The retrieval system associates chunks with project information, allowing retrieved evidence to be interpreted in the context of its originating repository.
This means that retrieval is used as an evidence-gathering mechanism rather than simply as context for text generation.
The Fact-Checker Agent subsequently uses source evidence chunks when verifying generated claims.
The combination of semantic retrieval and explicit fact checking provides a stronger connection between generated answers and the underlying repository material.
The application uses structured database storage alongside vector storage.
The development configuration uses SQLite:
DATABASE_URL=sqlite:///./insight.db
The application architecture also supports PostgreSQL as a database option, as reflected in the repository's database architecture.
ChromaDB is used for semantic vector storage:
CHROMA_PERSIST_DIRECTORY=./chroma_data
The Docker deployment stores the SQLite database and ChromaDB data under the application's persistent data directory.
The two storage mechanisms serve different purposes:
Structured database: project metadata and structured information.
ChromaDB: semantic representations and retrieval.
This separation allows deterministic project analysis and semantic search to operate independently.
Because the application accepts external repository URLs and natural-language questions and relies on external LLM services, input handling, verification, and credential management are important parts of the system.
Repository inputs are processed by the ingestion pipeline before entering the analysis workflow.
Natural-language questions are routed through the Query/Router Agent before being sent to an execution branch.
The unsupported-query branch provides an explicit path for questions outside the system's supported analytical operations.
The Fact-Checker Agent verifies claims against source evidence and mathematical calculations.
Claims can receive verified, partial, or unverified statuses.
This verification layer is intended to reduce unsupported claims reaching the final answer.
API credentials are configured through environment variables rather than hard-coded into application configuration.
The repository provides an .env.example file containing configuration for OpenRouter, model selection, database storage, and ChromaDB persistence.
Sensitive API keys should remain outside source control.
Runtime failures and application events can be monitored through the application's service output and container logs.
The Docker deployment provides separate API and frontend containers, making service-level failures distinguishable during deployment and debugging.
The application depends on several components, including external repositories, LLM services, databases, vector storage, and the LangGraph workflow.
The Docker deployment includes an API health check:
GET /health
The API container is checked every 10 seconds with a five-second timeout and three retries. The container also uses an unless-stopped restart policy.
The frontend is configured to depend on the API service becoming healthy before starting:
condition: service_healthy
This prevents the frontend from being started before the backend service has passed its health check.
The query workflow also includes an explicit unsupported-query branch, providing controlled handling for questions outside the supported query categories.
The application exposes a health endpoint for service monitoring:
http://localhost:8000/health
The Docker Compose configuration uses this endpoint as the API container's health check.
The system's architecture also makes several stages independently observable, including ingestion, retrieval, aggregation, fact checking, and final response generation.
This separation helps developers identify whether an incorrect result originates from repository analysis, retrieval, deterministic computation, verification, or response generation.
The system provides a FastAPI backend and a Streamlit frontend.
The Streamlit interface allows users to submit repository URLs, browse indexed repositories, ask questions, and view generated insights and fact-checking information.
The FastAPI backend provides the REST API used to interact with the system.
The local services are:
Streamlit Web UI: http://localhost:8501
FastAPI documentation: http://localhost:8000/docs
Health check: http://localhost:8000/health
The Docker deployment exposes the same services through ports 8501 and 8000.
The system extends language-model capabilities through specialized tools and infrastructure.
GitHub ingestion retrieves repository structures and files.
The parser and code extractor transform repository material into analyzable content.
The semantic chunker prepares repository content for vector indexing.
ChromaDB provides semantic retrieval.
The deterministic aggregation component performs mathematical calculations.
The LangGraph workflow coordinates these components.
FastAPI exposes the backend capabilities through REST endpoints, while Streamlit provides the user-facing interface.
This architecture separates language-model reasoning from repository access, retrieval, computation, storage, and verification.
The Cross-Publication Insight Assistant can support several practical scenarios where information must be compared across multiple technical projects.
Researchers can analyze collections of AI/ML repositories to identify commonly used frameworks, databases, model providers, architectures, and implementation patterns.
Engineering teams can examine technology adoption across a collection of projects and determine how frequently particular frameworks, vector databases, or storage technologies appear.
Development teams can use cross-project repository analysis to identify implementation patterns, dependencies, architectural characteristics, and technology choices across multiple codebases.
Technical reviewers and research communities can analyze collections of projects using consistent criteria rather than manually inspecting every repository.
Teams can ask questions across existing projects to identify recurring implementation patterns and locate examples of particular technologies or architectures.
The system is designed to support human analysis rather than replace it. Retrieved evidence remains important because repository documentation and implementation quality can vary.
The project includes a dedicated evaluation suite under app/evaluation/.
The evaluation framework measures three major components:
Query routing evaluates whether questions are correctly classified into aggregate, retrieve, compare, and unsupported categories.
Fact-checker evaluation measures whether supported claims are classified as verified and unsupported claims are identified as unverified.
Aggregation evaluation checks the correctness of counts, ratios, and percentages across variable datasets.
The evaluation suite can be executed with:
python evaluation/run_eval.py
The repository defines benchmark targets of greater than 95% for query routing accuracy, greater than 90% for fact-checker verification accuracy, and 100% for deterministic aggregation precision. These values are benchmark targets defined by the project, not reported experimental results.
The project also provides an evaluation test:
python -m pytest tests/test_evaluation.py -v
The repository provides pytest-based testing commands covering the implemented system.
All tests can be executed with:
python -m pytest
Verbose output can be enabled with:
python -m pytest -v
The dedicated evaluation tests can be executed with:
python -m pytest tests/test_evaluation.py -v
The testing and evaluation structure is designed to validate the application's components across its implementation phases.
The application requires:
Python 3.10+ and an OpenRouter or OpenAI API key are explicitly listed as project prerequisites.
git clone https://github.com/rafia-10/Cross-Publication-Insight-Assistant-.git cd Cross-Publication-Insight-Assistant-
python -m venv .venv
On Windows:
.venv\Scripts\activate
On Linux/macOS:
source .venv/bin/activate
pip install -r requirements.txt
Create the environment file:
cp .env.example .env
Configure the required model and storage settings:
OPENROUTER_API_KEY=your_openrouter_api_key OPENROUTER_BASE_URL=https://openrouter.ai/api/v1 OPENROUTER_MODEL=openai/gpt-4o DATABASE_URL=sqlite:///./insight.db CHROMA_PERSIST_DIRECTORY=./chroma_data
The repository's environment configuration defines these settings for the application.
Start the FastAPI backend:
uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload
In a second terminal, start the Streamlit frontend:
streamlit run frontend/streamlit_app.py
The application is then available through:
http://localhost:8501
The FastAPI interactive documentation is available at:
http://localhost:8000/docs
The README documents these local startup commands and service addresses.
The project includes a Docker Compose configuration for running the FastAPI backend and Streamlit frontend as separate services.
Start the application with:
docker compose up --build
The services are exposed at:
http://localhost:8501 http://localhost:8000/docs http://localhost:8000/health
The API container uses port 8000, while the Streamlit container uses port 8501. The frontend waits for the API health check before starting.
Stop the services with:
docker compose down
Repositories can be analyzed through the API endpoint:
POST /projects/analyze
Example:
curl -X POST http://localhost:8000/projects/analyze \ -H "Content-Type: application/json" \ -d '{ "urls": ["https://github.com/langchain-ai/langgraph"], "source_type": "github" }'
The endpoint ingests the repository, extracts relevant files, runs the Project Analyzer Agent, and indexes semantic chunks into ChromaDB.
Questions can be submitted through:
POST /query
Example:
curl -X POST http://localhost:8000/query \ -H "Content-Type: application/json" \ -d '{ "user_query": "What percentage of projects use LangGraph?" }'
The query is routed according to its intent and processed through the corresponding workflow branch.
Indexed projects can be retrieved through:
GET /projects
This endpoint lists the projects currently stored in the database.
The primary API endpoints are:
| Endpoint | Method | Purpose |
|---|---|---|
/health | GET | Returns service status |
/projects/analyze | POST | Ingests and analyzes repositories |
/query | POST | Executes the LangGraph query workflow |
/projects | GET | Lists indexed projects |
FastAPI interactive documentation is available at:
http://localhost:8000/docs
The API reference and example requests are documented in the repository README.
The application is designed to run through either a local Python environment or Docker Compose.
Dependencies are installed from requirements.txt.
Application configuration is supplied through environment variables.
Structured project data is stored in SQLite in the default configuration, while ChromaDB stores semantic retrieval data.
The Docker Compose configuration persists application data through a mounted ./data directory. It runs the FastAPI backend and Streamlit frontend as separate containers and uses an API health check to coordinate service startup.
The repository organizes implementation into ten phases covering:
The current system focuses on information that can be extracted from publicly accessible project repositories and supported publications.
Repository quality varies, and projects may not explicitly document every technology or architectural decision.
Therefore, absence of a technology in retrieved evidence should not automatically be interpreted as proof that the technology is not used.
Semantic retrieval quality also depends on the quality of chunking and embeddings.
Large or poorly structured repositories may require more sophisticated indexing strategies.
LLM-based components can produce incorrect interpretations. The fact-checking and evidence-grounding stages are designed to reduce this risk but cannot eliminate it completely.
The application also depends on external services such as GitHub and model providers, which can introduce latency, availability, and rate-limit constraints.
Future development can extend the ingestion layer to support additional publication and document formats.
Potential improvements include:
A particularly useful direction is evaluating the system against a conventional single-agent RAG baseline to determine whether the multi-agent architecture provides measurable improvements for cross-project analysis tasks.
The Cross-Publication Insight Assistant demonstrates an approach to cross-project AI/ML analysis based on specialized agents, external tools, structured storage, semantic retrieval, deterministic computation, explicit verification, and system-level safeguards.
The contribution of the system is not simply the use of multiple LLM agents.
Instead, the system separates responsibilities across the reasoning pipeline.
Project analysis transforms heterogeneous repositories into structured knowledge.
Semantic retrieval provides evidence.
Deterministic aggregation performs numerical operations.
The Query/Router Agent selects the appropriate execution path.
The Fact-Checker Agent verifies generated claims against evidence and calculations.
The Summarizer Agent communicates the verified findings.
This architecture creates a workflow designed to maintain a stronger connection between generated insights and their underlying evidence.
The project demonstrates a broader design principle for agentic AI systems: reliable analysis can be strengthened by combining language models with specialized agents, deterministic tools, structured data, retrieval, and explicit verification rather than relying on a single unconstrained generation step.
The complete implementation is publicly available at:
https://github.com/rafia-10/Cross-Publication-Insight-Assistant-
The repository contains the application source code, ingestion pipeline, agent implementations, LangGraph workflow, storage components, evaluation framework, FastAPI backend, Streamlit interface, Docker configuration, and project documentation.