AI/ML projects are increasingly built around rapidly evolving frameworks, architectures, retrieval systems, and evaluation practices. However, understanding trends across multiple projects usually requires manually inspecting repositories, README files, dependency manifests, and implementation details.
The Cross-Publication Insight Assistant addresses this problem by providing a multi-agent system that analyzes multiple GitHub repositories and extracts structured project information, enabling users to ask cross-project questions such as:
The system combines LangGraph-based multi-agent orchestration, repository ingestion, semantic retrieval, deterministic aggregation, fact-checking, and natural-language summarization. A FastAPI backend exposes the system through REST endpoints, while a Streamlit interface provides an interactive user experience.
The primary goal is not simply to summarize individual projects, but to enable grounded reasoning across multiple projects.
AI/ML repositories contain a large amount of information about technologies, architectures, tools, models, databases, and implementation patterns. However, this information is distributed across README files, source code, dependency files, configuration files, and documentation.
A user interested in cross-project questions must typically inspect each repository individually and manually compare the findings.
For example, answering:
"What percentage of these projects use LangGraph?"
requires identifying all relevant projects, determining whether each project actually uses LangGraph, counting the matches, calculating the percentage, and presenting the result with supporting evidence.
A conventional LLM chatbot can generate an answer, but it may hallucinate project characteristics or make arithmetic mistakes.
The Cross-Publication Insight Assistant automates this process through a multi-agent architecture.
The system accepts a collection of public GitHub repositories and a natural-language query. It analyzes the projects, stores structured metadata and searchable evidence, determines the appropriate query strategy, executes the required reasoning workflow, verifies the resulting claims, and generates a grounded response.
The system supports three major query patterns:
The Cross-Publication Insight Assistant operates as a pipeline beginning with a collection of project URLs and an optional natural-language query. The projects are first ingested and analyzed so that their technical characteristics can be represented in both structured and semantic forms.
At the center of the system is a LangGraph workflow that coordinates the specialized agents. The workflow currently contains four principal AI agents: the Project Analyzer, Query Router, Fact-Checker, and Summarizer. These agents do not perform the same task. Each has a specific responsibility within the overall reasoning process.
The Project Analyzer examines an individual project and extracts useful technical attributes from its repository structure and contents. These attributes can include frameworks, LLM providers, storage technologies, architectures, tools, and other implementation characteristics. Importantly, the analyzer also associates extracted information with evidence and confidence information rather than producing an unconstrained textual description.
Once projects have been indexed, the Query Router interprets the user's question and determines which reasoning path should be executed. The current architecture supports aggregate analysis, semantic retrieval, and comparative analysis, while unsupported queries can be rejected rather than forcing the system to generate an unreliable answer.
This routing mechanism is important because different questions require fundamentally different forms of reasoning. A question asking for a percentage should not be answered through semantic retrieval alone, while a question asking “Which projects use vector databases?” does not require the system to calculate a percentage.
After the selected reasoning path produces its findings, the Fact-Checker evaluates the claims against the underlying project evidence. Finally, the Summarizer converts the verified findings into a concise natural-language response while preserving the supporting evidence.
In simplified form, the workflow can be represented as:
Input → Validation → Query Routing → Analysis/Retrieval → Fact Checking → Summarization → Final Answer
This separation makes the system easier to reason about and reduces the likelihood that an unsupported LLM-generated statement reaches the user.
The first major challenge is transforming heterogeneous repositories into information that can be analyzed consistently.
When a GitHub repository is submitted, the ingestion layer retrieves relevant repository information including its file tree, README documentation, dependency manifests, configuration files, and selected source-code files. The parser then transforms this raw repository content into analyzable project information.
The Project Analyzer Agent operates on this material and produces structured metadata. Rather than asking the model to simply summarize the repository, the system asks it to identify specific technical characteristics. For example, a project may be represented as using LangGraph for orchestration, PostgreSQL for persistence, ChromaDB for vector storage, and a particular LLM provider.
This structured representation is important for cross-project analysis. If ten projects are analyzed independently and each produces the same type of metadata, the system can subsequently calculate trends across those projects without repeatedly asking an LLM to interpret the repositories.
At the same time, the original repository content is preserved for semantic retrieval. This creates two complementary representations of each project: a structured representation for analytical operations and an evidence-oriented representation for retrieval and verification.
LangGraph serves as the orchestration layer for the system. Its role is not simply to call several agents sequentially, but to maintain state and direct execution according to the type of user query.
The first specialized component is the Project Analyzer Agent. Its responsibility is project-level understanding. Given repository content, it identifies technologies, architectural patterns, models, storage systems, and other relevant characteristics.
The Query Router Agent operates at the question level. It classifies a user's query into an appropriate execution branch such as aggregate, retrieve, compare, or unsupported. It also extracts relevant concepts that downstream components can use.
The Fact-Checker Agent acts as a verification layer. Its purpose is particularly important in an LLM-based system because a fluent answer is not necessarily a correct answer. The fact checker compares generated claims against retrieved evidence and mathematical results and assigns verification statuses such as verified, partial, or unverified.
Finally, the Summarizer Agent is responsible for communicating the verified results. It does not independently determine the facts. Instead, it receives findings and evidence produced by earlier stages and turns them into a readable response.
This division of responsibilities creates a useful distinction between reasoning, computation, verification, and communication.
The system supports several query patterns, with each pattern using a different execution strategy.
For aggregate questions such as “What percentage of these projects use LangGraph?”, the system first analyzes the relevant project metadata. It then counts the projects satisfying the requested condition and calculates the percentage using a deterministic calculation tool. The LLM is therefore not responsible for arithmetic.
For example, if eight out of ten analyzed projects use LangGraph, the calculation engine produces:
8 / 10 × 100 = 80%
The resulting value is passed to the verification stage together with the underlying project evidence. This approach prevents a language model from incorrectly calculating or modifying the numerical result.
The retrieval path is designed for questions such as “Show me projects that use vector databases.” Instead of scanning every repository manually, the system performs semantic retrieval over indexed project content using embeddings and ChromaDB. Relevant evidence chunks are retrieved and associated with their corresponding projects. The system can then identify the matching projects and explain why each one satisfies the query.
The comparison path addresses questions where the user wants to understand differences between groups of projects. For example, a user could compare projects using LangChain with projects using CrewAI. The system groups projects according to the requested criteria and compares their available attributes, such as orchestration framework, storage technology, architecture, model usage, or evaluation methodology.
These three paths demonstrate why the architecture uses query routing rather than treating every question as a standard RAG query.
Semantic retrieval is implemented using ChromaDB and embeddings. Project content is divided into smaller chunks before being indexed. This allows the system to retrieve specific sections of repository documentation or code that are relevant to a user's question.
A major design consideration is maintaining the relationship between retrieved evidence and its originating project. Each indexed chunk is associated with project-level metadata, allowing retrieval to be filtered and interpreted in the context of the relevant repository.
Retrieval is therefore not used simply to generate an answer. It serves as an evidence-gathering mechanism for subsequent reasoning and verification.
This distinction is particularly important for the Fact-Checker Agent. When the system claims that a project uses a particular framework, for example, the claim should be traceable to repository content rather than being based solely on the language model's prior knowledge.
The system uses two complementary storage mechanisms because structured project metadata and semantic project content have different storage requirements.
PostgreSQL is used as the persistent structured database. It stores project-level information and metadata that can be queried deterministically. This makes it appropriate for operations such as identifying analyzed projects, storing structured attributes, and maintaining relationships between projects and extracted information.
ChromaDB serves a different purpose. It stores vector representations of project content and enables semantic similarity search. This makes it appropriate for retrieving relevant textual or code evidence based on the meaning of a user's query rather than exact keyword matches.
Using the two systems together avoids forcing one database to perform two fundamentally different jobs. PostgreSQL provides reliable structured persistence, while ChromaDB provides semantic retrieval.
One of the central problems addressed by the architecture is hallucination.
A conventional LLM-based assistant might answer a question about several repositories entirely from generated reasoning. Even when the answer sounds convincing, it may contain incorrect framework usage, invented project characteristics, or incorrect numerical calculations.
The Cross-Publication Insight Assistant therefore introduces verification as an explicit stage of the workflow.
The Fact-Checker receives claims generated by the analysis stage together with relevant evidence. It evaluates whether the claims are supported by the underlying project information. Claims can be classified as verified, partially supported, or unverified.
The architecture also separates deterministic calculations from language-model reasoning. Percentages, counts, and ratios are calculated by dedicated software rather than generated by the LLM. This creates a stronger boundary between probabilistic interpretation and deterministic computation.
The final summarization stage is consequently based on verified information rather than raw model output.
The system is exposed through a FastAPI backend that provides REST endpoints for project ingestion, querying, health monitoring, and retrieving indexed projects.
A Streamlit frontend provides the user-facing interface. Users can submit repository URLs, inspect indexed projects, enter natural-language questions, and view the resulting insights together with fact-checking information.
This architecture separates the presentation layer from the agentic backend. The Streamlit interface is therefore not responsible for the reasoning itself; it communicates with the FastAPI service, which executes the underlying workflow.
The application can also be deployed using Docker Compose, allowing the backend and frontend services to be run in isolated containers with a consistent environment.
The system extends the capabilities of the language models through several purpose-built tools.
The GitHub ingestion tools allow the system to retrieve repository structures and source material. The repository search and ChromaDB integration provide semantic retrieval capabilities. The aggregation calculation tools perform deterministic mathematical operations required for cross-project statistics.
These tools are important because the project is designed around the principle that an agent should use external capabilities when those capabilities provide greater reliability than unconstrained language generation.
The result is a hybrid system in which the LLM handles interpretation and language generation, while specialized tools handle repository access, retrieval, storage, and computation.
Evaluation is an important part of the system because a multi-agent workflow can fail at several different stages. A correct final answer requires not only good language generation, but also correct routing, retrieval, analysis, computation, and verification.
The project therefore includes a dedicated evaluation framework. Query routing is evaluated using intent classification accuracy across supported query categories. The Fact-Checker is evaluated against ground-truth claims to determine whether supported and unsupported statements are correctly identified. The aggregation engine is evaluated separately because its numerical operations should be deterministic.
The repository also includes automated tests covering the implemented components and an evaluation pipeline based on Ragas-related evaluation infrastructure.
Rather than evaluating the assistant only on whether its final response sounds correct, this framework evaluates individual system capabilities. This makes it possible to identify where errors originate and provides a foundation for future benchmarking.
The application is designed to be reproducible through environment-based configuration and containerization. Dependencies are specified in the project configuration, while sensitive credentials such as API keys are supplied through environment variables rather than being embedded in source code.
Docker and Docker Compose provide a standardized deployment environment for the backend and frontend services. The FastAPI service exposes interactive API documentation, while the Streamlit application provides a graphical interface for experimentation.
The project can therefore be used both as a development prototype and as a foundation for further experimentation with cross-project AI/ML analysis.
The current system primarily focuses on publicly accessible GitHub repositories and the project information that can be reliably extracted from them. Repository quality varies significantly, and a project may not explicitly document every technology or architectural decision. Consequently, absence of evidence should not automatically be interpreted as evidence that a technology is not used.
Another limitation is that semantic retrieval quality depends on the quality of chunking and embeddings. Very large or poorly structured repositories may require more sophisticated indexing strategies.
Future development could extend the ingestion layer to support a broader range of Ready Tensor publications, improve repository-level code analysis, introduce richer temporal trend analysis, and provide more sophisticated comparative visualizations. Additional evaluation could also benchmark the system against a conventional single-agent RAG baseline to quantify the benefits of the multi-agent architecture.
The Cross-Publication Insight Assistant demonstrates an approach to cross-project AI/ML analysis based on specialized agents, external tools, structured storage, semantic retrieval, deterministic computation, and explicit verification.
The key contribution of the system is not simply the use of multiple LLM agents. Rather, it is the separation of responsibilities across the complete reasoning pipeline. Project analysis transforms heterogeneous repositories into structured knowledge; semantic retrieval provides evidence; deterministic tools perform numerical operations; specialized agents route and interpret queries; fact checking verifies claims; and summarization communicates the final result.
By combining these components through LangGraph, the system provides a practical framework for answering questions that span multiple AI/ML projects while maintaining a stronger connection between generated insights and their underlying evidence.
The project therefore explores a broader principle for agentic AI systems: reliable intelligence is not achieved by asking one model to do everything, but by combining language models with specialized agents, deterministic tools, structured data, retrieval, and verification.
Reproducibility
The complete implementation, including the application source code, evaluation framework, Docker configuration, prompts, tests, and deployment configuration, is publicly available in the project repository: