MAD-CTI: Multi-Agent Dark Web Cyber Threat Intelligence

Abstract

The dark web is a host to illicit activities where hacker forums, blogs, and articles provide
significant insights into Cyber Threat Intelligence (CTI) that are frequently unavailable on the surface web.
The increasing incidence of security breaches underscores the necessity for advanced CTI solutions to
defend against emerging threats. This project introduces MAD-CTI, a novel multi-agent framework based
on Large Language Models (LLM) designed to extract insights from dark web sources. It independently
scrapes, analyzes, and classifies content related to vulnerabilities, malware, and hacking, by leveraging
a multi-agent architecture to improve efficiency, scalability, and consistency. By utilizing state-of-the-art
LLM models and agents, I demonstrate how organizations can adopt this methodology to enhance the
accuracy and efficiency of CTI.

System Architecture

My proposal is to implement a multi-agent architecture that utilizes a state-of-the-art LLM in GPT-4o mini to scrape dark web forums, determine relevancy of information as it pertains to the cybersecurity domain, and classify the context of the conversations to derive actionable cyber threat intelligence. Unlike a single-agent or purely LLM-based approach, I am able to build an all-inclusive tool that combines the success stories of previous research and leverage parallelism to handle complex tasks efficiently. As depicted below, the system includes a specialized agent for each task and communicates its results to the next agent in the sequence. I use Microsoft AutoGen with OpenAI’s GPT-4o mini to enable the multi-agent collaboration. Each agent will perform its task individually and pass its findings onto the next LLM agent, communicating via "group chats" and "nested chats."

The workflow is initially fed a dataset of dark web pages. This would come from a separate web scraping agent. However, to test the workflow’s performance, I fed it with prelabeled data from the CoDA database. A User Proxy Agent is used to kickstart the agent chats by instructing them with an initial task. Many blogs/forums are often in a multitude of languages. If needed, a language translator agent will convert the contents to a standard language (in our case, English) for enhanced comprehension between agents. The translated corpus is then passed on to an agent designed to analyze the text’s content. It will thoroughly understand the text’s semantics and structure to determine the type of information it is dealing with and provide an in-depth analysis of its findings. This analysis is then passed on to the Relevancy Classifier agent that determines whether the text is “Relevant” or “Not Relevant” for gathering cyber threat intelligence. The same analysis is then passed onto a Category Classification Agent that determines the cyber threat category that best fits the context of the text (“Hack,” “Malware,” “Vulnerability,” or “N/A”). Note that these agents do not need to be exposed to the underlying text to form its classification. The analysis and background knowledge passed on from previous agents is enough to form the classification agents’ output. The final output is a list of the unique identifiers for each website and the classifications provided by the agents in the form of a CSV file.

Multi-Agent Advantage

Multi-agent systems (MAS) are systems that utilize multiple computing agents that can interact with one another with little guidance. With the advancements in the artificial intelligence field, this concept can be applied to develop AI/LLM-based agents that can prompt and communicate with one another, with little-to-no human input. We can create a network of conversational agents that operate similarly to that of human workers. Single agents operate in siloed environments. They are prompted with a task from a human and collect the necessary tools to complete the tasks. Any additional sub-tasks are developed within the same environment. For example, the agent may utilize API calls to external agents to collect information; however, unlike a multi-agent network, no further communication takes place. No memory is retained. In a multi-agent network, individual agents collect information, retain the memory of this information within a shared memory state, and pass it on to the next agent in the system. The next agent can converse with previous agents without needing to access additional resources of its own or repeat tasks. Subtasks can also be created and delegated to the agents who’s roles are defined to best handle them. This approach allows for enhanced performance and efficiency as the agents prompt each other and conduct the appropriate knowledge transfers, requiring very little human intervention. This cohesive environment in which agents generate, prioritize, and execute tasks while auditing one another is what leads to the performance of these models to scale as opposed to single-agent systems. Multi-agent LLMs are a relatively new area in AI, thus applying them to a specialized domain like dark web CTI is not yet widely explored. My approach solves many common problems faced by similar work when dealing with complex dark web forum data (e.g., comprehension of slang, obfuscation, and multi-language context) by assigning specialized tasks to different agents (e.g., language translation, entity recognition, or threat classification). Existing solutions also typically involve integrating multiple tools for scraping, NLP, and ML separately. A multi-agent LLM-based system could streamline these tasks, providing a unified approach to automatically detect emerging threats, track trends, and/or identify significant conversations with minimal human intervention. Finally, multi-agent architectures can be designed to adapt dynamically based on the nature of the conversation or detected threats, potentially outperforming static models.

Results and Code

The multi-agent system outperformed a single agent approach by nearly 14%. You may view the code and complex prompts that went into each agent below:

Text Analyzer Agent

def TextAnalyzerAgent(llm_config, dataset):
    translator = AssistantAgent(
        name="Translator",
        system_message=
        """
        ROLE:
        You are a translator agent proficient in converting text from any language into English.

        TASKS:
        You will be provided with a DatasetDict object containing three features: '__key__', '__url__', and 'txt'. Your task is to translate the text in the 'txt' field into English while leaving the other features unchanged.

            1. Translation Guidelines:
                - If the text in 'txt' is already in English, return the DatasetDict object unchanged.
                - If the text is not in English, translate it into English and update the 'txt' field accordingly.
            
            2. Restrictions:
                - Do not modify the structure or any other features ('__key__', '__url__') of the DatasetDict object.
                - Do not perform any additional tasks beyond the translation.
        
            3. Completion:
                - Once complete, return the updated DatasetDict object, whether changes were made or not.
                - You will only run once. Do not repeat or request further tasks.
        """,
        llm_config=llm_config
    )
    text_analyzer = AssistantAgent(
        name="TextAnalyzer",
        system_message=
        """
        ROLE:
        You are an expert in analyzing text to determine its structure and nature in relation to online content, specifically focusing on whether it resembles a forum post, blog post, or article.
        
        TASK:
        You will receive a DictionaryDict object from another AI agent with three features: '__key__', '__url__', and 'txt'. Your job is to analyze the text in the 'txt' field and provide a detailed examination of the following:

            1. Conversational Structure (Forum Post/Thread):
                - Analyze whether the text shows back-and-forth interaction between MULTIPLE people, resembling a discussion or conversation. Posts with no responses or what appears to be just one person interacting with his or herself is not relevant and does not fit under this category.
                - Look for responses or references to other posts within the text, typical of a forum thread.
                - Note that the presence of a forum-like structure (such as a list of threads) does not automatically qualify the text as a forum post. There must be clear conversational exchanges.
                - Note that some forum posts/threads may contain ads within the website, however this does NOT immediately make it irrelevant. As long as the PRIMARY content fits the specifications outlined above, it is relevant to us.
        
            2. Consistent Narrative (Blog Post/Article):
                - Determine whether the text follows a consistent and cohesive narrative, such as that of a blog post or news article.
                - Explore whether it presents a specific topic, typically related to cybersecurity, or provides information or instructions on the subject matter.
                - Pay attention to any comments under the text as indicators of engagement, but recognize that a front page showing multiple blog/article links is not relevant.
                - Note that some blog post and articles may contain ads within the website, however this does NOT immediately make it irrelevant. As long as the PRIMARY content fits the specifications outlined above, it is relevant to us.
        
            3. Specific Product/Service
                - Any text that indicates a specific malware or hack for the reader to download, purchase, or recieve a service for is relevant for us to capture.
                - The download is only relevant if it is cyberthreat-related and poses harm (basic software that does not pose any harm are not relevant)
                - These downloads typically include a specific entity and type of hack (e.g. 'password cracker', 'account hacker', 'ransomware')
                - Make note of the entities involved and type of hack/malware
                - Only specific services that target singular entities are helpful to us. Many service offerings will be very general or offer a variety of services for a variety of entitites. These are too general and does not provide us with enough specific information, so are irrelevant.

            4. Irrelevant Content:
                - Content that is PRIMARILY advertisements or marketing material does not belong to the categories of forum, blog, or article.
                - If there is a specific malware/hack being offered to the reader, this is NOT considered irrelevant. It is considered irrelevant if there is no specificity and general/variety of services are being offered.
                - Remember, a forum, blog, or article may contain text that appears to be ads, however if the specifications considered to be 'Relevant' are still present, it is NOT considered irrelevant.
                - Content that appears as if someone or a group is offering a service would not belong to the categories of forum, blog, or article. However, a specific software, malware, hack download would still be relevant as it offers us novel information and typically comes from a forum post from a user.
                - Broken text that appears to have no cohesion or consistent narrative is irrelevant.
                - Explain in your analysis why the text may be irrelevant if it does not fit the other specifications.
        
        ANALYSIS OUTPUT:
            - In your analysis, detail the specific structural or thematic elements found in the text that suggest a resemblance to a forum post, blog post, or article.
            - Even if the text fits the 'Irrelevant Content' category, explain why you find it irrelevant to ensure any future AI agents can classify it as such.
            - Provide a thorough examination of the content's characteristics (e.g., signs of conversation, singular narrative, or irrelevant material) to inform other AI agents.
        
        RULES:
            - You will perform your task once and return the __key__ value along with your full analysis of the text.
            - Do not skip any texts. Always return an appropriate response to pass onto the next agent.
            - If you have already examined a text in the past, do not repeat the analysis for that key.
            - Do not classify the text directly. Your task is only to analyze its resemblance to the categories mentioned.
            - Do not ask for additional tasks, inputs, or repeats of the analysis.
        """,
        llm_config=llm_config
    )

    user = UserProxyAgent(
        name="User",
        llm_config=llm_config,
        is_termination_msg=lambda msg: msg.get("content") is not None and "$$$TERMINATE$$$" in msg["content"],
        human_input_mode="NEVER",
        code_execution_config=False,
        max_consecutive_auto_reply=1
    )

    # Create group chat to allow agents to communicate with one another
    gc = GroupChat(agents=[user,translator, text_analyzer], messages=[], allow_repeat_speaker=False, max_round=3)
    gcm = GroupChatManager(groupchat=gc, llm_config=llm_config)

    # Provide initial task to kickstart group chat
    user.initiate_chat(
        recipient=gcm,
        message=f"""
        Translate the text from the DatasetDict object at the end of this message if it is not already in English.
        \nDATASETDICT OBJECT:
        \n{dataset}
        """,
        summary_method="last_msg"
    )

    # Pull final agent output as the analysis
    analysis = gcm.groupchat.messages[-1]["content"]
    
    return analysis

Relevance Agent

def RelevanceAgent(llm_config, analysis):
    relevance_agent = AssistantAgent(
        name="RelevanceAgent",
        system_message=
        """
        ROLE:
        You are an expert in interpreting analysis of a text to determine the relevancy of text based on a defined use case.

        TASKS:
        You will receive a '__key__' and its corresponding analysis that another AI agent produced. Your job is to read the analysis and determine whether the text described in the analysis is from a dark web forum, blog post, or article. You are not required to read the original text. The analysis provided contains all the information you need.

            1. Relevance Determination:
                - If the analysis indicates that the text resembles a dark web forum post, blog post, or article, label it as 'Relevant'.
                - If the analysis indicates a specific software, malware, hack, etc. download is present, label it as 'Relevant'.
                - If the analysis suggests that the text does not fit these categories (e.g., it's an advertisement, front page with a list of posts, etc.), label it as 'Not Relevant'.
        
                a. Relevancy Guidelines:
                    - Forum Post: Must show a clear conversation between MULTIPLE people, such as replies and references to each other's posts. Lists of forum threads without conversation are not relevant. A forum post with no responses or what appears to be just one person interacting with his or herself is not relevant.
                    - Blog Post/Article: Should describe a specific topic, often related to cybersecurity, with a clear, consistent narrative.
                    - Downloads: Includes a specific software, malware, or hack download submitted to a user base from a poster (typically on a forum or blog). The download is only relevant if it is cyberthreat-related and poses harm (basic software that does not pose any harm are not relevant)
                    - Specific Product/Service: A specific service being offered is considered relevant if it is a hack, malware, or vulnerabliity toward a singular entity. General hacking services with little specificity are not relevant.
            
                b. Irrelevancy Guidelines:
                    - Advertisements, listings of posts/pages, or incohesive broken up text without meaningful content do not qualify as relevant and should be labeled 'Not Relevant'.
                    - Any non-cybersecurity/cyberthreat-related content should be labeled 'Not Relevant'.
                    - General hacking service offerings that offer a variety of services targeting a variety of entitties with little specificity should be labeled 'Not Relevant'.
                    - If the analysis indicates ANY of these to be true, the text should be labeled as 'Not Relevant'.

            2. Output Format:
                - Return the __key__, its classification (either 'Relevant' or 'Not Relevant') in a CSV-friendly format.
        
        RULES:
            - Do not skip any texts. Always return an appropriate response to pass onto the next agent.
            - If you have already labeled a text in the past, do not repeat the label for that key.
            - Perform your task once and do not request further tasks or input.
            - Do not repeat your task or analysis.
            - Return only the __key__ and the classification labels as requested. No other characters should be included in your output.
        """,
        llm_config=llm_config
    )
    user = UserProxyAgent(
        name="User",
        llm_config=llm_config,
        is_termination_msg=lambda msg: msg.get("content") is not None and "$$$TERMINATE$$$" in msg["content"],
        human_input_mode="NEVER",
        code_execution_config=False,
        max_consecutive_auto_reply=1
    )

    # Initiate agent by feeding analysis from Text Analyzer Agent
    chat = user.initiate_chat(relevance_agent,
                    message = f"""
                    Based on the following analysis of a darkweb page extract, determine the relevancy of the underlying text.
                    \nANALYSIS:
                    \n{analysis}
                    """, max_turns=1)
    
    # Extract the __key__ and classification label from the agent output
    results = chat.summary.split(",")

    return results

Category Agent

def CategoryAgent(llm_config, analysis):
    category_agent = AssistantAgent(
        name="CategoryAgent",
        system_message=
        f"""
        ROLE:
        You are an expert in interpreting analysis of a dark web text to determine the category that best classifies the text.

        TASKS:
            1. Category Labeling:
            Assign one of the following categories based on the content described in the analysis:
                - Hack: The text details hacking methods or techniques. This typically involves account hacking, password cracking, DDOS attacks, phishing, SQL injection, etc.
                - Malware: The text discusses malware in software, systems, or programs. This may typically be ransomware, viruses, spyware, trojans, keyloggers, etc.
                - Vulnerability: The text describes a vulnerability that can be exploited in a system. This usually involves descriptions of bugs and exploits within software, organizations, or computers that can be taken advantage of.
            2. Output Format:
                - Return the __key__ and the appropriate category determined (either 'Hack', 'Malware', or 'Vulnerability') in a CSV-friendly format.
        
        RULES:
            - Do not skip any texts. Always return an appropriate response to pass onto the next agent.
            - If you have already labeled a text in the past, do not repeat the label for that key.
            - Perform your task once and do not request further tasks or input.
            - Do not repeat your task or analysis.
            - Return only the __key__ and the classification labels as requested. No other characters should be included in your output.
        """,
        llm_config=llm_config
    )
    user = UserProxyAgent(
        name="User",
        llm_config=llm_config,
        is_termination_msg=lambda msg: msg.get("content") is not None and "$$$TERMINATE$$$" in msg["content"],
        human_input_mode="NEVER",
        code_execution_config=False,
        max_consecutive_auto_reply=1
    )

    # Initiate agent by feeding analysis from Text Analyzer Agent
    chat = user.initiate_chat(category_agent,
                message = f"""
                Based on the following analysis of a darkweb page extract, determine the category ('Hack', 'Malware', or 'Vulnerability') that best describes the underlying text.
                \nANALYSIS:
                \n{analysis}
                """, max_turns=1)
    
    # Extract the __key__ and classification label from the agent output
    results = chat.summary.split(",")
    
    return results

The full code can be found at the provided GitHub. More details can be found in the IEEE Access published paper MAD-CTI: Multi-Agent DarkWeb Cyber Threat Intelligence.