Tags: llm-security, prompt-injection, guardrails, nlp, presidio, deberta, sentence-transformers, owasp

Publication type: Technical Article / Implementation & Applications

License: Apache 2.0

Author: Manuela Schrittwieser

Repository: https://github.com/MANU-de/llm-leak-detector

Demo video: https://www.loom.com/share/5bda54e4707a458d920e838b2b97e769

Abstract

Large Language Model (LLM) applications face security risks on both sides of the inference boundary. Malicious user prompts can attempt to override system instructions, while model outputs may disclose personally identifiable information (PII), credentials, or proprietary intellectual property. Traditional Data Loss Prevention (DLP) tools often rely on static keyword lists and cannot detect semantically paraphrased code leaks or adversarial jailbreak phrasing framed in administrative language.

Sentinel-LLM V2 addresses this gap with a modular, bi-directional security sidecar implemented in Python. The system combines four specialized detection engines—prompt injection classification, NER-based PII analysis, regex secret matching, and hybrid semantic IP comparison—within a unified orchestration layer that supports blocking, redaction, and cross-layer hardening. When inbound analysis flags injection or low-confidence safe classifications, outbound intellectual property (IP) detection automatically becomes more sensitive, reducing the attack surface for chained exploits.

This document describes the problem context, system architecture, detection methodology, decision logic, implementation details, and reproducibility instructions for Sentinel-LLM V2. It is intended for publication on Ready Tensor and for use by security engineers, ML practitioners, and researchers evaluating LLM guardrail patterns.

Keywords: LLM security, prompt injection, data leakage, guardrails, Presidio, DeBERTa, sentence-transformers, OWASP LLM Top 10, bi-directional filtering

---

Table of Contents

1. Introduction
2. Threat Model and Design Goals
3. System Architecture
4. Detection Engines
5. Cross-Layer Hardening and Decision Logic
6. Implementation Reference
7. Configuration and Thresholds
8. Reproducibility
9. Evaluation Scenarios
10. Limitations
11. Future Work
12. References

---

1. Introduction

1.1 Problem Statement

Deploying LLMs in production introduces two distinct but related security challenges:

1. Inbound threats (OWASP LLM01 — Prompt Injection): Attackers craft inputs designed to bypass system instructions, extract hidden prompts, or coerce the model into unsafe behavior. These attacks may use explicit jailbreak language or subtle administrative phrasing that evades naive filters.

2. Outbound threats (OWASP LLM06 — Sensitive Information Disclosure): Model outputs may contain PII, API keys, database connection strings, or fragments of proprietary source code—either from training data memorization, context window leakage, or hallucination.

Most early LLM guardrails focus exclusively on output filtering. Sentinel-LLM V2 extends this model by evaluating both user prompts and model outputs through a coordinated pipeline, enabling defense-in-depth without requiring access to model weights or internal logits.

1.2 Version 2 Enhancements

Sentinel-LLM V1 provided outbound scanning for PII, secrets, and semantic code similarity. V2 adds:

1.3 Intended Use Cases

• Security engineering portfolios demonstrating practical LLM guardrail design
• Pre-production evaluations of LLM application risk before middleware integration
• Educational demonstrations of OWASP LLM Top 10 mitigations
• Foundation for production middleware (planned FastAPI wrapper in V3)

---

2. Threat Model and Design Goals

2.1 Assets Protected

2.2 Assumptions

• Text is evaluated before delivery to the end user (post-generation filter) or before forwarding to the LLM (pre-inference filter).
• The guardrail operates as an external sidecar; it does not modify model weights.
• Proprietary code snippets are maintained in a local in-memory vault (demo scale: 3 entries).
• Models are downloaded from Hugging Face on first run and cached locally.

2.3 Out of Scope

• Model-level adversarial fine-tuning or RLHF alignment
• Network-level DLP for non-LLM traffic
• Encrypted or multimodal inputs (images, audio)
• Real-time streaming token-by-token filtering

2.4 Design Goals

1. Hybrid detection: Combine ML classification with deterministic heuristics to reduce single-point-of-failure risk.
2. Modularity: Isolate each engine in dedicated modules for independent testing and replacement.
3. Tunability: Expose confidence thresholds for security teams to balance false positives and false negatives.
4. Explainability: Return structured findings with labels, scores, and detection methods.

---

3. System Architecture

3.1 Component Overview

Sentinel-LLM V2 consists of three Python modules and a Streamlit presentation layer:

┌─────────────────────────────────────────────────────────────────────┐
│                         app.py (Streamlit UI)                       │
│   Sliders: PII threshold | Code similarity threshold                │
│   Metrics: Injection status | PII | Secrets | IP leaks              │
└───────────────────────────────┬─────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    detector.py (LLMLeakDetector)                    │
│                         run_report() orchestrator                   │
│                                                                     │
│  ┌─────────────────┐   ┌──────────────┐   ┌─────────────────────┐  │
│  │ Injection       │   │ Presidio     │   │ Secret Matcher      │  │
│  │ Detector        │   │ PII Scanner  │   │ (Regex)             │  │
│  │ (inbound)       │   │ (outbound)   │   │ (outbound)          │  │
│  └────────┬────────┘   └──────┬───────┘   └──────────┬──────────┘  │
│           │                 │                       │              │
│           │    Cross-Layer  │                       │              │
│           └────── Hardening ──┼───────────────────────┘              │
│                             ▼                                       │
│                  ┌─────────────────────┐                            │
│                  │ Hybrid IP Comparator │                           │
│                  │ (outbound)           │                           │
│                  └──────────┬──────────┘                            │
│                             ▼                                       │
│                  ┌─────────────────────┐                            │
│                  │ Decision Logic       │                           │
│                  │ BLOCK or CLEAN       │                           │
│                  └─────────────────────┘                            │
└─────────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────────┐
│              injection_engine.py (InjectionDetector)              │
│         DeBERTa-v3 classifier + regex heuristic patterns          │
└─────────────────────────────────────────────────────────────────────┘

3.2 Processing Pipeline

Every security audit follows a fixed four-stage pipeline:

Input Text
    │
    ▼
[Stage 1] Inbound Injection Scan
    │
    ▼
[Stage 2] Cross-Layer Hardening (adjust IP threshold if suspicious)
    │
    ▼
[Stage 3] Outbound Scans (PII, Secrets, IP Leakage)
    │
    ▼
[Stage 4] Decision Logic (BLOCK with reason, or CLEAN with redaction)
    │
    ▼
Structured Report (status, final_text, findings)

3.3 Technology Stack

---

4. Detection Engines

4.1 Inbound: Prompt Injection Detector

Module: injection_engine.py
Model: protectai/deberta-v3-base-prompt-injection
Approach: Hybrid ML classification with regex override

The injection detector implements a three-step scan:

Step A — Heuristic Signature Matching

Before invoking the ML model, the engine checks input against a curated list of high-risk regex patterns. These patterns target jailbreak phrases that may evade classifiers when framed in professional or administrative language:

If any pattern matches, the engine returns INJECTION (Heuristic) with confidence 1.0, bypassing the ML classifier entirely.

Step B — ML Behavioral Classification

When no heuristic match occurs, text is passed to the DeBERTa-v3 classifier via Hugging Face's pipeline("text-classification"). The model returns a label (SAFE or injection-related) and a confidence score.

Step C — Low-Confidence Safe Flagging

If the model returns SAFE with confidence below 90%, the input is relabeled as SUSPICIOUS. This conservative policy triggers cross-layer hardening without immediately blocking the request, reflecting a "guilty until proven innocent" stance for borderline inputs.

Return values: (label: str, score: float)

---

4.2 Outbound: PII Detection

Module: detector.py → scan_pii()
Engine: Microsoft Presidio AnalyzerEngine

Presidio performs Named Entity Recognition (NER) over English text, detecting:

• EMAIL_ADDRESS
• PERSON
• US_SSN (via custom pattern recognizer)

A custom SSN recognizer supplements Presidio's default registry:

ssn_pattern = Pattern(
    name="ssn_pattern",
    regex=r"\b\d{3}-\d{2}-\d{4}\b",
    score=1.0
)

Results are filtered by a configurable confidence threshold (pii_threshold, default 0.4). Lower thresholds increase recall at the cost of false positives.

When the final status is CLEAN, detected PII is replaced via Presidio's AnonymizerEngine with the token [REDACTED_PII].

---

4.3 Outbound: Secret Scanning

Module: detector.py → scan_secrets()
Approach: Deterministic regex matching

Two secret categories are defined:

Secrets are redacted with [REDACTED_SECRET] during the clean-path output sanitization.

---

4.4 Outbound: Hybrid IP Leakage Detection

Module: detector.py → scan_code_leakage()
Approach: Two-layer hybrid (keyword first, semantic fallback)

This engine protects a proprietary code vault—an in-memory list of sensitive snippets representing internal functions and connection strings. V2 uses a two-pass strategy:

Layer A — Fuzzy Keyword Match

For each vault snippet, the engine extracts:

1. Full identifier: The function or variable name (e.g., internal_secure_auth_protocol)
2. Fuzzy segment: The last three underscore-separated tokens (e.g., proprietary_risk_score)

If either appears in the input text (case-insensitive), a match is recorded with score 1.0 and method Keyword Match (Fuzzy).

Layer B — Semantic Similarity

If Layer A finds no matches, the input is embedded using Sentence-Transformers and compared against pre-computed vault embeddings via cosine similarity:

Matches exceeding the configurable code_threshold (default 0.7) are flagged with method Semantic Similarity.

This hybrid design catches both explicit identifier mentions and paraphrased reproductions that keyword lists would miss.

---

5. Cross-Layer Hardening and Decision Logic

5.1 Cross-Layer Hardening

When the injection scan returns any label other than SAFE (including SUSPICIOUS), the IP similarity threshold is automatically reduced:

effective_code_threshold = max(0.1, code_threshold - 0.2)

Rationale: Adversarial prompts that attempt to extract proprietary logic often combine injection techniques with indirect code requests. Tightening outbound IP detection during suspicious inbound states reduces the window for chained attacks.

5.2 Blocking Conditions

The response is BLOCKED when either condition is true:

1. Injection label starts with INJECTION (includes heuristic detections)
2. One or more IP leakage findings are present

Blocked responses return a standardized message:

[BLOCKING RESPONSE: {reason} DETECTED]

where {reason} is the injection label or IP_LEAKAGE.

5.3 Clean Path and Redaction

When no blocking condition is met, status is CLEAN. The pipeline applies sequential redaction:

1. Presidio anonymizes PII entities → [REDACTED_PII]
2. Secret values are string-replaced → [REDACTED_SECRET]

PII and secrets do not trigger blocking in V2; they are sanitized. Blocking is reserved for injection and IP leakage, reflecting a policy that credential exposure requires redaction while proprietary code disclosure requires hard stops.

5.4 Report Schema

The run_report() method returns a structured dictionary:

{
    "status": "BLOCKED" | "CLEAN",
    "final_text": str,          # Redacted output or block message
    "findings": {
        "inj": (label: str, score: float),
        "pii": list,              # Presidio RecognizerResult objects
        "secrets": list,          # {"type": str, "value": str}
        "leaks": list             # {"score": float, "matched_snippet": str, "method": str}
    }
}

---

6. Implementation Reference

6.1 Module Responsibilities

6.2 Core Orchestration

The central audit method coordinates all engines:

def run_report(self, text: str, pii_threshold: float, code_threshold: float):
    # 1. Inbound check
    inj_label, inj_score = self.scan_injection(text)

    # 2. Cross-layer hardening
    effective_code_threshold = (
        code_threshold if inj_label == "SAFE"
        else max(0.1, code_threshold - 0.2)
    )

    # 3. Outbound checks
    pii = self.scan_pii(text, pii_threshold)
    secrets = self.scan_secrets(text)
    code_leaks = self.scan_code_leakage(text, effective_code_threshold)

    # 4. Decision logic
    if inj_label.startswith("INJECTION") or code_leaks:
        status = "BLOCKED"
        reason = inj_label if inj_label.startswith("INJECTION") else "IP_LEAKAGE"
        redacted = f"[BLOCKING RESPONSE: {reason} DETECTED]"
    else:
        status = "CLEAN"
        # ... redaction logic ...

    return {"status": status, "final_text": redacted, "findings": {...}}

6.3 Model Loading and Caching

• DeBERTa-v3: Loaded once via @st.cache_resource in the Streamlit app, preventing reload on every interaction.
• Sentence-Transformer: Vault embeddings are pre-computed at initialization to avoid redundant encoding during scans.
• Presidio: AnalyzerEngine and AnonymizerEngine are instantiated once per detector lifecycle.

6.4 First-Run Behavior

On first execution, Hugging Face downloads:

• protectai/deberta-v3-base-prompt-injection (~440 MB)
• all-MiniLM-L6-v2 (~90 MB)

SpaCy model en_core_web_lg is installed via requirements.txt. Expect 2–5 minutes for initial setup depending on network speed.

---

7. Configuration and Thresholds

7.1 UI Controls

7.2 Internal Constants

7.3 Proprietary Vault (Demo)

The default vault contains three representative snippets:

proprietary_vault = [
    "def internal_secure_auth_protocol(user_id, secret_salt):",
    "db_connection = create_engine('postgresql://internal_prod_db:5432')",
    "def calculate_proprietary_risk_score(data):"
]

Production deployments should replace this with organization-specific code corpora, ideally backed by a vector database (planned for V3).

---

8. Reproducibility

8.1 Prerequisites

• Python 3.11 or later
• Virtual environment (recommended)
• Approximately 2 GB disk space for model artifacts
• Internet access for first-run model downloads

8.2 Installation

git clone https://github.com/MANU-de/llm-leak-detector.git
cd llm-leak-detector
python -m venv venv
source venv/bin/activate   # Windows: venv\Scripts\activate
pip install -r requirements.txt

8.3 Running the Application

streamlit run app.py

The application opens in the default browser at http://localhost:8501.

8.4 Key Dependencies

Note: The pinned requirements.txt reflects a full environment freeze, including CUDA-related packages from the author's development machine. CPU-only deployments may require dependency adjustment.

8.5 Repository Structure

llm-leak-detector/
├── app.py                 # Streamlit UI
├── detector.py            # Main orchestrator and outbound engines
├── injection_engine.py    # Inbound injection detector
├── requirements.txt       # Pinned dependencies
├── README.md              # Project overview
├── LICENSE                # Apache 2.0
└── docs/
    └── sentinel-llm-v2-technical-documentation.md

---

9. Evaluation Scenarios

The following test inputs validate each detection layer. Paste them into the application text area and run a security audit.

9.1 Prompt Injection (Expected: BLOCKED or SUSPICIOUS with hardening)

9.2 PII Leakage (Expected: CLEAN with redaction)

Input:

The user John Doe (j.doe@email.com) requested an update for SSN 000-11-2222.

Expected: Status CLEAN; names, email, and SSN replaced with [REDACTED_PII].

9.3 Secret Leakage (Expected: CLEAN with redaction)

Input:

To access the production DB, use API_KEY: sk-ant-api03-abcdefg12345.

Expected: Status CLEAN; API key replaced with [REDACTED_SECRET].

9.4 IP Leakage (Expected: BLOCKED)

Input:

I will create a function called internal_secure_auth_protocol(user_id, secret_salt) for the backend.

Expected: Status BLOCKED; fuzzy keyword match on function identifier.

9.5 Semantic IP Leak (Expected: BLOCKED at lower thresholds)

Paraphrased descriptions of vault logic without exact identifier names may trigger Layer B semantic matching when similarity exceeds the configured threshold.

---

10. Limitations

1. Demo-scale vault: The proprietary code vault contains three hardcoded snippets. Enterprise deployments require scalable vault management (vector DB integration planned).
2. English-only: Presidio and SpaCy are configured for English (language='en'). Multilingual PII detection is not supported.
3. Batch, not streaming: The guardrail evaluates complete text blocks; token-by-token streaming protection is not implemented.
4. No audit export: Findings are displayed in the UI but not persisted to JSON/SIEM (planned).
5. Environment-specific dependencies: Full pip freeze output may include CUDA packages incompatible with all hardware.
6. Heuristic pattern coverage: Regex signatures cover known jailbreak families but cannot guarantee coverage of novel attack vectors.
7. PII does not block: Policy choice redacts PII rather than blocking; organizations requiring hard stops on any PII exposure may need policy adjustments.

---

11. Future Work

---

12. References

1. OWASP Foundation. OWASP Top 10 for Large Language Model Applications. https://owasp.org/www-project-top-10-for-large-language-model-applications/
2. Protect AI. deberta-v3-base-prompt-injection. Hugging Face Model Hub. https://huggingface.co/protectai/deberta-v3-base-prompt-injection
3. Microsoft. Presidio: Data Protection and De-identification SDK. https://microsoft.github.io/presidio/
4. Reimers, N., & Gurevych, I. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. https://www.sbert.net/
5. Project Repository. Sentinel-LLM / llm-leak-detector. https://github.com/MANU-de/llm-leak-detector
6. Technical Demo. Sentinel-LLM Loom Walkthrough. https://www.loom.com/share/5bda54e4707a458d920e838b2b97e769

---

@misc{sentinel_llm_v2,
  author       = {Manuela Schrittwieser},
  title        = {Sentinel-LLM V2: A Bi-Directional Guardrail for Prompt Injection and Sensitive Data Leakage},
  year         = {2026},
  publisher    = {Ready Tensor},
  howpublished = {\url{https://github.com/MANU-de/llm-leak-detector}},
  note         = {Apache License 2.0}
}

---