NGT is a Retrieval-Augmented Generation (RAG) framework designed to address the challenges of applying Large Language Models (LLMs) in the telecommunication domain. Specifically, it handles the complex nature of telecom standard documents, particularly 3rd Generation Partnership Project (3GPP) documents. This paper presents the architecture of NGT and provides a benchmarking review, comparing its performance with reference LLMs and other retrieval systems. The results highlight the efficiency of NGT in improving documentation processing and accessibility within the telecom sector.
Telecommunication standard documents, such as those from 3GPP, are highly technical and difficult to navigate. Traditional methods for processing such documents require extensive domain expertise. With the rise of LLMs, there is an opportunity to automate and streamline this process. However, general-purpose LLMs struggle with domain-specific jargon and structured references. NGT is introduced as a specialised RAG framework to bridge this gap by integrating domain-specific retrieval mechanisms with generative AI models.
Key Points:
Complexity of telecom documentation
Limitations of general-purpose LLMs
Introduction of NGT as a solution
The use of Large Language Models (LLMs) in technical domains has gained significant traction in recent years. Various Retrieval-Augmented Generation (RAG) frameworks have been proposed to enhance domain-specific applications.
One of the foundational works in RAG is by Lewis et al. (2020), which introduced a framework for knowledge-intensive NLP tasks by incorporating retrieval mechanisms to improve response accuracy【2】. This concept has since been extended to industry-specific applications, including telecommunications.
In the telecom domain, Bornea et al. (2024) highlighted the challenges of adapting RAG models for processing telecommunications documents, particularly in handling complex, highly structured documents like 3GPP specifications【3】. Additionally, Nikbakht et al. (2024) introduced TSpec-LLM, an open-source dataset aimed at improving LLM comprehension of 3GPP standards【6】.
The role of linguistic intelligence in telecom-focused LLMs has been explored by Ahmed et al. (2024), emphasizing the importance of fine-tuning language models to align with domain-specific knowledge【4】. Similarly, Zhou et al. (2024) provided a comprehensive review of LLM applications in telecommunications, discussing key techniques and future opportunities【7】.
Furthermore, Yilma et al. (2024) proposed TelecomRAG, a system designed to enhance document retrieval for telecom standards, demonstrating the feasibility of RAG-based approaches in improving document accessibility【8】.
Despite these advancements, existing solutions lack adaptability to telecom-specific documentation, often struggling with cross-referencing multiple standards and handling structured data efficiently. NGT aims to bridge this gap by offering a highly optimized retrieval pipeline that is specifically tailored for telecom documentation processing.
NGT employs a structured pipeline to process complex 3GPP telecom documentation, combining retrieval-augmented generation (RAG) with domain-specific optimizations. The system architecture ensures accurate information extraction and context-aware response generation through four integrated phases:
Data Processing
Document Pre-processing: Raw 3GPP specifications (TS/TR documents) are cleaned and normalized, removing tables/boilerplate while preserving technical content.
Chunking and Indexing: Documents are segmented into 1,000-character chunks with 200-character overlaps using sliding windows, maintaining contextual continuity for specifications like 38.331 (RRC protocols).
Vectorization: A pretrained transformer model (e.g., BERT) generates 768-dimensional embeddings, capturing semantic relationships between telecom concepts (e.g., "beamforming" ↔ "MIMO").
Retrieval and Augmentation
Retrieval Mechanism: Qdrant's HNSW index performs approximate nearest-neighbor search (recall@5 > 0.92) to retrieve top-5 relevant chunks for queries like "5G-NR UE capability reporting.
Context Augmentation: Retrieved chunks are prepended to queries as context (avg. +420 tokens), enabling GPT-3.5 Turbo to reference exact 3GPP clauses (e.g., "As specified in TS 38.331 Section 5.6.1...").
Response Generation
LLM Integration: GPT-3.5 Turbo is fine-tuned on telecom QA pairs to optimize syntax for technical responses (e.g., prioritizing normative "shall" over descriptive language).
Filtering and Ranking: A confidence-scoring layer rejects low-certainty responses (<70%) and prioritizes answers citing specific 3GPP releases/sections.
Evaluation and Benchmarking
Accuracy Metrics: Domain experts scored responses on 10 curated queries (e.g., "Explain CA bandwidth classes"), achieving 85% accuracy vs. 70-80% for GPT-4/Mistral.
Retrieval Precision: F1=0.82 on relevance classification (e.g., correctly retrieving 38.413 for "5G core network slicing").
Efficiency and Latency: <500ms latency on consumer hardware (AMD Ryzen 5/16GB RAM), enabling real-time use by engineers.
.jpeg?versionId=cNZzleTcaUgTmv648Y7Wqv6KI0EIAi7P&Expires=1787155746&Key-Pair-Id=K3EVG0QTJR4SK0&Signature=tSYgURc3eKqBhdlPz5bhMQwiF4XDZgyezm-UZNwzjW9aem107AYsYRqkzU72H6vCId9gGzboRoVQ~DpufI9vcA-11qhPViFHODt2TClFIKWcVEhoaj9bzfsiZEsz~PAqhShZBrp-V3ngf-fo0vPqj1Uq9sQOKBQczbnh-N6zxmBXADAVsoM6ov4JojwT6sM14EpFUnPEd-38w079eCRyqZDSEy880hcF0PKya0ncOHjnmShKRqf2vLZG0kP6KxQ8JH8529XU7U-~3IO0o4P7Nkp77c~SrbVockDit-V25x1OF31F8Cg1HrhlutrXMn2Np91d2f0vKAGNBNAuh31hOg__)
A series of experiments were conducted to evaluate NGT’s effectiveness. These experiments focused on three primary use cases and several performance metrics.
Use Cases:
Explaining Concepts: NGT retrieved relevant telecom information and provided domain-specific explanations.
Information Aggregation: The model synthesized responses from multiple 3GPP documents to answer complex queries.
Information Validation: NGT verified telecom-related statements by referencing authoritative sources.
Testing Methodology:
Benchmarking Against Other Systems: NGT was compared to GPT-3.5, GPT-4.0, Mistral, and Google Search to assess accuracy and retrieval efficiency.
Evaluation Metrics: The system was evaluated based on response accuracy, latency, retrieval precision, and comprehensibility.
Testing Environment: The experiments were conducted on a high-performance system with AMD Ryzen 5 PRO 7530U, 16GB RAM, and optimized indexing for telecom datasets.
Benchmarking Questions: A set of 10 telecom-specific questions covering industry standards, troubleshooting, and theoretical concepts was used to compare model performance.
Scoring System: Responses were scored from 0 to 2, with 2 points for correct and well-structured answers, 1 for partially correct responses, and 0 for incorrect responses.
A series of experiments were conducted to evaluate NGT’s effectiveness. These experiments were designed to assess the model’s ability to process complex telecom documentation efficiently and accurately. The evaluation focused on three primary use cases and several performance metrics.
The benchmarking results showed that NGT outperformed general-purpose LLMs in handling telecom documentation. The retrieval and augmentation mechanisms significantly improved response accuracy, making NGT a reliable tool for telecom professionals.
Key Findings:
High Recall Rate: Ensuring that the most relevant sections of documents were retrieved, leading to a more accurate representation of the original content.
Improved Response Accuracy: By integrating retrieval-augmented techniques, NGT produced responses that were not only contextually relevant but also grounded in telecom standards.
Performance Comparison: When compared with general-purpose LLMs, NGT demonstrated superior precision in extracting telecom-specific information while reducing the likelihood of generating incorrect responses.
Efficiency Metrics: The system maintained an optimal trade-off between response speed and accuracy, ensuring real-time usability.
.jpeg?versionId=QlaZnP_FcRSX3G9d0ml74lAno3rpO98J&Expires=1787155746&Key-Pair-Id=K3EVG0QTJR4SK0&Signature=Z~Y3IdewrDccVDI~teo7s0tA3bSDZ~Bqjpg2QSbTwqSTuTqY-JH0RcuNNYtwiAYdx6W9pU9yER-Fs~sKO35TW7UR69WEEhagjsDtXEu~bt5xjzRsy74ruDTwIX-kTYziGQ589LHZitTO-FcCNKX7-LQ~fvwc9GHXhjwbUNwjh7Iv~qWGQUPsQl-poqLdtjHijbHFGEZySDfgnuU~AnR8hoBar20R43kmXOqjnvYTCV5NqaBN3GoKiCNXy3jbzXy6Ro~4uHM7sTmLpJGdiBlvOi~b5uNmcTAn2lJQSlqBj0-arUYFElz9PBuxpnbhU7wrqsCBGfl-iSr9Kvd9CFf56A__)
The results indicate that domain-specific retrieval-augmented systems like NGT can significantly enhance technical document processing. Unlike general-purpose LLMs, which often generate inaccurate or overly generic responses, NGT maintains precision by grounding its responses in authoritative telecom documentation.
Interpretation of Results:
-Domain-Specific Effectiveness: NGT’s approach of fine-tuning retrieval for telecom documentation proved essential for handling complex and structured content.
Reduction of Hallucinations: One of the key improvements seen was the reduction of AI-generated misinformation, a common problem in LLMs when dealing with niche topics.
User Feedback: Initial testing with telecom professionals indicated a noticeable improvement in document navigation and understanding.
Challenges & Future Improvements:
Keeping Document Repositories Up to Date: Ensuring that the retrieval mechanism is always referencing the most recent telecom standards.
Optimizing Retrieval Efficiency: While accuracy has improved, there is scope to enhance query processing speed and computational efficiency.
Expanding Adaptability to Other Technical Domains: The methodology applied in NGT can potentially be expanded beyond telecommunications to industries such as healthcare, finance, and legal document analysis.
The benchmarking results showed that NGT outperformed general-purpose LLMs in handling telecom documentation. The retrieval and augmentation mechanisms significantly improved response accuracy, making NGT a reliable tool for telecom professionals. The system demonstrated:
High recall rate – Ensuring that the most relevant sections of documents were retrieved.
Improved response accuracy – Providing structured responses based on authoritative sources.
NGT demonstrates the potential of combining RAG frameworks with domain-specific applications. By addressing the challenges associated with telecom standard documentation, it provides a scalable and accurate solution for industry professionals.
Future Research Directions:
Expanding the dataset
Refining retrieval mechanisms
Improving user interaction interfaces
While NGT has shown promising results in improving telecom documentation processing, there are several areas that can be further enhanced:
The following publications were referenced in this study:
A comprehensive list of references used in the study, including:
The author acknowledges Orange Labs for their support in this research and the provision of telecom datasets. Special thanks to Ayoub Bousselmi, my supervisor at Orange Labs in Chatillon, for his guidance, as well as Ajayi Idowu, Tobias Odion,Ife Ebo Olalekan, and Abdullahi Isa Ahmad for their unwavering support during the project.
The author acknowledges Orange Labs for their support in this research and the provision of telecom datasets.