Abstract

NGT is a RAG framework specifically developed to address the challenges of applying Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems in the telecommunication domain. It focuses on handling the complex nature of telecom standard documents, particularly 3rd Generation Partnership Project (3GPP) documents. NGT aims to pave the way for utilising LLMs in telecommunications and provides guidelines for RAG implementation in other technical domains. In this paper, the architecture of NGT is described, and a bench-marking review is presented, comparing the system to some reference Large Language Models (LLMs) and other systems.

Introduction

Large Language Models (LLMs) are transformer-based neural language models trained on vast amounts of text data to understand and generate human-like language. These models are designed to learn patterns, context, and grammar, allowing them to generate coherent and contextually relevant responses. Examples include GPT-3, GPT-4, PaLM, LLaMA, and BERT.

Retrieval-Augmented Generation (RAG) enhances LLM capabilities by integrating a retrieval mechanism, which selects relevant documents based on an input query and provides contextual information for more accurate responses. This approach reduces hallucinations and improves reliability.

This paper introduces NGT, a framework that combines an LLM and RAG to assist telecommunication professionals in processing large telecom document corpora, particularly 3GPP specifications. It details the architecture of NGT, presents experimental results, and compares its accuracy to other LLM and non-LLM-based systems.

Methodology

A. Data Preparation
The data preparation process includes:

Creation of a data lake of 3GPP specification documents.
Chunking to divide large documents into manageable segments (1000-character chunks with 200-character overlap).
Embedding to convert textual data into dense vector representations for semantic analysis.
Indexing using Qdrant (Hierarchical Navigable Small World structure) to store and retrieve vectors efficiently.
B. System Design
NGT operates through three key stages:

Retrieve: The system processes the user query, vectorizes it, and retrieves semantically relevant documents using Qdrant.
Augment: Retrieved documents are integrated with the query to provide enriched context before sending it to the LLM.
Generate: The LLM (GPT-3.5 Turbo) processes the augmented query and generates a coherent response.

Experiments

A. Use Cases
NGT was evaluated based on three primary use cases:

Explaining Concepts: Providing clear definitions for telecom-related queries.
Information Aggregation: Extracting and summarising information from multiple documents.
Information Validation: Cross-checking queries against telecom databases for accuracy.
B. Testing Methodology
The model was bench-marked against GPT-3.5, GPT-4, Mistral, and Google Search using:

Ten telecom-related questions derived from 3GPP documentation.
A colour-coded evaluation system (Green = correct, Yellow = partially correct, Red = incorrect) with corresponding scores (2, 1, 0).
C. Bench-marking Results
NGT outperformed other models, scoring 85% accuracy compared to 80% for GPT-3.5, GPT-4, and Mistral, and 70% for Google Search.
The system excelled in providing contextually relevant responses for telecom-specific queries.

Results

The benchmarking results showed that NGT performed better than general-purpose LLMs when handling telecom documentation. The retrieval and augmentation mechanisms significantly improved response accuracy, making NGT a reliable tool for telecom professionals.

Conclusion

NGT provides a structured retrieval, augmentation, and generation (RAG) approach to telecom documentation processing, ensuring more accurate and contextually rich responses. Future work includes expanding NGT’s capabilities to track evolution in telecom standards and provide summarised insights on key specification updates.