Project Description

This project represents the final phase of LoRAfrica deployment, making the system accessible to users on the web through scalable serverless infrastructure using Modal and an interactive frontend built with Streamlit.

Aim & Objectives

Aim
Deploy LoRAfrica via Modal and Streamlit for seamless web-based user interaction.

Objectives

• Benchmark performance using Naive and vLLM approaches while measuring TTFT, ITL, TPOT, E2E, and TPS
• Deploy LoRAfrica using Modal serverless GPU infrastructure
• Enable logging and observability with LangSmith, including PII redaction
• Build and render a Streamlit-based frontend for user interaction

Methodology

The first phase of the project with making LoRAfrica available on user consumer hardware before deploying unto Modal and Streamlit. The base model microsoft/Phi-4-mini-instruct was merged with the LoRA adapters of DannyAI/phi4_african_history_lora_ds2_axolotl which was then pushed to HuggingFace. This allowed a unified model for which developers can further fine-tune id desired.

The Base model and Lora Model were quantised using Llama.cpp in the q4_k_m.gguf format, which reduced the model size to about 2GB (base model) and adapters to 6MB. Make it fully available and runable on consumer hardware via platforms such as Ollama and Llama.cpp interactive UI apps.

Below are the relevant links for LoRAfrica for CPU

• Ollama Model Repo
• LoRAfrica for CPU
• LoRAfrica GGUF HuggingFace Repo

LoRAfrica's Serverless Compute Deployment involved the following;

• Benchmarking TTFT, ITL, TPOT, E2E, and TPS with naive approach (baseline) against the vLLM approach.
• Deployment of Model using Modal while enabling LangSmith traces and serving the Model frontend via Streamlit

Metrics Explained

Below are the brief overview of the metrics measured;

1. TTFT (Time to First Token)

Measures the latency of the first response.

Unit: ms

---

2. ITL (Inter-Token Latency)

The time elapsed between two consecutive tokens.

Unit: ms

---

3. TPOT (Time Per Output Token)

The average generation time per token, excluding the prefill (TTFT) phase.

Unit: ms

---

4. E2E (End-to-End Latency)

The total time from request to the final token.

Unit: ms

---

5. TPS (Tokens Per Second)

The overall throughput of the inference engine.

Unit: tokens/sec

---

Summary Table

Benchmarks were conducted using a custom dataset as shown below;

SYSTEM_PROMPT = "You are a helpful AI assistant specialised in African history which gives concise answers to questions asked."
TEST_PROMPTS = [
    "Who was the richest king in history from the Mali Empire?",
    "Explain the significance of the Great Zimbabwe ruins.",
    "Describe the impact of the Ashanti Empire on West African trade.",
    "What was the role of the Kingdom of Aksum in early Christianity?",
    "Detail the leadership of Queen Amina of Zazzau."
]

For the vLLM approach, the dataset above was converted to a .jsonlformat, as vLLM expects such as shown below;
Template used;

full_prompt = f"<|system|>\n{SYSTEM_PROMPT}\n<|user|>\n{user_q}\n<|assistant|>\n"

Example format

{"prompt": "<|system|>\nYou are a helpful AI assistant specialised in African history which gives concise answers to questions asked.\n<|user|>\nWho was the richest king in history from the Mali Empire?\n<|assistant|>\n", "output_len": 128}

Results

The results of the naive vs vLLM approach are shown and explained briefly below;

1. Global TPS (Tokens Per Second)

Interpretation:

• vLLM achieves ~10–100× higher throughput than the naive approach across all batch sizes.
• Naive approach (FastAPI + HuggingFace) TPS saturates at ~17 tokens/sec, indicating GPU underutilization.

---

2. TTFT (Time To First Token, ms)

Interpretation:

• Naive approach suffers from extreme TTFT at high batch sizes; vLLM maintains low and predictable TTFT.

---

3. TPOT (Time Per Output Token, ms)

Interpretation:

• TPOT is consistently lower for vLLM, showing faster per-token generation when compared to the Naive approach.

---

4. ITL (Inter-Token Latency, ms)

Interpretation:

• vLLM shows much lower ITL compared to naive, confirming faster streaming and token delivery.

---

5. E2E (End-to-End Latency, ms)

Interpretation:

• vLLM significantly reduces E2E latency compared to naive, especially at high batch sizes.
• Naive approach exhibits huge spikes at P95/P99 due to GPU contention, Python threading overhead, and lack of efficient batching.
• vLLM shows low tail latency, making it much more predictable and scalable.

---

Overall Analysis

• Throughput: vLLM outperforms naive FastAPI by ~10–100×, scaling efficiently with batch size and concurrency.
• Token Generation: TPOT and ITL are much lower and stable in vLLM.
• Latency: TTFT and E2E are dramatically reduced in vLLM; naive FastAPI suffers long-tail spikes.
• System Behavior:
  • Naive approach is bottlenecked by Python threading, HuggingFace generation inefficiencies, and GPU underutilization.
  • vLLM optimizes scheduling, streaming, and batching for high concurrency.

This is why vLLM was used for modal deployment via Modal's Serverless Compute.

Deployment

Modal

Modal is a serverless cloud platform for running AI and machine learning workloads without managing infrastructure. It handles scaling, GPU provisioning, and execution automatically, allowing developers to focus on building and deploying models.

Key Features

• Serverless execution (no infrastructure setup)
• Automatic scaling (CPU & GPU)
• On-demand GPU access
• Fast deployment with low latency
• Code-first workflow

In this setup the,
L4 GPU and 2 T4 GPU (a fallback safety net) was deployed. Which allows 100 concurrent requests with an allowance of 10 containers. An idle container was not allowed so as to manage costs in infrastructure management

Estimating Cost Per Token for L4/T4 GPUs

1️⃣ Formula

Cost per token = GPU cost per hour ÷ tokens per hour throughput

• GPU cost per hour: from Modal
  • $0.80 for L4,
  • $0.59 for T4
• Tokens per hour: estimate based on model’s generation speed

---

2️⃣ Estimate Tokens Per Hour

Max output tokens per request: 128
The Baseline Estimates

• Assume L4 throughput ≈ 1,000,000 tokens/hour
• Assume T4 throughput ≈ 750,000 tokens/hour

---

3️⃣ Compute Cost Per Token

L4:

cost per token = 0.80 / 1,000,000 = 0.0000008 USD/token

T4:

cost per token = 0.59 / 750,000 ≈ 0.000000787 USD/token

---

4️⃣ Split Input vs Output Cost (example calculation for a request on L4 compute)

If your prompt is ~50 tokens, output is 128 tokens:

• Total tokens per request = 50 + 128 = 178

Fractional cost:

input cost per token  = 0.0000008 * (50 / 178) = 0.000000224
output cost per token = 0.0000008 * (128 / 178) = 0.000000576

---

Actual Compute Cost

input_cost = input_tokens * input_cost_per_token = 
50 * 0.000000224 = 0.0000112 USD

output_cost = output_tokens * output_cost_per_token = 
128 * 0.000000576 = 0.000073728 USD

total_cost = input_cost + output_cost = 
0.0000112 + 0.000073728 = 0.000084928 USD

This means per-request cost is extremely low; Making serverless LLM deployment economically viable at scale. Moreover, this means 10,000 requests would cost less than $1, highlighting the cost-efficiency of optimised LLM deployment using modern GPU infrastructure.

LangSmith

LangSmith is a development and monitoring platform for building, debugging, and evaluating LLM-powered applications. It provides deep visibility into how models and agents behave, making it easier to trace, test, and improve performance.

Key Features

• End-to-end tracing of LLM calls and workflows
• Debugging and inspection of prompts, responses, and chains

In this project, it was used to trace the LLM responses and ensuring PIIs were being redacted on the developer's side to ensure privacy of users.

Redaction of users' information was achieved using a hybrid approach; regular expressions and Microsoft's Presidio anonymizer & Presidio analyzer; which recognises entities allowing for redaction where regular expressions fails.

For further safety concerns; the system prompt was structured in a way that it hidden as a secret on Modal and immune to prompt injections. I cannot share the full prompt but here is an example way to do this;

"""
### ROLE
You are LoRAfrica, an expert-level repository of African Historical Knowledge........

### DOMAIN LAYER


### SECURITY & ANTI-INJECTION LAYERS
- **The Obsidian Firewall:** 

In addition to this, the code is structured in a way that rejects systems prompts provided by users; thereby ensuring safety of the platform.

Streamlit

Streamlit is an open-source Python framework for building and sharing interactive web applications for data science and machine learning projects. It allows developers to create UI components directly in Python without needing frontend development skills.

Key Features

• Simple Python-based UI development
• Real-time interaction and updates
• Easy integration with data science & ML libraries
• Fast deployment and sharing
• Built-in support for visualizations

Streamlit was used to quickly build and deploy an interactive frontend interface for LoRAfrica.

With the frontend being managed by Streamlit, there were some noticeable latency; a drawback well known in Streamlit but good for testing purposes.

NB: Before the use of Streamlit, server side was tested using POSTMAN to send requests and generate outputs from the model.

Conclusion

LoRAfrica demonstrates an end-to-end pipeline for deploying a domain-specific Large Language Model in LLMOps, from fine-tuning and benchmarking to scalable production deployment. By combining LoRA fine-tuning with efficient inference strategies (naive and vLLM), the project evaluates key performance metrics such as TTFT, ITL, TPOT, E2E, and TPS to optimise real-world performance.

The system leverages Modal for serverless GPU-based deployment, enabling automatic scaling and simplified infrastructure management, while Streamlit provides an accessible web interface for user interaction. Additional integrations such as LangSmith ensured observability, logging, and PII redaction for safer and more reliable usage.

Overall, the project highlights how modern tools can be orchestrated to deliver a scalable, efficient, and production-ready AI application tailored to African history knowledge.

Key Contributions

• Built a domain-specific LLM using LoRA fine-tuning for African history
• Benchmarked inference performance using both naive and vLLM approaches
• Measured and analysed key latency and throughput metrics (TTFT, ITL, TPOT, E2E, TPS)
• Deployed the model using serverless GPU infrastructure via Modal
• Developed an interactive frontend using Streamlit for real-time user interaction
• Integrated LangSmith for monitoring, logging, and PII redaction

Issues Faced

• Setting up LiteLLM as a proxy to monitor billing and token usage which would allow alerting messages on SLACK
• Trying to rent the needed GPU on Runpod to condut experimentation of naive and vLLM approaches
• Pattern recognition to redact PII's in LangSmith

Example responses

• Malicious prompt-1
  

• Malicious prompt-2
  

• POSTMAN Base Model
  

• POSTMAN LoRAfrica Model
  

• Streamlit Interface
  

• LangSmith tracing and Redaction-1
  

• LangSmith tracing and Redaction-2
  

LoRAfrica-Streamlit