Abstract

This publication documents the deployment strategy and implementation for a Nigerian news headline generation system, building upon a fine-tuned Qwen 2.5 0.5B model from Module 1. I implemented a custom FastAPI inference server using HuggingFace Transformers with automatic CPU/GPU detection, and containerised it with Docker for production portability. The deployment maintains the quality improvements from Module 1 (ROUGE-1: 31.81%, +17% over baseline) while achieving production-grade latency (<200ms on GPU) through 4-bit quantization. I conducted comprehensive cost analysis comparing GPU vs CPU deployment scenarios, and designed a monitoring and observability strategy using industry-standard tools. Security measures implemented in the current prototype include Pydantic input validation and structured error handling; rate limiting, authentication, and container hardening are designed for production and documented as a roadmap. The complete implementation includes automated testing with 20+ real Nigerian news examples across multiple categories.

Key Contributions:

• Custom FastAPI inference server using HuggingFace Transformers with automatic CPU/GPU detection
• Docker containerisation with model cache persistence, health checks, and restart policy
• Comprehensive cost analysis for GPU (~140/month) scenarios
• Monitoring and observability strategy using CloudWatch, Prometheus, and LangSmith
• Pydantic input validation and error handling implemented; full production security roadmap documented
• Automated testing suite with 20+ real Nigerian news examples across 8 categories

Repository: GitHub
Model: HuggingFace

---

1. Use Case Definition

1.1 Problem Statement

Nigerian news organisations face a measurable productivity bottleneck: journalists and editors spend 15–20 minutes crafting headlines per article, directly constraining content velocity and publication timeliness. Generic, Western-trained headline models fail to capture Nigerian context — local institutions (INEC, EFCC, NDDC), party abbreviations (PDP, APC), geographic markers, and the editorial conventions of outlets like Vanguard, Punch, and ThisDay.

The system must generate concise, engaging, editorially appropriate headlines (5–15 words) from Nigerian news article excerpts in real time (<200ms on GPU).

1.2 Target Users

News Editors at Nigerian media outlets (Arise TV, Vanguard, Punch, ThisDay): rapid headline generation for breaking news; 50–200 articles/day per outlet.

Content Management Systems: automated headline suggestions in CMS workflow; 500–1,000 requests/day; API integration required.

News Aggregators: standardising headlines across sources; 10,000+ requests/day; high throughput and cost efficiency critical.

1.3 Input / Output Examples

Example 1 — Political News

Input: "President Bola Tinubu has approved the appointment of new heads for several federal agencies as part of his administration's restructuring efforts..."

Fine-tuned output: "Tinubu Approves New Appointments for Federal Agencies" (7 words, concise)

Baseline output: "President Tinubu's Government Restructures Federal Agencies with New Leadership" (10 words, verbose)

Example 2 — Economic News

Input: "Nigeria's inflation rate climbed to 33.40% in July 2024, according to the latest report from the National Bureau of Statistics..."

Fine-tuned output: "Nigeria Inflation Rate Climbs to 33.40% in July 2024" (prioritises the key figure)

Baseline output: "National Bureau of Statistics Reports Record-High Inflation Driven by Food and Energy"

Example 3 — Sports News

Input: "The Super Eagles of Nigeria secured their qualification for the 2025 Africa Cup of Nations after a commanding 3-1 victory over South Africa..."

Fine-tuned output: "Super Eagles Secure AFCON 2025 Qualification After South Africa Win" (uses correct terminology)

Baseline output: "Nigeria's National Team Advances to AFCON Following Three-Goal Performance"

1.4 Success Criteria

1.5 Traffic Expectations

Peak hours follow Nigerian news cycles: 6–9 AM and 5–8 PM WAT. Weekend traffic runs at roughly 40% of weekday volume. Major events (elections, disasters, sports finals) generate 5–10× spikes. Approximately 90% of traffic originates from Lagos, Abuja, and Port Harcourt.

---

2. Model Selection & Configuration

2.1 Model Configuration Summary

2.2 Model Selection Rationale

Selecting the right model for production deployment requires balancing quality, latency, cost, and operational complexity. Three candidate models were evaluated:

Qwen 2.5 0.5B Instruct was selected for four reasons:

• Size efficiency: at 494M parameters it fits comfortably on a T4 GPU with 4-bit quantization (~4 GB VRAM), leaving headroom for concurrent requests.
• Strong baseline: Qwen 2.5 delivers competitive performance for its size, confirmed by the 27.16% ROUGE-1 zero-shot baseline — a higher floor than comparable-size alternatives.
• Quantization compatibility: documented and tested compatibility with NF4 4-bit quantization via BitsAndBytes, enabling a 75% reduction in memory footprint with negligible quality loss.
• Deployment simplicity: the model loads directly via AutoModelForCausalLM.from_pretrained() with no proprietary inference runtime required.

2.3 Quantization Configuration

4-bit NF4 (NormalFloat) quantization is applied at inference time using the BitsAndBytes library. The exact configuration from deploy_nigerian_headlines.py:

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,   # nested quantization saves ~0.4 GB
    bnb_4bit_compute_dtype=torch.bfloat16,  # stable dequantization on CUDA
)

Double quantization applies a second round of quantization to the quantization constants themselves, reducing memory by a further 0.1–0.4 GB. On CPU, the model loads at full float32 precision (~12 GB RAM, 800–1,200ms latency) — acceptable for low-traffic or batch deployments.

2.4 Fine-Tuning Summary (Module 1 Results)

The base model was fine-tuned using QLoRA on 4,286 samples from the okite97/news-data dataset (AriseTv Nigerian news corpus). Only 1.08M of 494M parameters (0.22%) were trained, using rank-8 LoRA adapters targeting the query and value projection layers. Training completed in 18 minutes on a single T4 GPU.

The merged model is publicly available at huggingface.co/Blaqadonis/Qwen2.5-0.5B-Nigerian-News-Headlines and is loaded directly by the deployment server.

---

3. Deployment Strategy

3.1 Platform Selection

Several deployment platforms were evaluated before choosing self-hosted FastAPI on EC2:

Self-hosted FastAPI was selected because the model is small enough that managed infrastructure overhead is not justified, the 4-bit NF4 quantization configuration requires direct BitsAndBytes access not available on managed endpoints, and the deployment must be reproducible across environments with no vendor lock-in.

Geographic Region: ap-south-1 (Mumbai) is the recommended primary region for Nigerian deployments, offering the lowest latency from Lagos and Abuja via West Africa's undersea cable connections. eu-west-1 (Ireland) is the recommended failover region. us-east-1 is avoided due to the additional round-trip latency from West Africa.

3.2 Architecture Overview

The deployment consists of a custom FastAPI application server built on HuggingFace Transformers, containerised with Docker, serving the quantized fine-tuned model over HTTP. This is a direct inference setup — not a managed inference service. The model is loaded via AutoModelForCausalLM, tokenization is handled by the application, and generation is called via model.generate(). This design gives full control over prompt construction, quantization, streaming, and benchmarking logic.

Request flow: client → FastAPI endpoint → build_prompt() → tokenizer → model.generate() → extract_headline() → JSON response

3.3 FastAPI Server

The server (deploy_nigerian_headlines.py) implements automatic device detection on startup:

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
DTYPE  = torch.bfloat16 if DEVICE == "cuda" else torch.float32

On GPU, the model loads with 4-bit quantization and device_map="auto". On CPU, it loads at full float32 precision with low_cpu_mem_usage=True. A single codebase handles both environments with no code changes needed.

3.4 Prompt Construction

Each request is formatted using Qwen's chat template via tokenizer.apply_chat_template, wrapping the task instruction and excerpt in the expected user turn structure:

You are an expert Nigerian news headline writer.
Generate a compelling, concise headline that captures the essence
of the following news excerpt. Make it engaging and suitable for Nigerian media.

## News Excerpt:
{excerpt}
## Headline:

Post-generation, extract_headline() strips the prompt prefix, isolates the first output line, and removes surrounding quotes or special characters.

3.5 Docker Containerisation

The Docker image builds on nvidia/cuda:12.1.0-base-ubuntu22.04, installs Python 3.11, and exposes port 8000. The following features are implemented in the current repository:

• Model cache volume: weights are persisted in a named Docker volume (model-cache:/app/model_cache), so the 4 GB model download only occurs on first start. Subsequent restarts use the cached weights.
• GPU / CPU switching: the deploy.resources block in docker-compose.yml reserves one NVIDIA GPU. Commenting it out and setting DEVICE=cpu switches to CPU-only mode without rebuilding the image.
• Health check: Docker polls GET / every 30 seconds with a 10-second timeout and 3 retries. The container is only marked healthy after the model has fully loaded and the server is responding.
• Restart policy: unless-stopped ensures automatic recovery from crashes, OOM events, or host reboots.
• HuggingFace token: passed as HF_TOKEN environment variable — not baked into the image.

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8000/"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 60s

3.6 Scaling Strategy

MVP (100–500 req/day): single container on a t3.xlarge (CPU) or g4dn.xlarge (GPU). No load balancer needed. The Docker health check and restart policy provide sufficient reliability for a pilot deployment.

Growth (1,000–5,000 req/day): two GPU containers behind an AWS Application Load Balancer. The model cache volume is mounted read-only on both containers, avoiding duplicate downloads. The ALB distributes requests across both containers and handles health-based routing automatically.

Scale (10,000+ req/day): Kubernetes (EKS) or Docker Swarm with horizontal pod autoscaling based on CPU utilisation and request queue depth. Redis response caching is added for repeated or near-identical excerpts (common in breaking news cycles), reducing effective GPU load by an estimated 15–20%.

For predictable traffic spikes — elections, major sports finals, disaster coverage — pre-warmed spot instance pools can be provisioned 30–60 minutes in advance using AWS Auto Scaling scheduled actions, avoiding cold-start latency under sudden load.

3.7 Streaming Support

The /generate_stream endpoint uses HuggingFace's TextIteratorStreamer running in a background thread. This allows clients to begin rendering tokens as they are produced rather than waiting for the full sequence, improving perceived responsiveness for interactive CMS integrations. The Python client library (client.py) exposes this as a generator:

for token in client.generate_stream("Nigeria's inflation rate..."):
    print(token, end='', flush=True)

3.8 Latency Benchmarking Endpoint

The /ttft_itl_batched endpoint measures TTFT, TPOT, and ITL using synthetic random token inputs at configurable batch sizes. It uses ThreadPoolExecutor to simulate concurrent requests and returns mean, median, and P99 statistics, allowing operators to characterise inference performance on any hardware before committing to production capacity.

---

4. Cost Analysis

4.1 Infrastructure Cost Model

Two primary deployment scenarios are analysed on AWS EC2 at on-demand pricing, 24/7 uptime.

The GPU instance is recommended for any deployment where P99 latency matters to end users. The CPU instance is viable for batch processing workflows where 800–1,200ms latency is acceptable.

4.2 Per-Request Cost

At 10,000 requests/day on CPU, the 800–1,200ms latency means a single instance handles at most ~1 req/s, requiring multiple CPU instances that eliminate the cost advantage entirely.

4.3 Cost per 1,000 Requests

4.4 Cost Optimization Strategies

• Spot instances: the single largest cost lever, saving 60–70%. Headline generation is stateless — requests can be retried on spot interruption with no data loss.
• Scheduled off-hours scaling: Nigerian news cycles are predictable — peak 6–9 AM and 5–8 PM WAT, quiet 11 PM–5 AM. Scaling to zero or one instance overnight saves approximately 25% of monthly instance hours.
• Response caching (Redis): the same breaking story excerpt often circulates across multiple newsrooms simultaneously. Caching responses for 10–15 minutes reduces GPU compute by an estimated 15–20% at medium-to-high traffic.
• Batch processing: client.py implements batch_generate() which sends multiple excerpts sequentially. At scale, a true server-side batching implementation would allow the model to process multiple inputs per model.generate() call, improving GPU utilisation.
• Reserved instances: a 1-year EC2 reserved instance saves 30–40% over on-demand for stable baseline load, complemented by spot capacity for peak absorption.

4.5 Comparison with RAG-Based Alternative

A retrieval-augmented generation (RAG) pipeline was evaluated as an alternative architecture. For headline generation, which requires learning style and compression patterns rather than retrieving dynamic facts, fine-tuning is both cheaper and more effective:

The fine-tuned model has learned that Nigerian political headlines use "Nigeria:" prefixes, party abbreviations like PDP and APC, active voice, and 7–10 word targets. A RAG system would retrieve example headlines and ask an LLM to imitate them — a slower, costlier, and less consistent approach for a task where the patterns are stable and learnable.

---

5. Monitoring & Observability Plan

5.1 Overview

The monitoring strategy described in this section is designed for production deployment. The current prototype uses Uvicorn's default request logging. Implementing this stack would be the next step before moving from prototype to production.

5.2 Monitoring Stack

Each layer answers a different question. CloudWatch answers "is the machine healthy?" Prometheus answers "is the API behaving correctly?" LangSmith answers "is the model producing good outputs and at what cost?"

5.3 Metrics to Track

5.4 Infrastructure Metrics (CloudWatch)

• GPU utilization — target 60–80% under normal load; alert if > 95% sustained for 5 minutes
• GPU memory — alert if > 90% of 16 GB (14.4 GB), signalling OOM risk
• CPU utilization — alert if > 80% (unexpected when GPU handles inference)
• EC2 instance status check — alert immediately on failure

5.5 Application Metrics (Prometheus)

A Prometheus /metrics endpoint would expose:

• headline_requests_total — counter by status code, used to compute error rate
• headline_request_duration_seconds — histogram for P50/P95/P99 latency
• headline_ttft_seconds — histogram specifically for time-to-first-token
• headline_active_requests — gauge for concurrent request count
• headline_tokens_total — counter for input and output tokens separately

Grafana dashboards visualise these as time-series panels with configurable alert thresholds.

5.6 LLM Telemetry (LangSmith)

LangSmith provides request-level tracing including the full prompt sent to the model, the raw output before post-processing, token counts, and latency breakdown. This enables:

• Diagnosing unexpected outputs — seeing exactly what prompt was constructed for a failing request
• Cost tracking — aggregating token usage across requests to compute real cost per headline
• Output quality monitoring — a random 1% sample of outputs is reviewed weekly against the 5–15 word target

5.7 Alerting Strategy

5.8 Logging Strategy

In production, every request would emit a structured JSON log entry containing: timestamp, request ID, excerpt character count, generated headline, TTFT in milliseconds, input and output token counts, device used, and any error details. Logs would be shipped to CloudWatch Logs with a 30-day retention policy.

A weekly automated report would aggregate: total request volume and error rate, latency distribution (P50/P95/P99 TTFT), headline length compliance rate, and estimated cost per request for the week.

---

6. Security Considerations

6.1 What Is Currently Implemented

The current prototype implements the following:

• Pydantic input validation: the GenerateRequest schema enforces correct Python types for all fields (excerpt: str, max_new_tokens: int, temperature: float, top_p: float, do_sample: bool) and applies safe default values. Malformed requests are rejected by FastAPI before reaching the model.
• Structured error handling: all three inference endpoints wrap model operations in try/except blocks and return HTTPException(status_code=500, detail=str(e)) rather than exposing raw Python stack traces to clients.
• Parameterised prompt construction: user-supplied excerpts are passed into build_prompt() as a function argument and injected into a fixed template string. This structurally separates instruction text from user content and reduces the attack surface for prompt injection compared to direct string concatenation.
• Environment variable secrets: the HuggingFace token is read from HF_TOKEN environment variable and is not hardcoded anywhere in the codebase.
• Docker health check: the healthcheck block in docker-compose.yml ensures the container only receives traffic once the model has loaded and GET / returns HTTP 200.

6.2 API Authentication

In production, all inference endpoints would require an X-API-Key header validated against a table of SHA-256 hashed keys stored in a database. Anonymous access would be permitted only on GET / (health check), which does not invoke the model. For internal CMS integrations, mutual TLS (mTLS) would be added to ensure requests originate from trusted client certificates.

6.3 Rate Limiting

Rate limiting would be applied at two levels:

• Gateway level: Nginx or AWS API Gateway enforces 60 requests per minute per IP address, returning HTTP 429 with a Retry-After header.
• Application level: a per-API-key limit (default: 1,000 requests/hour) enforced via a Redis-backed counter, allowing high-volume legitimate clients to be granted higher limits without opening the door to anonymous abuse.

6.4 Input Validation (Production Roadmap)

Beyond what Pydantic currently enforces, the production hardening plan includes:

• Reject excerpts exceeding 2,000 characters with HTTP 400 before tokenization
• Clamp max_new_tokens to a server-side ceiling of 200
• Strip control characters and null bytes from excerpts before prompt construction

6.5 PII Handling

News excerpts submitted to the API may contain names, phone numbers, or other identifying information present in news articles. In production: request bodies would not be logged by default. If debug logging is enabled, logs would be stored in a CloudWatch log group with restricted IAM access and a 7-day retention policy. Prompt content would never be sent to third-party analytics services. The LangSmith integration would be configured with data redaction enabled for PII patterns before traces are transmitted.

6.6 Access Control

Access to the inference endpoint is restricted to holders of valid API keys. Access to CloudWatch logs and Prometheus metrics is restricted to the engineering team via IAM roles — no public access. LangSmith project access is restricted to the model owner. The Docker host accepts management connections only via AWS Systems Manager Session Manager; no IAM users have direct SSH key access to the production instance.

6.7 Container Hardening (Production Roadmap)

• Dedicated non-root app user (UID 1000) — application process should not run as root inside the container
• Read-only root filesystem with writable mounts only for /app/model_cache and /tmp
• Container placed in a private VPC subnet; inbound access only via AWS ALB on port 443
• Trivy image scanning in CI pipeline; builds with critical CVEs blocked from deployment
• AWS Secrets Manager for HuggingFace token and API keys in production

6.8 Transport Security

All production traffic would be routed through an AWS Application Load Balancer with an ACM-managed TLS certificate. HTTP connections would be redirected to HTTPS with a 301 response at the ALB level. The application does not handle TLS directly, so certificate rotation requires no code changes. TLS 1.2 is the minimum permitted version; TLS 1.0 and 1.1 are disabled on the ALB listener.

6.9 Output Safety

Generated headlines would pass through a post-processing step before being returned:

• Length check: headlines outside 3–20 words trigger a fallback re-run with do_sample=False
• PII regex scan stripping Nigerian phone number and NIN patterns from generated text
• Editorial blocklist scan; matches withheld from response and logged for model quality review

---

7. Testing & Validation

7.1 Test Suite

The test suite (client/test_requests.py) covers 8 categories with 20+ real Nigerian news examples, runnable against both local and Docker deployments via --url http://localhost:8000:

• Single generation — greedy decoding (do_sample=False) and stochastic sampling (do_sample=True)
• Streaming generation — verifies token-by-token output via /generate_stream
• Batch processing — 3 to 10 excerpts using batch_generate()
• Domain categories — politics, economy, sports, health, crime, business; each with at least two real Nigerian news excerpts
• Latency benchmarking — 20 prompts at batch size 4, measuring TTFT, TPOT, and ITL
• Edge cases — empty string input, excerpt exceeding 512 tokens, special characters, numerals, mixed-language input
• Error handling — malformed JSON body, missing required fields, incorrect field types
• Real-world workflow — 5-article newsroom simulation with end-to-end timing and headline quality check

7.2 Latency Benchmarks

Measured via POST /ttft_itl_batched (20 prompts, batch size 4, T4 GPU, 4-bit quantization):

P99 TTFT of 199ms remains within the 200ms production target. On CPU, TTFT rises to 800–1,200ms — acceptable for batch workflows but unsuitable for interactive use.

---

8. Conclusion

This project delivers a working deployment prototype for Nigerian news headline generation, taking a QLoRA fine-tuned Qwen 2.5 0.5B model with 17–41% ROUGE improvements over baseline and making it accessible via a production-structured API. The key outcomes are:

• Sub-200ms P99 TTFT on GPU through 4-bit NF4 quantization and direct HuggingFace Transformers inference
• Cost as low as $0.0013 per headline at 10,000 requests/day on spot GPU instances
• Portable, reproducible Docker deployment with automatic CPU/GPU fallback, model cache persistence, and health-based restart
• A three-layer production monitoring strategy (CloudWatch, Prometheus, LangSmith) with specific metrics, alert thresholds, and notification routing
• Basic security implemented in the prototype (Pydantic validation, structured error handling, parameterised prompt construction, environment variable secrets), with a complete production security roadmap covering authentication, rate limiting, container hardening, PII handling, output safety, and HTTPS

The gap between the current prototype and a fully production-hardened system is well-defined: implementing the security roadmap in Section 6 and the monitoring stack in Section 5 are the two primary next steps before handling real user traffic.

Model: huggingface.co/Blaqadonis/Qwen2.5-0.5B-Nigerian-News-Headlines
Code: github.com/Blaqadonis/nigerian-news-headlines-qlora

---

References

1. Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv
   .14314.
2. Hu, E. J. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv
   .09685.
3. Bai, J. et al. (2023). Qwen Technical Report. arXiv
   .16609.
4. Lin, C. Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out, 74–81.
5. Okite97. (2024). Nigerian News Dataset. HuggingFace Datasets. https://huggingface.co/datasets/okite97/news-data