GitHub Repository: https://github.com/samrat-kar/llm-deploy-monitor
HuggingFace Model: https://huggingface.co/samrat-kar/qwen2.5-1.5b-sql-qlora
W&B Report: QLoRA Fine-Tuning of Qwen2.5-1.5B — Training Curves & Metrics
Module 1 Publication: From Prose to SQL: QLoRA Fine-Tuning of Qwen2.5-1.5B

Tags: LLM Deployment RunPod Text-to-SQL FastAPI QLoRA Qwen2.5 Monitoring LLMED Production Engineering

---

Introduction

In Module 1 of this program, I fine-tuned Qwen2.5-1.5B-Instruct on ~78,000 text-to-SQL examples using QLoRA, achieving a 75.4% exact match rate on a held-out validation set — up from 0% with the base model. The resulting model, samrat-kar/qwen2.5-1.5b-sql-qlora, is published on HuggingFace.

This publication covers the deployment plan for that model: taking it from a HuggingFace artifact to a live HTTP service, including the infrastructure choices, cost breakdown, monitoring strategy, and security considerations.

---

1. Use Case Definition

Problem Statement

Writing SQL queries is a recurring bottleneck for data analysts and backend developers — especially when working with unfamiliar schemas or performing exploratory analysis. The goal is a lightweight internal tool that accepts a natural-language question and a set of CREATE TABLE statements, and returns a single executable SQL query. No chat interface, no multi-turn conversation — just a clean, fast query generation endpoint.

Target Users

The service is internal — not consumer-facing. Usage is limited to a small team of 5–15 people.

Input / Output Examples

Example 1 — Aggregation

Question: How many singers do we have?

Schema:

CREATE TABLE singer (Singer_ID int, Name text, Country text, Song_Name text)

Output: SELECT count(*) FROM singer

---

Example 2 — Filtering

Question: Find all employees in the Engineering department with a salary above 80000.

Schema:

CREATE TABLE employee (Employee_ID int, Name text, Department text, Salary real)

Output: SELECT * FROM employee WHERE Department = 'Engineering' AND Salary > 80000

---

Example 3 — JOIN

Question: List each student's name and the dorm they live in.

Schema:

CREATE TABLE student (StuID int, Fname text, LName text)
CREATE TABLE lives_in (stuid int, dormid int)
CREATE TABLE dorm (dormid int, dorm_name text)

Output: SELECT T1.Fname, T1.LName, T3.dorm_name FROM student AS T1 JOIN lives_in AS T2 ON T1.StuID = T2.stuid JOIN dorm AS T3 ON T2.dormid = T3.dormid

Success Criteria

Traffic Expectations

This is an internal developer tool, so traffic is modest and bursty — concentrated during working hours, quiet at nights and weekends.

These figures inform the deployment strategy: a persistent pod that runs during business hours and is stopped overnight is more cost-effective than always-on infrastructure.

---

2. Model Selection & Configuration

Model Card

Why This Model

The model was fine-tuned specifically on the b-mc2/sql-create-context dataset (~78K examples) for this exact task. It outperforms the base model dramatically on SQL generation:

The 1.5B parameter size is a deliberate trade-off: it fits comfortably in a single consumer GPU with 4-bit quantization (~3 GB VRAM), inference is fast (300–800 ms on an RTX 3090), and the focused fine-tuning compensates for the smaller capacity. A larger model like Llama 3.2 7B or Mistral 7B would improve accuracy on complex multi-table joins but would cost 4–5× more per hour for this use case.

Trade-offs Considered

Size vs. quality: A 7B model would score higher on complex joins and nested subqueries. For our use case — mostly single-table aggregations and two-table joins — the 1.5B fine-tuned model delivers comparable quality at a fraction of the cost.

Quantization impact: 4-bit NF4 quantization reduces the model from ~6 GB (bfloat16) to ~1.5 GB on disk and ~3 GB VRAM. Empirically, there is no measurable accuracy drop on text-to-SQL tasks at this compression level, since SQL generation does not require high numerical precision.

API vs. self-hosted: Using the OpenAI API or HuggingFace Inference API would eliminate infrastructure management, but would cost 10–30× more per request for this usage volume and would prevent fine-tuned model serving without additional setup.

---

3. Deployment Strategy

Platform: RunPod GPU Pod

I chose a RunPod GPU Pod over serverless or managed API options for the following reasons:

1. Model already on HuggingFace — the pod loads weights directly at startup with no custom container build required.
2. Familiar workflow — SSH access, persistent disk, and a standard PyTorch environment mean the deployment mirrors local development.
3. Cost-predictable — hourly billing while running; stopping the pod stops the cost. For a business-hours tool, this is more economical than always-on managed endpoints.
4. Low-latency warm starts — with the model cached on the volume disk after the first load, subsequent pod starts skip the download and reach ready state in ~30 seconds.

Alternatives considered:

Infrastructure Details

Serving Architecture

User / Notebook
     │
     │  HTTPS (RunPod proxy)
     ▼
┌─────────────────────────────┐
│       RunPod Pod            │
│                             │
│  FastAPI server (port 8000) │
│  ├── GET  /health           │
│  └── POST /generate         │
│         │                   │
│         ▼                   │
│  Transformers pipeline      │
│  (model loaded in GPU VRAM) │
└─────────────────────────────┘

• Endpoint type: Real-time (synchronous HTTP)
• Scaling strategy: Single-pod, single-GPU. For the projected load (≤80 req/hr peak), a single pod with request queuing in FastAPI is sufficient. If load grows beyond ~200 concurrent requests, adding a second pod behind a simple load balancer (or migrating to RunPod Serverless) would be the next step.
• Startup sequence: Pod boots → startup.sh installs dependencies → FastAPI starts → model loads from HF cache → /health returns ok.

Operating Schedule

To minimize cost, the pod runs only during business hours:

• Start: Monday–Friday, 8
  AM
• Stop: Monday–Friday, 6
  PM
• Weekend: Pod stopped (minimal expected traffic)

This gives ~50 GPU-hours/week at 20/week, or ~$85/month.

---

4. Cost Analysis

Monthly Cost Breakdown

At 100–200 requests/day × 22 weekdays = ~2,200–4,400 requests/month, the cost per 1,000 requests is approximately 40.

Cost Optimization Strategies

1. Response caching

Text-to-SQL queries from a dev team are highly repetitive — the same question against the same schema will appear many times. A simple in-process LRU cache (keyed on sha256(question + context)) can eliminate 20–40% of GPU compute:

from functools import lru_cache
import hashlib

@lru_cache(maxsize=512)
def cached_generate(question: str, context: str) -> str:
    ...

For a persistent cache that survives pod restarts, a Redis instance on the same pod (or RunPod volume) stores the (hash → sql) mapping.

2. Pod scheduling (start/stop automation)

RunPod's API supports programmatic pod start/stop. A small cron job or GitHub Action can:

• Start the pod at 8
  AM on weekdays
• Stop it at 6
  PM

This eliminates ~14 idle GPU-hours/day that would otherwise accrue on an always-on pod.

3. 4-bit quantization (already applied)

The model is already loaded in 4-bit NF4. This reduces VRAM usage from ~6 GB to ~3 GB, making it possible to use a cheaper GPU (e.g., RTX 3080 10 GB at $0.26/hr) and reducing the cost by ~33%.

4. Request batching (future)

The current server processes requests one at a time. If peak load grows, enabling dynamic batching (grouping concurrent requests into a single forward pass) can increase throughput 3–5× on the same hardware without additional cost. transformers supports this natively via batch_size in the pipeline API.

5. Spot instance pricing

RunPod Community Cloud (spot) is already 30–40% cheaper than Secure Cloud. Spot pods can be interrupted, but for a developer tool with re-tryable requests this is acceptable.

---

5. Monitoring & Observability Plan

Metrics to Track

Monitoring Stack

Application-level logging (already in server.py)

The server logs every request:

2026-04-10 09:12:43 INFO Generated 9 tokens in 312ms

These logs are written to stdout (visible in RunPod terminal) and can be tailed or piped to a file on the volume.

LangFuse integration plan

To add LangFuse tracing to the server, wrap the generation call:

from langfuse import Langfuse

langfuse = Langfuse()  # reads LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY from env

trace = langfuse.trace(name="sql-generate")
span = trace.span(name="model-inference", input={"question": req.question, "context": req.context})
# ... run inference ...
span.end(output={"sql": sql}, metadata={"latency_ms": latency_ms, "tokens": tokens_generated})

This gives a time-series view of:

• Which questions are being asked most
• Which inputs produce long or unexpected outputs
• Latency trends over time

Alerting Strategy

Alert conditions:

Notification channels: Email (primary) and a Slack #alerts-sql-api channel via webhook.

Runbook for common issues:

---

6. Security Considerations

API Authentication

RunPod's proxy provides a pod-specific URL (https://<pod-id>-8000.proxy.runpod.net) that is not guessable. As an additional layer, the server validates a static API key passed as a request header:

from fastapi.security.api_key import APIKeyHeader
from fastapi import Security, HTTPException

API_KEY_HEADER = APIKeyHeader(name="X-API-Key")

def verify_api_key(api_key: str = Security(API_KEY_HEADER)):
    if api_key != os.environ["SQL_API_KEY"]:
        raise HTTPException(status_code=403, detail="Invalid API key")

The key is set as an environment variable on the pod and never committed to the repository.

Rate Limiting

Since the service is internal and low-traffic, a simple per-IP rate limit is sufficient. Using slowapi:

from slowapi import Limiter
from slowapi.util import get_remote_address

limiter = Limiter(key_func=get_remote_address)

@app.post("/generate")
@limiter.limit("30/minute")
def generate(request: Request, req: GenerateRequest):
    ...

Limit: 30 requests/minute per IP — generous enough for normal use, restrictive enough to prevent runaway loops.

Input Validation

Prompt injection is the primary attack surface: a malicious user could craft a question or context field to override the system prompt.

Mitigations:

1. Pydantic field length limits — question capped at 500 characters, context capped at 2,000 characters. Inputs exceeding these limits return a 422 error before the model is called.
2. Output validation — the server checks that the generated output starts with a SQL keyword (SELECT, INSERT, UPDATE, DELETE, WITH). Outputs that don't match are flagged and logged, and a 422 is returned.
3. System prompt is server-side — the system prompt is not user-supplied; it is set in model_config.yaml and cannot be overridden via the API.

PII Handling

SQL queries typically don't contain PII, but the question field might (e.g., "Find all records for customer John Smith"). Policy:

• Logs retain inputs for 7 days (for debugging), then are deleted from the volume.
• Logs are stored only on the RunPod volume — never forwarded to external services.
• LangFuse traces, if enabled, are set to mask the raw prompt after 30 days.
• Users are instructed not to paste real production data into the question field.

Access Control

---

Summary

This deployment plan covers the full lifecycle of taking samrat-kar/qwen2.5-1.5b-sql-qlora from a HuggingFace artifact to a production-ready internal service:

• Use case: Text-to-SQL for data analysts and developers, ~100–200 requests/day
• Model: 1.5B fine-tuned model with 75.4% exact match, loaded in 4-bit quantization
• Platform: RunPod GPU Pod (RTX 3090), running during business hours
• Cost: ~20–40 per 1,000 requests, reducible by 30–40% with caching and scheduling
• Monitoring: FastAPI structured logs + LangFuse tracing + RunPod dashboard metrics
• Security: API key auth, per-IP rate limiting, input length validation, output keyword validation