AIT (AI Terminal): Deploying a Fine-Tuned LLM for Shell Command Generation

Module 2 Capstone Project — LLM Engineering & Deployment Certification

---

Executive Summary

AIT (AI Terminal) is a CLI tool that translates natural language descriptions into shell commands. This publication documents the complete deployment strategy for a fine-tuned Qwen3-0.6B model, including infrastructure choices, cost analysis, monitoring plan, and security considerations.

Key Highlights:

• Fine-tuned Qwen3-0.6B model specialized for terminal command generation
• Deployed on HuggingFace Dedicated Inference Endpoint
• LiteLLM proxy on HuggingFace Spaces for API management
• Go CLI client for cross-platform distribution
• Total estimated cost: $0.50/hour (scale-to-zero enabled, Nvidia T4 GPU)

---

1. Use Case Definition

Problem Statement

Developers frequently need to recall complex shell commands for tasks like file manipulation, system administration, and data processing. Searching documentation or Stack Overflow interrupts workflow and reduces productivity.

AIT solves this by:

• Converting natural language descriptions to executable shell commands
• Supporting multiple platforms (Linux, macOS, Windows)
• Providing instant, contextual command generation

Target Users

Input/Output Examples

Success Criteria

Note: Higher p95 latency is expected due to HuggingFace Dedicated Endpoint cold starts and LiteLLM response caching. Once warmed up, median latency is 1.29s with many cached responses returning in ~0.2s.

Traffic Expectations

The deployment uses scale-to-zero to handle variable traffic efficiently.

---

2. Model Selection & Configuration

Model Details

Why Qwen3-0.6B?

1. Size Efficiency: 600M parameters provides excellent quality-to-cost ratio for command generation
2. Fine-Tuned Specialization: Trained specifically on terminal command datasets from Module 1
3. Fast Inference: Small model size enables sub-second generation on GPU
4. Low Cost: Runs efficiently on minimal hardware

Quantization Decision

No quantization applied because:

• Model is already small (600M params)
• FP16 fits comfortably in 16GB VRAM (Nvidia T4)
• Quantization would minimally reduce costs but could impact command accuracy

---

3. Deployment Strategy

Architecture Overview

┌─────────────────┐     ┌─────────────────────────────────────────────┐
│   AIT CLI       │────▶│  LiteLLM Proxy (HuggingFace Spaces)         │
│   (Go binary)   │     │  ┌─────────────┐    ┌────────────────────┐  │
└─────────────────┘     │  │  LiteLLM    │───▶│  HF Endpoint Proxy │  │
                        │  │  (port 7860)│    │  (port 8000)       │  │
                        │  └─────────────┘    └─────────┬──────────┘  │
                        └───────────────────────────────┼─────────────┘
                                                        │
                        ┌───────────────────────────────▼─────────────┐
                        │  HuggingFace Dedicated Inference Endpoint   │
                        │  (Qwen3-0.6B Terminal Instruct)             │
                        └─────────────────────────────────────────────┘
                                                        │
                        ┌───────────────────────────────▼─────────────┐
                        │  Supabase PostgreSQL                        │
                        │  (Virtual keys, spend tracking)             │
                        └─────────────────────────────────────────────┘

Platform Selection

Why HuggingFace Dedicated Endpoints?

HuggingFace Dedicated Endpoints was selected because:

• Scale-to-zero: No charges when idle
• Managed TGI: Optimized inference server included
• Simple deployment: One-click from model repository
• Pay-per-hour: Only charged for actual compute time

Infrastructure Details

Why GPU (Nvidia T4)?

For production use with low latency:

• GPU inference latency: 0.2-0.5 seconds (excellent UX)
• CPU inference latency: 1-3 seconds (slower)
• GPU cost: $0.50/hour (with scale-to-zero)
• 16GB VRAM: Ample headroom for the 600M model

The faster response time justifies the GPU cost for a better user experience.

---

4. Cost Analysis

Estimated Monthly Costs

Cost Per Request

Cost Optimization Strategies

1. Scale-to-Zero

• Endpoint automatically scales down after 15 minutes of inactivity
• No charges during idle periods
• Savings: 50-80% compared to always-on deployment

2. Response Caching (LiteLLM)

litellm_settings:
  cache: true
  cache_params:
    type: "local"
    ttl: 600  # 10 minutes

• Identical prompts return cached responses
• Savings: 10-30% reduction in endpoint calls

---

5. Monitoring & Observability Plan

Metrics to Track

Monitoring Stack

LiteLLM Observability Features

general_settings:
  enable_user_auth: true
  max_budget: 10.0
  budget_duration: "30d"

Tracked automatically:

• Request count per virtual key
• Token usage per request
• Latency distribution
• Error rates by error type
• Spend per key/user

6. Security Considerations

API Authentication

Virtual Key Security

# Users receive scoped virtual keys
sk-xxxxxxxxxxxxxxxxxxxx

# Keys are:
# - Revocable instantly
# - Budget-limited ($5/month default)
# - Rate-limited (60 RPM)
# - Model-scoped (only "default" model)

Rate Limiting

Input Validation

PII Handling

Access Control

Secrets Management

---

7. Deployment Instructions

Prerequisites

• HuggingFace account with write access token
• Supabase account (free tier)
• Go 1.21+ (for building CLI)

Step 1: Deploy Model Endpoint

1. Go to HuggingFace Inference Endpoints
2. Create new endpoint from Eng-Elias/Qwen3-0.6B-terminal-instruct
3. Select CPU instance, enable scale-to-zero
4. Wait for status: "Running"

Step 2: Deploy LiteLLM Proxy

1. Create HuggingFace Space (Docker SDK)
2. Set secrets: HF_TOKEN, LITELLM_MASTER_KEY, DATABASE_URL, HF_ENDPOINT_URL
3. Upload files from deploy/litellm/hf-spaces/
4. Wait for build to complete

Step 3: Create Virtual Key

curl -X POST https://your-space.hf.space/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"models": ["default"], "max_budget": 5.0}'

Step 4: Install and Configure CLI

# Install
go install github.com/Eng-Elias/ait@latest

# Configure
ait setup
# Enter: https://your-space.hf.space/v1/chat/completions
# Enter: sk-your-virtual-key
# Enter: default

Step 5: Test

ait "list all files larger than 100MB"
# Output: find . -size +100M

---

8. Conclusion

The architecture balances simplicity with production requirements, using managed services (HuggingFace, Supabase) to minimize operational overhead while maintaining full control over the deployment.

---

Links

• GitHub Repository: github.com/Eng-Elias/ait
• Fine-Tuned Model: huggingface.co/Eng-Elias/Qwen3-0.6B-terminal-instruct
• LiteLLM Proxy: eng-elias-litellm.hf.space
• Module 1 Project: Fine-Tuning Qwen3 for Terminal Commands

---