DebugLLM: Fine-Tuning Phi-3 Mini with LoRA for Automated Python Bug Fixing

---

TL;DR

This project fine-tunes Phi-3 Mini (3B parameters) using LoRA (Low-Rank Adaptation) on the MBPP dataset to build a lightweight, specialized model that automatically detects and repairs common Python code bugs — including syntax errors, indentation mistakes, incorrect loops, and variable misuse — all trainable on a single NVIDIA T4 GPU in under one hour.

---

1. Introduction & Motivation

Debugging is one of the most time-consuming tasks in a developer's workflow — especially for beginners. While large frontier models like GPT-4 can handle code repair, they require expensive API calls and are not tailored for lightweight, offline, or embedded use cases.

This project explores a targeted question:

Can a compact open-weights LLM, fine-tuned with parameter-efficient methods, reliably fix common Python bugs without the overhead of a massive model?

The answer, as demonstrated in this work, is yes — with meaningful caveats.

Key Contributions

• A reproducible LoRA fine-tuning pipeline for code bug repair on consumer-grade hardware
• A structured bug-fix dataset derived from the MBPP benchmark
• Qualitative evaluation showing improved performance over the base model on syntax and structural errors
• Discussion of failure modes and practical limitations

---

2. Background

2.1 The Problem: Automated Bug Fixing

Automated Program Repair (APR) has been studied for decades, but LLM-based approaches have opened new possibilities. Rather than symbolic rule-matching, neural models can generalize across syntactic patterns and produce human-readable corrections.

2.2 Why Phi-3 Mini?

Microsoft's Phi-3 Mini (~3.8B parameters) is a strong choice for this task because:

• It is trained with high-quality instruction data, making it receptive to task-specific fine-tuning
• Its 8K token context window comfortably accommodates Python function-level code
• It runs efficiently on a single GPU (16GB VRAM), making experimentation accessible

2.3 Why LoRA?

Low-Rank Adaptation (LoRA) enables fine-tuning of large models by injecting small trainable rank-decomposition matrices into selected layers — while freezing the original weights. This means:

• ~99% fewer trainable parameters compared to full fine-tuning
• Significantly reduced GPU memory usage
• Faster iteration and easy sharing (only adapters need to be saved/distributed)
• Reduced risk of catastrophic forgetting of base model knowledge

---

3. Dataset

3.1 Source

The dataset is derived from the MBPP (Mostly Basic Python Problems) benchmark by Google Research, available on HuggingFace:

🔗 google-research-datasets/mbpp

MBPP contains ~1,000 beginner-level Python programming problems with reference solutions, making it well-suited for constructing supervised bug-fix pairs.

3.2 Dataset Construction

Each MBPP sample was transformed into an instruction-style bug-fix pair:

1. The reference solution was taken as the "correct" version
2. Synthetic bugs were introduced programmatically (missing colons, wrong indentation, incorrect variable references, etc.)
3. The buggy version became the input, and the correct version became the target output

This approach provides clean, verifiable ground truth for every training sample.

3.3 Dataset Split

3.4 Bug Categories Covered

3.5 Prompt Format

Each training example follows a structured instruction template:

### Instruction:
Fix the bug in the following Python code.

### Input:
for i in range(5)
    print(i)

### Response:
for i in range(5):
    print(i)

This format aligns with Phi-3's instruction-tuning style and ensures consistent model behavior at inference time.

3.6 Preprocessing Steps

1. Loaded MBPP dataset via HuggingFace datasets library
2. Programmatically introduced bugs into reference solutions
3. Applied instruction prompt template to each example
4. Tokenized using Phi-3 tokenizer (with padding/truncation to 256 tokens)
5. Shuffled training set and split into train/validation

---

4. Methodology

4.1 Base Model

4.2 LoRA Configuration

from peft import LoraConfig

lora_config = LoraConfig(
    r=16,                          # Rank of update matrices
    lora_alpha=32,                 # Scaling factor
    lora_dropout=0.05,             # Dropout for regularization
    target_modules=["q_proj", "v_proj"],  # Apply LoRA to attention layers
    bias="none",
    task_type="CAUSAL_LM"
)

Design Rationale:

• r=16: A moderate rank that balances expressiveness and memory efficiency
• alpha=32 (= 2×r): Standard scaling that provides stable gradient updates
• target_modules=["q_proj", "v_proj"]: Attention projection layers capture the most task-relevant patterns; targeting both query and value matrices provides effective adaptation
• lora_dropout=0.05: Light regularization to prevent overfitting on the small dataset

4.3 Training Configuration

4.4 Training Infrastructure

4.5 Training Pipeline

1. Load Base Model (Phi-3 Mini, FP16)
        ↓
2. Apply LoRA Adapters (PEFT)
        ↓
3. Load & Preprocess Dataset (MBPP → Bug-Fix pairs)
        ↓
4. Train via HuggingFace Trainer (3 epochs)
        ↓
5. Monitor Loss via W&B Dashboard
        ↓
6. Evaluate on Held-Out Test Set
        ↓
7. Save & Publish LoRA Adapters to HuggingFace Hub

5. Results

5.1 Training Dynamics

The training loss decreased steadily across all three epochs, indicating consistent learning without signs of divergence or severe overfitting on the training set. Validation loss tracked closely with training loss throughout, suggesting reasonable generalization given the dataset size.

📊 Training metrics and loss curves are tracked live on the W&B Dashboard.

5.2 Qualitative Evaluation: Base vs. Fine-Tuned Model

Example 1 — Missing Colon (Syntax Error)

Input (Buggy Code):

for i in range(5)
    print(i)

Base Model Output:

# Often reproduced the bug or added unrelated explanation text
for i in range(5)
    print(i)

Fine-Tuned Model Output:

for i in range(5):
    print(i)

---

Example 2 — Indentation Error

Input (Buggy Code):

def greet(name):
print("Hello, " + name)

Fine-Tuned Model Output:

def greet(name):
    print("Hello, " + name)

---

Example 3 — Variable Name Error

Input (Buggy Code):

total = 0
for num in range(10):
    total += number
print(total)

Fine-Tuned Model Output:

total = 0
for num in range(10):
    total += num
print(total)

5.3 Performance Summary

Note: Quantitative metrics (e.g., exact match accuracy, CodeBLEU) were not computed due to dataset and tooling constraints. This is acknowledged as a limitation — see Section 7.

---

6. Implementation Details

6.1 Repository Structure

Phi3-debugLLM-LoRA/
├── data/
│   └── prepare_dataset.py        # MBPP → bug-fix pair conversion
├── training/
│   └── train.py                  # LoRA fine-tuning script
├── evaluation/
│   └── evaluate.py               # Inference and qualitative testing
├── notebooks/
│   └── demo.ipynb                # End-to-end walkthrough
└── README.md

6.2 Inference Example

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "microsoft/Phi-3-mini-4k-instruct",
    torch_dtype=torch.float16,
    device_map="auto"
)

model = PeftModel.from_pretrained(base_model, "Sud1212/phi3-debug-llm-lora")
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")

prompt = """### Instruction:
Fix the bug in the following Python code.

### Input:
for i in range(5)
    print(i)

### Response:"""

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

6.3 Reproducibility

All training hyperparameters, dataset preprocessing steps, and model configurations are documented in the GitHub repository. The LoRA adapter weights are publicly available on HuggingFace Hub for direct use or further fine-tuning.

---

7. Discussion

7.1 What Worked Well

LoRA Efficiency: The parameter-efficient fine-tuning approach worked exactly as intended — the model adapted to the bug-fixing task in ~1 hour on a T4 GPU, with only a small fraction of parameters updated (~0.5% of total model weights).

Instruction Template Consistency: Adopting a structured prompt format (Instruction / Input / Response) significantly improved the model's ability to follow the task correctly at inference time, compared to unstructured prompting.

Syntax-Level Bug Fixing: The model demonstrated reliable performance on syntactically well-defined errors — missing colons, indentation mismatches, and simple variable name errors — the categories most consistently represented in the training data.

7.2 Challenges Encountered

Dataset Scale: With ~900 training samples, the model's generalization to unseen bug patterns is inherently limited. Larger and more diverse datasets would improve robustness.

Prompt Sensitivity: Minor changes to the instruction phrasing at inference time occasionally produced inconsistent outputs. This is a known challenge with smaller instruction-tuned models.

Evaluation Gap: Without automated quantitative metrics (exact match, CodeBLEU, pass@k), it's difficult to precisely quantify improvement. Evaluation was primarily qualitative — a recognized limitation of this project.

Complex Logical Bugs: Multi-line logic errors (e.g., incorrect algorithm design, wrong conditional logic) remain largely out of reach for a model trained on this dataset. These require semantic understanding beyond surface-level pattern matching.

---

8. Limitations

---

9. Future Directions

Several extensions of this work could meaningfully improve both capability and reliability:

1. Larger and More Diverse Datasets: Incorporating datasets like BugsInPy, CodeNet, or synthetically augmented MBPP variants would improve generalization across bug categories.

2. Quantitative Evaluation Pipeline: Implementing automated metrics — CodeBLEU, exact match accuracy, and pass@k using unit tests — would provide rigorous, reproducible benchmarks.

3. Broader Bug Type Coverage: Extending training to cover algorithmic logic errors, API misuse, and type errors would push the model toward more practical utility.

4. Multi-Language Support: Adapting the pipeline for JavaScript, Java, or C++ bugs with language-specific datasets.

5. Integration as a VS Code / IDE Extension: Packaging the model as a lightweight local IDE plugin for real-time bug suggestions without API dependency.

6. RLHF / DPO Alignment: Using human preference feedback or Direct Preference Optimization to further align outputs toward syntactically and semantically correct fixes.

7. Quantization for Edge Deployment: Applying INT4/INT8 quantization (e.g., via GGUF/llama.cpp) to enable the model to run on CPU-only environments.

---

10. Conclusion

This project demonstrates that compact open-weights LLMs can be efficiently adapted for specialized developer tasks using parameter-efficient fine-tuning — without large compute budgets or proprietary APIs.

The fine-tuned Phi-3 Mini + LoRA model:

• ✅ Learns common Python bug-repair patterns from a modest dataset
• ✅ Improves debugging performance over the base model on syntax and structural errors
• ✅ Trains end-to-end in under one hour on a single T4 GPU
• ✅ Is fully reproducible and publicly available

While limitations around dataset scale and evaluation depth remain, this work establishes a clean, reusable pipeline for task-specific LLM adaptation — one that can be extended with richer data, broader bug coverage, and quantitative benchmarking.

The core insight: You don't need a 70B model to fix a missing colon. With the right training setup, a 3B model can learn to do it reliably.

---

11. Resources

---

12. Technologies Used

---

13. Author & Contact

Sudarshan Maddi
Student, Woxsen University

🔗 GitHub | 🤗 HuggingFace | 📊 W&B

For questions, issues, or collaboration inquiries, please open an issue on the GitHub Repository.

---

Published on ReadyTensor · Educational Content → Academic Solution Showcase