How to Fine Tune LLM for Production: A Field Guide
I spent three months in 2025 fine-tuning a 70B parameter model for a healthcare triage system. The first two months were a disaster. We had a model that could recite medical textbooks perfectly — and couldn't tell a migraine from a hangover in practice.
This is the guide I wish I'd read before burning $47,000 on GPU time.
What this is: A practical walkthrough of how to fine tune LLM for production — covering the decisions you'll actually face, the mistakes you'll make, and the trade-offs nobody writes about. This isn't a theory piece. It's what I've learned building production AI systems at SIVARO since 2023.
The Fine-Tuning Trap (Most People Start Wrong)
Here's the lie that gets repeated: fine-tuning fixes everything. Bad output format? Fine-tune. Poor domain knowledge? Fine-tune. Hallucinations? Fine-tune harder.
That's wrong. LLM Fine-Tuning Explained gets it right — fine-tuning is about behavior modification, not knowledge injection. The model already knows your domain from pre-training. What it doesn't know is how you want it to behave.
I tested this with a legal document extraction pipeline. We tried fine-tuning on 10,000 legal contracts. The model got worse. Why? Because the contracts it was trained on were public case law — not the specific formatting our client's law firm used. The base model already knew contract law. It didn't know how to output XML with specific tags.
Real insight: Before you fine-tune, prove you can't solve the problem with prompting + RAG. In my experience, 60% of "fine-tuning needs" are actually prompt engineering problems.
When You Actually Need to Fine-Tune
You need fine-tuning when:
- Output format must be exactly right — JSON schemas, XML structures, specific tone
- You need to suppress certain behaviors — the model keeps apologizing, hedging, or refusing valid requests
- You want to teach a new capability — tool use, chain-of-thought reasoning in your domain
The mistake? Fine-tuning for knowledge. Don't. Google Cloud's guide on fine-tuning LLMs makes this clear: fine-tuning changes the model's weights slightly. It's not a database. Use retrieval for facts, fine-tuning for behavior.
The Three Questions Before Any Fine-Tune
Question 1: What's Your Baseline?
Don't touch a training script until you've run 100 test examples through the base model with your best prompt. Record every output. You need to know:
- Where does it fail? (Format, reasoning, refusal, hallucination?)
- What's the failure rate? (If it's <20%, you might not need fine-tuning)
- Can you fix it with a better system prompt?
At a fintech startup in 2024, I watched a team spend 3 weeks curating a fine-tuning dataset. The base model worked fine with two additional examples in the prompt. They wasted $12,000. Don't be them.
Question 2: Can You Afford the Data?
Fine-tuning data is expensive to produce. Not because of labeling costs — because of quality filtering.
Here's what nobody tells you: you'll discard 40-60% of your initial dataset.
We built a dataset of 50,000 customer support conversations for a logistics company. After cleaning — removing PII, fixing formatting, removing contradictory responses, deleting examples where the human agent was wrong — we had 22,000 usable examples.
That's normal. Budget for it.
Question 3: Which Model Are You Starting With?
This matters more than any hyperparameter choice. OpenAI's model optimization guide shows you can fine-tune GPT-4o mini for about $25 per million training tokens. Fine-tuning a 70B open model? You're looking at $300-500 per training run on A100s.
Pick the smallest model that can express the capability. Then fine-tune. If it doesn't work, move up. Going big first burns money and time.
Building Your Dataset (This Determines Everything)
Your dataset is your product. Bad data = bad model. Period.
What Good Data Looks Like
Each example should be:
- Demonstrative — shows the exact behavior you want
- Consistent — all examples follow the same rules
- Clean — no hallucinations, no contradictions, no formatting errors
I use a 80/10/10 split: 80% standard examples, 10% edge cases, 10% adversarial examples (where the model might be tempted to do the wrong thing).
Format Matters More Than You Think
For chat models, your format must match the model's training format. For Llama 3, that's:
python
# Training example format for Llama 3
example = {
"messages": [
{"role": "system", "content": "You are a medical triage assistant. Always ask about emergency symptoms first."},
{"role": "user", "content": "My chest hurts when I breathe deeply."},
{"role": "assistant", "content": "I need to ask some urgent questions first. Are you having any: 1) Difficulty breathing 2) Pain radiating to your arm or jaw 3) Feeling faint? [triage_level: EMERGENCY]"}
]
}
One client used a different separator in their format. The model learned to produce gibberish after 500 tokens. Wasted 4 training runs before we caught it.
Data Augmentation That Actually Works
Don't just duplicate examples. Generate variations:
python
# Weak augmentation that hurts
augmented = [
"The patient has chest pain.",
"Patient reports chest pain.",
"Chest pain is the patient's complaint."
]
# Strong augmentation that helps
augmented = [
{"role": "user", "content": "My chest hurts."},
{"role": "user", "content": "Can you help me? I have pain in my chest when I exert myself."},
{"role": "user", "content": "I'm 45, male, and I feel pressure in my chest. Should I worry?"}
]
The first group teaches nothing. The second teaches the model to handle variation in how people describe symptoms.
The Training Loop (What Actually Matters)
Hyperparameters Are Overrated
Everyone obsesses over learning rate and LoRA rank. Here's the truth: default values work for 80% of cases. What actually matters:
- Batch size — small enough to fit in memory, large enough for stable gradients
- Learning rate schedule — cosine decay with warmup is safe
- Number of epochs — this is where you'll screw up
On overfitting: Generative AI Advanced Fine-Tuning for LLMs covers this well. The classic sign is training loss going down while eval loss goes up. But I've seen models that looked fine on loss curves and still overfit.
Here's a better test: generate 50 unseen prompts, all variations of your training task. If the output style shifts toward your training data (starts using your specific phrasing instead of natural language), you're overfitting.
How to avoid overfitting when fine-tuning LLM: Use more data, not stronger regularization. I run 3 epochs max. If the loss curve looks flat, I stop — not push through.
LoRA vs Full Fine-Tune
Full fine-tune is dying. LoRA (Low-Rank Adaptation) does 90% of the work with 1% of the compute.
I fine-tune a 70B model on 4 A100s with LoRA. Same task, full fine-tune needs 8 A100s and takes 5x longer. The quality difference? In our tests at SIVARO, LoRA matched full fine-tune on 3 of 4 benchmarks. The only gap was on complex multi-step reasoning — and even that narrowed with more training data.
Monitoring That Actually Helps
Don't just watch loss curves. Watch these:
python
# Custom evaluation during training
def evaluate_checkpoint(model, test_cases):
scores = {
"format_accuracy": 0, # Did output match required format?
"refusal_rate": 0, # Is model refusing valid requests?
"hallucination_rate": 0, # Are claims verifiable?
"output_length_variance": 0 # Are responses too repetitive?
}
for case in test_cases:
output = model.generate(case["prompt"])
scores["format_accuracy"] += check_format(output, case["expected_format"])
scores["refusal_rate"] += 1 if is_refusal(output) else 0
return scores
One checkpoint might have great loss but high refusal rate. Another might have higher loss but better format compliance. The loss curve won't tell you which is which.
Production Deployment (Where Projects Die)
The Evaluation Overhead
Fine-tuning doesn't end when training stops. The real work begins.
You need:
- A holdout test set (500+ examples)
- Human evaluation on 100 samples per checkpoint
- A/B testing infrastructure
A startup I advised in 2025 skipped human evaluation. Their fine-tuned model started using passive voice 73% of the time — the training data was all legal documents. They deployed it for customer support. Customers complained the bot sounded "robotic and distant."
We spent 2 weeks fixing something that cost $200 in human evaluation time to catch.
Latency and Cost
Fine-tuned models aren't magic. A 70B model still takes 2-3 seconds per generation on A100s. If you need real-time (under 500ms), you have options:
- Use a smaller model (7B) fine-tuned aggressively
- Use speculative decoding with your fine-tuned model as the draft
- Cache common outputs
We use option 2 for our production systems. The draft model generates quickly, the fine-tuned model verifies. Gets latency under 300ms while keeping quality.
Monitoring Drift
Your fine-tuned model will degrade. Not because of weight decay (it doesn't happen). Because the input distribution shifts.
Users start asking different questions. New products launch. Regulations change.
Set up automated drift detection:
python
# Track output distribution shift
def drift_score(current_outputs, baseline_outputs):
# KL divergence between output distributions
return kl_divergence(
get_character_distribution(current_outputs),
get_character_distribution(baseline_outputs)
)
If drift score increases by 20% in a week, it's time to re-evaluate.
Cost Analysis (The Numbers Nobody Shows)
LLM Fine-Tuning Business Guide breaks down the economics. Here's what I've seen:
Training costs:
- 7B model with LoRA on 100K examples: $500-800
- 70B model full fine-tune on 100K examples: $5,000-10,000
- Data preparation (cleaning + labeling): $2,000-5,000 for 10K examples
Ongoing costs:
- Inference per 1M tokens (7B): $0.20
- Inference per 1M tokens (70B): $2.00
- Monitoring and re-evaluation per month: $500-2,000
The cheap part is training. The expensive part is maintaining quality over time.
FAQ
Q: How many examples do I need to fine-tune?
100-500 for simple format changes. 1,000-5,000 for behavior modification. 10,000+ for teaching new reasoning capabilities.
Q: How to avoid overfitting when fine-tuning LLM?
Use more diverse data, not more epochs. I cap at 3 epochs. Monitor output diversity — if responses get repetitive, you're overfitting.
Q: Should I fine-tune or use RAG?
RAG for knowledge. Fine-tuning for behavior. If you need both, do RAG first, then fine-tune the prompted model.
Q: Can I fine-tune on my laptop?
For 7B models with LoRA, yes. For anything larger, you need cloud GPUs.
Q: How long does fine-tuning take?
Small models (7B): 2-6 hours. Large models (70B): 24-72 hours. Data preparation: 1-4 weeks.
Q: What's the success rate for production fine-tuning projects?
About 60% in my experience. The failures are usually bad data or wrong use case.
Q: Do I need to fine-tune if I'm using GPT-4?
Rarely. GPT-4's instruction following is already strong. Fine-tune if you need guaranteed output format or cost reduction.
Q: What metrics matter most?
Task-specific accuracy. Not perplexity, not loss. Can the model do the job?
The Decision Framework
Here's how to decide if fine-tuning is worth it:
- Can I solve with prompting? (2 days)
- Can I solve with RAG? (1 week)
- Can I solve with fine-tuning? (4-6 weeks)
Each step adds time and cost. Only move to the next when you've proven the previous can't work.
This isn't conservative — it's practical. In 2024, a logistics company I consulted for spent $40,000 fine-tuning a model for warehouse routing. The base model with a well-crafted prompt and a 20-line Python script achieved 94% accuracy. Fine-tuning got them to 96%. The cost: $40,000 for 2%. Was it worth it? Their CFO said no.
What I'd Do Differently
If I started over:
- Spend 2 weeks on prompt engineering before touching fine-tuning
- Build evaluation before building training data
- Start with the smallest model that could work
- Budget 2x what you think for data cleaning
- Plan for model degradation from day one
The labs that produce the best fine-tuned models don't have secret techniques. They do the basics well — clean data, good evaluation, fast iteration.
That's it. That's the secret.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.