How to Fine-Tune an LLM for Production
I'm Nishaant Dixit. I run SIVARO, a product engineering shop that builds data infrastructure and production AI systems. We've shipped over 40 fine-tuned models into production since 2023. Some worked. Others burned money and taught us expensive lessons.
This guide is what I wish someone had handed me before we wasted $80K on a model that never made it past staging.
Fine-tuning an LLM for production isn't about squeeze squeezing more accuracy out of a benchmark. It's about building a system that delivers reliable, cost-effective results under real load — and doesn't collapse when the data distribution shifts. That's a different problem, and most tutorials skip it.
You're going to learn what fine-tuning actually buys you, when to do it (and when not to), how to pick data, how to train without blowing your budget, and how to ship without embarrassment. I'll include code, costs, and failure stories.
Let's start with the thing nobody says out loud.
Most Projects Shouldn't Fine-Tune
Every week I talk to founders who want to fine-tune a model. They've heard it's the secret sauce. They're wrong about 70% of the time.
If you're building a chatbot that answers questions about your product docs — don't fine-tune. Use RAG (retrieval-augmented generation). It's cheaper, faster to iterate, and you don't need ML engineers. We benchmarked this at SIVARO in early 2024: RAG beat fine-tuning on factual accuracy for 9 out of 10 use cases we tested.
Fine-tuning is for when you need the model to behave differently — not just know different facts.
Say you're building a medical coding system. The model needs to output structured ICD-10 codes in a specific format, follow rejection rules, and never hallucinate a code that doesn't exist. Prompt engineering won't reliably enforce those constraints. Fine-tuning can.
Or you're processing customer emails and need to extract order IDs, shipping dates, and complaint categories — consistently, even when the email is garbled. That's a behavioral change.
The Google Cloud guide on fine-tuning models puts it well: fine-tuning adapts the model's behavior, not just its knowledge. If you need the latter, use retrieval.
Our rule at SIVARO: if a prompt + 5 examples gets you 85% of the way, don't fine-tune. If you need 95%+ with tight constraints, fine-tune.
What Fine-Tuning Actually Changes
Let's get concrete. When you fine-tune a model like GPT-4o or Llama 3.1, you're not adding new weights. You're adjusting the existing weights through supervised learning on your dataset.
The base model already knows English, reasoning, and general knowledge. Fine-tuning nudges it toward your specific distribution. It learns to prefer your style, respect your output format, and prioritize certain patterns over others.
The OpenAI model optimization docs explain this well: fine-tuning changes the model's output distribution. It doesn't add facts — it changes how the model uses what it already knows.
This is critical. If your training data contains factual errors, the model will learn those errors. It can't distinguish between "this is the correct answer" and "this is the pattern from training." Garbage in, garbage out is not a cliché — it's physics.
I've seen teams fine-tune on transcripts of expert conversations, then wonder why the model occasionally gave bad advice. The transcripts contained corrections — "that's wrong, do it this way" — and the model learned both sides. Don't do this.
When You Should Fine-Tune: Real Criteria
Here's the checklist I use with clients:
-
Output format is non-negotiable. JSON schemas with strict validation. SQL queries that must compile. Legal documents with specific clause structures. Prompting can get close, but fine-tuning nails it.
-
You need speed. A fine-tuned model is often smaller than the one you'd need with prompting. We fine-tuned a 7B parameter model for a client that replaced GPT-4 calls. Latency dropped from 3 seconds to 400ms. Cost dropped 90%.
-
You have high-volume, repetitive tasks. If you're making 10 calls a day, fine-tuning doesn't pay for itself. If you're making 100K calls a day, it's mandatory. Stratagem Systems has a solid business guide on calculating ROI — their math matches ours.
-
You can't afford edge-case failures. A prompt-based system might fail 2% of the time. Fine-tuning can push that to 0.1%. For medical, legal, or financial use cases, that difference is existential.
If none of these apply, stop reading and go build a RAG pipeline. I mean it.
Data: The Only Thing That Matters
Fine-tuning is a data engineering problem with a machine learning surface.
The quality of your training data determines 90% of the outcome. Model architecture, hyperparameters, training tricks — those matter at the margins. Data quality matters at the core.
How to Build a Training Dataset
You need three things:
Diverse examples. If you're building an email classifier, don't give it 10,000 examples of cancellation requests and 10 of refund disputes. The model will bias toward what it saw most. We target at least 100 examples per category, minimum. 500 is better.
Correct outputs. Every example must be vetted. We use a two-person review process for production datasets. Yes, it's slow. Yes, it's worth it.
Edge cases explicitly. If the system should reject an empty input, include that. If it should flag ambiguous dates, include that. The model learns from what's in the data. If you don't show it edge cases, it will fail on them.
The blog post from To The New has a good breakdown of dataset construction. I'd add one thing: test your data splits. We once had a training set where 30% of examples were nearly identical. The model memorized those patterns and failed on slight variations.
Format Matters
Here's the format we use for fine-tuning OpenAI models:
json
{"messages": [
{"role": "system", "content": "You are a medical coder. Output only ICD-10 codes. Never explain."},
{"role": "user", "content": "Patient presents with acute bronchitis, fever, and productive cough for 3 days."},
{"role": "assistant", "content": "J20.9, R50.9, R05"}
]}
Each example is a complete conversation. The model learns to respond in this pattern. Don't use truncated examples — they create ambiguity.
For open-source models like Llama, use their chat template format:
python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")
conversation = [
{"role": "system", "content": "You are a SQL generator. Output only SQL."},
{"role": "user", "content": "Find all orders from 2025 with value over $1000"},
{"role": "assistant", "content": "SELECT * FROM orders WHERE order_date >= '2025-01-01' AND value > 1000;"}
]
encoded = tokenizer.apply_chat_template(conversation, tokenize=True)
Raphael Bauer's fine-tuning walkthrough covers these formatting details well — don't skip them. Formatting errors silently corrupt your training.
Data Size: How Much Do You Need?
Conventional wisdom says thousands of examples. Our experience says it depends.
For a simple task (classify sentiment into positive/negative/neutral), 500 examples can work. For complex generation (write legal contract clauses), we needed 5,000+. The rule: start with 500, evaluate, add more until performance plateaus.
We track validation loss. When it stops improving for 3 epochs, we have enough data. If it's still improving, we need more.
Training: The Practical Playbook
You have two paths: API-based fine-tuning (OpenAI, Anthropic) or self-hosted (Hugging Face, Axolotl, Unsloth).
API Fine-Tuning
This is the easiest path. OpenAI handles infrastructure, checkpointing, and deployment. You upload your data, kick off a job, and wait.
python
from openai import OpenAI
client = OpenAI()
file = client.files.create(
file=open("training_data.jsonl", "rb"),
purpose="fine-tune"
)
job = client.fine_tuning.jobs.create(
training_file=file.id,
model="gpt-4o-mini-2024-07-18",
hyperparameters={
"n_epochs": 3,
"batch_size": 4,
"learning_rate_multiplier": 1.0
}
)
print(f"Job ID: {job.id}")
Cost: around $25 for 100K tokens of training on GPT-4o-mini. GPT-4o is about 10x more expensive. We use mini for most tasks and only graduate when we need the extra capability.
The OpenAI docs have a full reference. Watch for the evaluation metrics they provide — they're honest about overfitting.
Self-Hosted Fine-Tuning
If you need control, data privacy, or cost efficiency at scale, self-host. We use Axolotl with 4x A100s for most jobs.
Here's a minimal config:
yaml
# axolotl config
base_model: meta-llama/Llama-3.1-8B
datasets:
- path: sivarofs://training_data.jsonl
type: chat_template
conversation: llama3
micro_batch_size: 4
gradient_accumulation_steps: 8
learning_rate: 2e-5
num_epochs: 3
lr_scheduler: cosine
warmup_steps: 100
output_dir: ./fine-tuned-llama
LoRA (Low-Rank Adaptation) is your friend. It reduces trainable parameters by 90%+ and cuts VRAM usage. We train 8B models on 4x A100s with LoRA at rank 16. Full fine-tuning of 8B parameters requires 8x A100s minimum.
The Coursera advanced fine-tuning course covers LoRA and QLoRA in depth. I'd argue QLoRA (quantized LoRA) is usually good enough — we use it for 80% of projects.
Hyperparameters That Actually Matter
Three settings dominate outcomes:
Learning rate. Start at 2e-5 for full fine-tuning, 2e-4 for LoRA. If loss spikes, halve it. If training is slow, double it. We've seen models diverge at 5e-5 on full fine-tuning.
Epochs. 3-5 is standard. More than 5 and you're memorizing, not learning. Check validation loss — when it increases, you've overfit.
Batch size. Bigger is better for gradient stability. Use as large as your GPU memory allows. 4 is the minimum for 8B models. 16+ is better.
Evaluation: The Hard Part
Everyone evaluates on a test set. Few evaluate on production traffic. That's the mistake.
Your test set is a snapshot. Production is a firehose. Data distribution shifts, users ask weird questions, edge cases multiply.
We use three evaluation stages:
1. Holdout set. 10% of your training data, untouched during training. This catches basic overfitting.
2. Adversarial set. Hand-crafted inputs that push boundaries. We pay domain experts to write 200 edge cases. If the model fails on any, we add those to training.
3. Shadow deployment. Run the fine-tuned model alongside your current system. Compare outputs for 1-2 weeks. Measure accuracy, latency, and cost.
The Google Cloud guide recommends A/B testing in production. We do this for every model. You'd be surprised how often the fine-tuned model scores better on benchmarks but worse on real traffic.
Production: Shipping Without Pain
Fine-tuning is 30% of the work. Production deployment is 70%.
Serving Infrastructure
API-based models serve themselves. You just call the endpoint. For self-hosted, you need a serving stack. We use vLLM:
python
from vllm import LLM, SamplingParams
model = LLM(model="./fine-tuned-llama")
sampling_params = SamplingParams(
temperature=0.1, # low for deterministic output
top_p=0.9,
max_tokens=512
)
outputs = model.generate("Classify this email: ...", sampling_params)
print(outputs[0].outputs[0].text)
vLLM supports continuous batching — crucial for throughput. We get 50-100 requests per second on a single A100 with an 8B model.
Monitoring
Track three metrics:
Latency p50 and p99. If p99 exceeds 2x p50, you have a batching or memory issue.
Response validity. Parse every output. If the format is wrong, log it and alert. We've caught model drift this way.
Cost per request. Fine-tuning should reduce cost, not increase it. If your per-request cost goes up, something is wrong.
Rollback Plan
Keep the base model ready. If your fine-tuned model starts failing (data drift, formatting regression), switch back to base + prompt. We've done this three times. It's not admitting defeat — it's being responsible.
When Fine-Tuning Fails
Let me tell you about a project that went wrong.
We fine-tuned a model for a healthcare client to extract medication names from clinical notes. We used 10,000 examples from a single hospital system. The model hit 97% accuracy on our test set.
In production, accuracy dropped to 72%.
Why? The hospital had standardized note formats. The model learned that format as a cue. Real-world data from other hospitals had different formats, different abbreviations, different headings. The model hadn't seen that variation.
We fixed it by including data from 15 different institutions. Accuracy climbed to 94%.
The lesson: your training data must reflect the full production distribution. If it doesn't, your model will fail in ways that surprise you.
Another common failure: fine-tuning on outputs from GPT-4. Teams generate "ideal" responses using GPT-4, then fine-tune a smaller model on those. The small model inherits GPT-4's biases and quirks, plus learns to approximate its errors. We've seen this compound failure.
Better approach: use human expert labels. Yes, they're expensive. Yes, they're worth it.
FAQ
How much does fine-tuning cost?
For API-based, $25-300 per training run depending on model size. GPT-4o is around $30 per million tokens. Self-hosted costs $100-500 in compute for a single training run on 8B models. The Stratagem guide has a detailed cost breakdown.
What models should I fine-tune?
GPT-4o-mini for API users. Llama 3.1 8B or 70B for self-hosted. Mistral Large if you need a different tradeoff. Don't fine-tune GPT-4o unless you've exhausted options — it's expensive and often unnecessary.
Can I fine-tune without coding?
OpenAI's fine-tuning UI lets you upload data and start a job. But you'll need code for evaluation, monitoring, and production deployment. No serious production system runs without scripting.
How long does training take?
GPT-4o-mini jobs take 30-90 minutes for 1,000 examples. Self-hosted on 4x A100s: 2-4 hours for 8B models. Larger models take proportionally longer.
What's the difference between fine-tuning and RAG?
RAG retrieves documents and includes them in the prompt. Fine-tuning changes the model's weights. Use RAG for factual questions. Use fine-tuning for behavioral changes. The To The New article explains this well.
How do I avoid overfitting?
Use a validation set. Stop training when validation loss increases. Use LoRA with low rank (8-16). Keep epochs between 3-5.
What size dataset do I need?
Minimum 500 examples for simple classification. 2,000-5,000 for generation tasks. More for complex domains. Quality over quantity — 500 clean examples beat 5,000 noisy ones.
Is fine-tuning illegal for regulated use?
No, but you need to ensure your training data doesn't violate privacy laws. For healthcare or finance, use de-identified data and don't fine-tune on patient PHI or customer PII. When in doubt, consult compliance.
The Bottom Line
Fine-tuning an LLM for production is a data engineering problem. Get the data right, and the model follows. Get it wrong, and no amount of hyperparameter tuning will save you.
Start with the question: do I actually need this? If yes, build a small dataset, run a quick experiment, and measure rigorously. If it works, scale. If it doesn't, figure out why the data failed.
We've shipped over 40 models this way. Some we killed after evaluation. Some we replaced after production drift. The ones that survived had one thing in common: they solved real problems for real users, not benchmark accuracy targets.
That's the goal. Not a better perplexity score. A system that works when it matters.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.