What Fine-Tuning Actually Changes
Every large language model is a neural network — a web of connected nodes. The connections are governed by weights, also called parameters. When someone says "a 70 billion parameter model," they mean a network with 70 billion floating-point numbers, each determining how strongly one node influences another.
A model's knowledge, personality, and behavior aren't stored anywhere explicitly — they emerge from the combination of these weights. Training is the process of adjusting weights so the network produces useful outputs. Fine-tuning continues that process, nudging weights in a specific direction without starting from scratch.
Fine-tuning predates LLMs. Pre-LLM practitioners fine-tuned Word2Vec and BERT for text tasks, AlexNet for images. Transfer learning — freezing early layers and replacing the final head — was the dominant pattern. LoRA applies the same intuition to billion-parameter models: freeze almost everything, inject small trainable adapter layers, achieve most of the performance at a fraction of the cost.
Fine-Tuning Is Not Your First Move
The principle is incremental improvement. Every step you take should be observable and attributable. When you skip steps, you lose the ability to know what's actually working — and you pay training costs for gains that prompting might have delivered for free.
-
1
Prompt engineering & instruction tuning
Write a detailed system prompt. Add few-shot examples. Define explicit tone, format, and constraint rules. Measure on a golden dataset before touching anything else. This is free and instantly reversible.
-
2
Context engineering & memory layers
Introduce dynamic context injection, conversation memory, and better-structured tool descriptions. Many behavioral gaps disappear when the model has richer, more structured context — without touching the model at all.
-
3
RAG: attach external knowledge
When the model lacks domain knowledge, give it access rather than burning it into weights. Start with simple vector search, then iterate toward hybrid retrieval and re-ranking. Measure again.
-
4
Consider fine-tuning only if a gap persists
After steps 1–3, if there's still a measurable gap that cheaper methods can't close and the ROI justifies training cost — fine-tuning enters the picture. This is step four, not step one.
The Economics of Incremental Progress
Here's why the order matters financially:
Two Reasons to Fine-Tune: Knowledge vs. Behavior
When fine-tuning is genuinely warranted, it's almost always because of one of two gaps. Diagnosing which gap you have changes the solution — and knowing that knowledge gaps are often better addressed by RAG first is one of the most common things practitioners get wrong.
My field notes
I spent about seven months going through GenAcademy sessions, documentation, and building side tools before this clicked. Looking back, I had two completely different types of gaps at different points — and I didn't know how to label them at the time.
My brother Sridhar Hassan gave me advice early in my career that I've thought about a lot while building in AI: "The technology doesn't matter. VB today, Java tomorrow — it'll all get replaced. But if you truly understand one backend, one frontend, full integration end to end — that understanding survives everything." Fine-tuning vs RAG is the same principle. The distinction between adjusting behavior and loading knowledge — that transfers across every model, every framework, every year.
When I finally mapped the spectrum — prompt engineering → RAG → RAFT → fine-tuning — it stopped being a list of separate tools and became one progression. Each technique fills a different gap. Getting that mental model right is more durable than memorizing which API to call.
What it is
The model doesn't know what it needs to know. Enterprise data, proprietary terminology, internal processes — none of this was in the pre-training corpus. Fine-tuning teaches the model to reason over domain-specific patterns naturally.
Engineering platform example
A B2B SaaS company's AI assistant knows Python and cloud APIs broadly, but not the team's internal service architecture, naming conventions, or proprietary SDK patterns. Fine-tuning on internal docs and code teaches it to reason over those correctly — not just look things up.
Critical nuance: fine-tuning ≠ memorization
Fine-tuning a model on your 500-page document does not make it reliably recall every fact in that document. It changes weights — a lossy compression. For hard factual lookup, RAG is always the right answer. Fine-tuning teaches the model to think like a domain expert. RAG gives it a reference library to look things up in. Both are often needed for high-stakes domains.
What it is
The model has enough knowledge but doesn't respond the way you need it to. Wrong tone, inconsistent formatting, missing persona, or behavior that drifts after long conversations — fine-tuning reshapes how it communicates, not just what it knows.
E-commerce support example
An online retailer's support AI must always respond in a fixed structure: Issue → Root Cause → Resolution Steps → Escalation trigger. A system prompt works for short sessions, but after many turns the format drifts. Fine-tuning bakes the structure into the weights so it holds at any conversation length.
Behavioral signals
• Model ignores format rules after long conversations
• Inconsistent tone across sessions despite identical prompts
• Brand voice can't be maintained at production scale
• Wrong register — too casual, too hedging, wrong persona
Try prompting first
Aggressive few-shot examples, structured output schemas, and layered system prompts resolve most behavioral issues. Fine-tune only when prompting proves too fragile or inconsistent at scale — not as a shortcut to avoid writing a good system prompt.
The Method Spectrum
These four approaches aren't competitors — they're a progression. Each one builds on what came before and is more powerful (and more expensive) than the last. Move to the next only when the current one is exhausted.
System prompts, few-shot examples, chain-of-thought instructions, output format constraints. The model is unchanged — you're just getting better at communicating with it.
Best for: Behavioral shaping, tone and format control, general-purpose cases where the model already has enough domain knowledge. Always the starting point.
Hard limit: Instructions live in the context window. Long conversations push early instructions out, behavior drifts, and deep domain expertise cannot be prompted in — the model either knows something or it doesn't.
Connect the model to an external database. At query time, retrieve relevant chunks and inject them into context. The model reasons over retrieved content — like an open-book exam. The document stays intact, word for word. Nothing is "burned" into weights.
Best for: Knowledge gaps, factual accuracy, up-to-date information, large document corpora. RAG preserves your data faithfully and lets you update it without retraining.
Hard limit: The model is still a generalist. It can look things up, but it doesn't think like a domain expert. Behavior and tone remain unchanged by the underlying training.
Fine-tunes the model to reason over retrieved context — not to memorize data, but to distinguish relevant chunks from noise and synthesize answers the way a domain expert would. A curated subset trains the model how to reason with retrieved content; the rest stays in the external database.
Best for: Domains where factual accuracy and expert-level reasoning style are simultaneously required — think financial analysis, logistics, enterprise search, or any field where both the data and the reasoning pattern matter.
Hard limit: Complex to implement. Requires both a mature retrieval pipeline and a carefully curated fine-tuning dataset. Usually built on top of existing RAG infrastructure, not a starting point.
You're changing the model itself. Weights shift so baseline behavior, domain fluency, and response style are permanently different. Think of a generalist consultant who goes back to specialize in one industry — the broad reasoning ability stays, but every instinct, vocabulary choice, and framing is now shaped by deep domain immersion.
Best for: Consistent behavioral change at scale, domain-specific tone that can't be maintained through prompting, custom tool-use patterns, latency-sensitive applications where RAG round-trips are too slow.
Hard limit: Expensive GPU training, requires high-quality curated datasets, risks catastrophic forgetting, cannot reliably memorize hard facts, and needs ongoing evaluation infrastructure. Not a one-time project.
The Decision Framework
Work through this diagnostic before reaching for fine-tuning. Move to the next question only when the current approach is genuinely exhausted and measured.
Iterate on your system prompt. Add structured output constraints, explicit rules, and format examples. This is the cheapest, most reversible option — never skip it.
Is it a knowledge gap (model doesn't know your domain) or a behavioral gap (it has the info but behaves wrong)?
Invest in retrieval quality: chunking strategy, metadata filters, hybrid search, re-ranking. Most knowledge gaps are RAG problems — not fine-tuning problems.
The model can access information but can't reason over it like an expert, latency is too high, or behavior remains inconsistent at production scale.
You've exhausted cheaper options and the math works. Start with parameter-efficient LoRA before committing to full fine-tuning. Always evaluate against a golden dataset.
Going from 70% to 80% doesn't always change business outcomes enough to justify training costs. The math has to work first.
Method Comparison at a Glance
| Dimension | Prompt Eng. | RAG | Fine-Tuning | RAFT |
|---|---|---|---|---|
| Changes model weights? | ✗ | ✗ | ✓ | ✓ |
| Adds domain knowledge? | ~ Limited | ✓ Yes | ~ Indirect | ✓ Yes |
| Changes behavior / tone? | ~ Fragile | ✗ | ✓ Durable | ✓ Durable |
| Reliable factual recall? | ✗ | ✓ Yes | ✗ | ✓ Yes |
| Training cost | ✓ None | ✓ Minimal | ✗ High | ✗ High |
| Implementation complexity | ✓ Low | ~ Medium | ✗ High | ✗ Very High |
| Reversible? | ✓ Instantly | ✓ Yes | ✗ New checkpoint | ✗ New checkpoint |
| Latency impact at inference | ✓ None | ~ Retrieval overhead | ✓ None | ~ Retrieval overhead |
The Big Misconception
The Open-Book Exam
You have no medical background but you're given a healthcare textbook during the exam. You can flip through it, find the relevant section, and write an answer based on what you read. You're referencing the information — not internalizing it. The book stays intact, word for word. Nothing is compressed or approximated.
Going to Medical School
Now you've actually studied medicine. You understand why a combination of symptoms points to one condition over another. You've internalized the reasoning, terminology, clinical mindset. But you still won't recall every dosage from memory — you look those up. The weights changed; the documents weren't photocopied in. That's the key difference.
This is exactly why the combination matters. RAG preserves your data faithfully and retrieves it on demand. Fine-tuning changes how the model reasons and communicates. Together, you get a model that thinks like a specialist and has access to authoritative references. That's RAFT.
RAFT: When You Need Both
RAFT (Retrieval-Augmented Fine-Tuning, from Microsoft Research) is the answer when neither RAG nor fine-tuning alone is sufficient. The insight: fine-tune the model to reason over retrieved context better — not to memorize data, but to think like a specialist who also has a reference library.
Why the name finally clicked for me
Seven months of GenAcademy sessions, documentation reads, and side builds. Then one session on fine-tuning. I remember thinking — this isn't starting from scratch. I'm not an empty model. Every prior session was retrieval: pulling in concepts, letting them sit, building a vocabulary for all of this.
That's what RAFT describes. Bring what you've already retrieved from all those weeks of accumulation. Let this session fine-tune the behavior — the way you actually apply and connect those concepts. The combination is what produces the picture.
The difference between RAG and fine-tuning in one line: one looks things up, the other thinks differently. RAFT is the pattern where you need both at once.
Supply chain AI
Fine-tuning teaches supply chain reasoning — lead-time tradeoffs, supplier scoring logic, demand seasonality patterns. RAG delivers live inventory levels, current pricing, and logistics feeds from the ERP. A supply planner who knows the domain cold but still checks live data before committing — that's the RAFT pattern.
Financial analysis AI
Fine-tuning gives the model analyst-grade reasoning: how to read earnings structure, interpret margin trends, and frame risk. RAG delivers live 10-K filings, current price data, and market feeds. Even an experienced analyst pulls up the actual filing before writing a note — the model should too.
Inside Fine-Tuning: SFT & Preference Optimization
When you do fine-tune, there's a progression of methods — each incrementally refining model behavior. They work best in this order, and you don't have to use all of them.
Step 1 — Supervised Fine-Tuning (SFT)
The most common starting point. You create a dataset of (input → desired output) pairs and tell the model: "For this input, this is how you should respond." The model adjusts weights to minimize the difference between its output and your target.
SFT works well for well-defined tasks with clear right answers — specific response formats, domain terminology, tool-calling patterns. It's interpretable, relatively affordable, and the right first step before reaching for more complex methods.
Step 2 — Preference Optimization (DPO / RLHF)
Some tasks have no single correct answer. Writing a summary, generating content, drafting emails — two excellent responses can look completely different. Preference optimization is for these cases: you give the model multiple response candidates and train on human judgments of which is better.
You've seen this in action: tools like ChatGPT sometimes show two responses and ask which you prefer. That preference data flows back into training, shaping the model toward the preferences of your specific user base — which may differ significantly from a general audience.
Example: Building an agent for a Japanese audience. Responses that land well for a Western audience may not align with Japanese communication norms. Preference optimization on data from Japanese users shapes the model toward those expectations without retraining from scratch.
Cost, Latency & Trade-offs
Fine-tuning has more costs than just compute. Here's a complete picture of what you're actually committing to — before opening a training job.
Not a $5–$10 operation. Even parameter-efficient LoRA requires significant cloud spend. Full fine-tuning of a large model can run into thousands of dollars per training run — and you'll iterate more than once before getting it right.
Quality of training data is the bottleneck, not the algorithm. Assembling a clean, representative, properly formatted fine-tuning dataset is frequently more labor-intensive than the training itself. Garbage in, garbage out — but amplified.
RAG adds retrieval round-trips. In real-time applications where sub-second response matters, a fine-tuned model with domain knowledge baked into its weights can be meaningfully faster than one that has to retrieve context first.
A fine-tuned model is a new artifact with its own lifecycle. It needs regression tests, monitoring, and re-evaluation over time. When your base model provider releases an updated version, your fine-tune may need to be redone.
Summary: The Specialist Protocol
- Prompting can't reliably maintain tone or format at production scale
- Custom tool-calling patterns need to be internalized by the model
- RAG latency is a bottleneck in a real-time application
- You need durable behavioral change — not hints in the context window
- The performance gap justifies the training cost and maintenance overhead
- You have a clean, representative, curated training dataset ready
- You haven't tried a detailed, well-structured system prompt yet
- You expect it to memorize factual data reliably — use RAG for that
- RAG hasn't been evaluated as a solution for the knowledge gap
- The accuracy delta doesn't translate to measurable business outcomes
- You don't have a curated, high-quality training dataset ready
- You have no evaluation infrastructure to measure improvement
Public Sources & Further Reading
LoRA Paper — Hu et al. 2021
arXiv:2106.09685 — Low-Rank Adaptation of large LMs
HuggingFace TRL
huggingface.co/docs/trl — SFT, DPO, RLHF training library
Unsloth GitHub
unslothai/unsloth — fast LoRA fine-tuning, open source
HuggingFace Datasets
huggingface.co/docs/datasets — data prep for fine-tuning
My workflow: A GenAcademy session introduced me to the specialist fine-tuning mindset — when to fine-tune vs. prompt-engineer, LoRA mechanics, and dataset quality as the bottleneck. From there I worked through the original LoRA and QLoRA papers, HuggingFace TRL documentation, and open-source tools like Unsloth, then built this guide by walking through each decision myself. This is my personal learning synthesis. A live session has far more depth and takeaways than any set of notes captures — for the full curriculum, reach out to GenAcademy directly.