
Short Answer: What is LoRA in AI?
LoRA (Low-Rank Adaptation) is a fine-tuning method that freezes a pretrained model’s original weights and trains a small set of added matrices instead. Because the update is stored as two small matrices, LoRA cuts the number of trainable parameters, the training memory, and the size of the saved file compared with full fine-tuning, while the base model stays unchanged.
Key Takeaways
- LoRA was introduced by Microsoft researchers in 2021. On GPT-3, they reported 10,000 times fewer trainable parameters and about one-third the GPU memory of full fine-tuning. (Hu et al.)
- The base model is frozen. Only a small adapter is trained and saved, so one base model can serve many tasks by swapping adapters.
- LoRA is a kind of fine-tuning, not an alternative to it. QLoRA is LoRA plus a quantized base model, which tackles a different memory problem.
- Whether LoRA matches full fine-tuning depends on the task. One study found it substantially underperformed on code and math but forgot less of what the base model knew. (Biderman et al.)
- Rank, target modules, and scaling are the main settings, and there is no single best configuration.
Customizing a large AI model used to mean retraining billions of numbers. LoRA changed that by training a small add-on instead. It is now a widely used way to adapt language and image models. This guide explains how it works, what the numbers in the original paper do and do not show, and where the method falls short. If the vocabulary is new, start with our explainers on fine-tuning and model parameters.
What Is LoRA in AI?
LoRA stands for Low-Rank Adaptation. It comes from a 2021 paper by researchers at Microsoft, LoRA: Low-Rank Adaptation of Large Language Models. The method freezes the pretrained weights and injects small trainable matrices into the layers of a Transformer model. It belongs to a family called parameter-efficient fine-tuning, or PEFT.
The paper’s headline numbers come from one setup: GPT-3 with 175 billion parameters, compared with full fine-tuning using the Adam optimizer. In that setting the authors reported:
- About 10,000 times fewer trainable parameters and a GPU memory requirement about 3 times lower.
- Training memory falling from 1.2 terabytes to 350 gigabytes.
- A saved checkpoint shrinking from 350 gigabytes to 35 megabytes, with rank 4 applied to only the query and value projection matrices.
- A 25% speedup during training.
These are results from specific configurations, not guarantees for every model. The authors also reported quality on par with or better than full fine-tuning on the models they tested (RoBERTa, DeBERTa, GPT-2, and GPT-3). Later research has found a more mixed picture, covered below.
How Does LoRA Work?
Picture a large pretrained model as a finished textbook. Full fine-tuning reprints the whole book to change a handful of pages. LoRA instead attaches a sticky note with corrections to specific pages. The textbook stays untouched, and the note supplies the adjustment.
In technical terms, a model contains weight matrices. Call one W. Ordinary fine-tuning learns a full-size update ΔW. LoRA freezes W and represents the update as the product of two much smaller matrices, B and A:
W′ = W + (α / r) · BA
Here r is the rank, which sets the size of the sticky note, and α is a scaling constant. In the paper, A starts with random values and B starts at zero, so the adapter changes nothing at the beginning of training and learns its adjustment from there. The authors’ hypothesis is that the change a model needs during adaptation has a low “intrinsic rank,” meaning it can be captured by far fewer numbers than the full matrix holds.
Frozen Base Model vs. Trainable Adapter: What Is the Difference?
| Base model | LoRA adapter | |
|---|---|---|
| Size | Large, often billions of parameters | Small, a tiny fraction of the base |
| State during training | Frozen | Trainable |
| Stored as | One large file | A small, separate file |
| Swappable | Shared across tasks | Yes, one per task |
Freezing the base model drives most of the savings. The paper explains that there is no need to calculate gradients or keep optimizer states for most parameters, which is where the memory reduction comes from. The saved output is a small adapter, not a full copy of the model.
One limit is easy to miss: the base model still has to be loaded during training and at inference. LoRA reduces trainable parameters and optimizer overhead, not the footprint of holding the base model itself. The paper notes the 350-gigabyte model is still needed at deployment, but storing 100 adapted versions takes about 354 gigabytes instead of 35 terabytes.
How Much Smaller Is a LoRA Adapter?
Illustrative arithmetic, not a benchmark. Take a single 4,096 by 4,096 weight matrix. A full update has about 16.8 million parameters. A LoRA adapter for the same matrix has r × (rows + columns) parameters:
| Rank | Adapter parameters for this one matrix | Share of the full matrix |
|---|---|---|
| r = 4 | 32,768 | 0.20% |
| r = 8 | 65,536 | 0.39% |
| r = 16 | 131,072 | 0.78% |
| r = 64 | 524,288 | 3.12% |
This is a per-matrix illustration. Total savings depend on how many layers get adapters and whether other parts stay trainable. For a documented whole-model example, Hugging Face’s PEFT documentation shows a rank-16 configuration on the query and value modules of an 86.5-million-parameter model, plus a trainable classifier head, which left 667,493 parameters trainable, or about 0.77% of the model.
What Does a LoRA Training Workflow Look Like?
- Load the pretrained base model.
- Freeze its original weights.
- Choose target modules. The original paper adapted only attention weights, usually the query and value projections, and left the rest frozen. In many models these modules are named
q_projandv_proj. - Insert the LoRA matrices A and B at a chosen rank into those modules.
- Train only the adapter matrices, plus any extra parts you explicitly select, such as a classification head.
- Save the adapter weights separately from the base model.
- At inference, either load the base model with the adapter or merge the two into one set of weights.
Merging matters for speed. Because the adapter is a simple addition to the weights, it can be folded into the base model, and the paper says this introduces no extra inference latency compared with a fully fine-tuned model. Hugging Face’s LoRA guide describes the same step and notes that its tools can also unmerge an adapter or load a different one later.
What Are the Key LoRA Settings?
- Rank (r). Controls the adapter’s capacity. A lower rank means smaller matrices and fewer trainable parameters. The paper found that a rank as low as one or two was enough in its GPT-3 experiments, and it used rank 4 for its headline checkpoint figure. Hugging Face’s PEFT library defaults to rank 8.
- Target modules. Which layers receive adapters. The paper studied attention weights only. PEFT’s documentation notes that QLoRA adds adapters to all the linear layers of a model, and offers an
all-linearoption to do the same. - Alpha (scaling). Controls how strongly the adapter’s output is applied, through the factor alpha divided by rank. A variant called rank-stabilized LoRA, or rsLoRA, divides by the square root of the rank instead. Its author argues the original factor slows learning at higher ranks, which has kept LoRA to very low ranks in practice.
- Dropout. A regularization setting for the adapter layers. PEFT’s default is zero.
There is no single ideal configuration. The right rank, modules, and scaling depend on the task, the amount of data, and how much the model’s behavior needs to change.
LoRA vs. Full Fine-Tuning vs. QLoRA: What Is the Difference?
| Feature | Full fine-tuning | LoRA | QLoRA |
|---|---|---|---|
| Base weights | Updated | Frozen | Frozen and quantized to 4-bit |
| Trainable parameters | All | Small adapter matrices | Small adapter matrices |
| Optimizer memory | High | Much lower | Much lower |
| Memory to hold the base model | Full | Full | Reduced by quantization |
| Saved file | A full copy of the model | A small adapter | A small adapter |
| Many tasks | One full copy per task | One adapter per task on a shared base | One adapter per task on a shared quantized base |
| Capacity to change the model | Highest | Limited by rank and target modules | Limited by rank and target modules |
LoRA is a type of fine-tuning, not a separate category. The real distinction is between updating every parameter and updating a small added set. Our guide to fine-tuning covers the standard approach.
QLoRA is not another name for LoRA. The QLoRA paper describes backpropagating through a frozen, 4-bit quantized model into LoRA adapters. Its authors report this reduced memory enough to fine-tune a 65-billion-parameter model on a single 48-gigabyte GPU while preserving the task performance of full 16-bit fine-tuning. LoRA addresses the cost of trainable parameters and optimizer state. Quantization addresses the cost of storing the base weights. They are two different bottlenecks that are often solved together.
Does LoRA Match Full Fine-Tuning?
The evidence is mixed, and it depends on the task.
- The original paper reported quality on par with or better than full fine-tuning on its benchmarks, and Hugging Face describes LoRA performance as comparable to fully fine-tuned models.
- A 2024 study, LoRA Learns Less and Forgets Less, compared the two on programming and mathematics. In standard low-rank settings, LoRA substantially underperformed full fine-tuning. The authors also found that full fine-tuning learned changes with a rank 10 to 100 times greater than typical LoRA configurations, which may explain some of the gap.
- The same study found an upside. LoRA better preserved the base model’s performance on tasks outside the target domain. In other words, it learned less and forgot less.
A reasonable reading is that LoRA works well when the needed change is modest, such as a format, tone, or narrow task, and that harder domain shifts may need a higher rank, more target modules, or full fine-tuning. Treat that as a rule of thumb to test, not a law.
What Are the Practical Tradeoffs and Limitations?
- Memory. LoRA lowers trainable-parameter and optimizer memory, but the full base model still has to load unless you also quantize it, as QLoRA does.
- Quality. LoRA is not guaranteed to match full fine-tuning. Results depend on rank, target modules, and data quality.
- Serving many tasks at once. The paper lists this as a limitation: once an adapter is merged into the weights, it is not straightforward to batch inputs for different tasks in a single pass. You can keep adapters unmerged and choose them dynamically when latency is less critical.
- New knowledge. A small adapter has limited capacity. For keeping a model current with facts that change, retrieval is often a better fit. See our explainer on RAG.
- Combining adapters. Tools such as PEFT can blend several LoRA adapters into one, but the result should be tested, since adapters trained separately were not designed to work together.
- Small-data risks still apply. An adapter can still fit a small dataset too closely. Our guide to overfitting explains the warning signs.
Where Is LoRA Used Beyond Chatbots?
The original paper focused on language models, but Hugging Face’s PEFT guide notes the technique can be applied to any dense layers in deep learning models, including diffusion models. Its Diffusers training guide calls LoRA a popular and lightweight technique that produces small weight files, a few hundred megabytes, and supports it for text-to-image models such as Stable Diffusion XL. Small, shareable adapters for a particular visual style or subject are a common use. Our explainer on text-to-image AI covers how those models work.
The same economics apply elsewhere: one frozen foundation model can support many small adapters, so a team can keep one copy of the base model and a separate adapter per client, product, or task.
What Does LoRA Mean for Businesses?
The following are our observations, not claims from the sources above.
- Custom models are cheaper to try. Training a small adapter needs less hardware than retraining a model, which lowers the cost of an experiment.
- You need access to the weights. Training your own adapter requires a model you can load, which is one reason open-weight models and small language models matter for customization.
- It is one option among several. Clear prompts and retrieval are usually cheaper first steps. Fine-tuning, with LoRA or otherwise, makes more sense for consistent style, format, or a narrow repeated task. Model distillation is a different technique that produces a smaller model, not an add-on.
- Ask vendors what was adapted. “Fine-tuned on your data” can mean a small adapter or a full retrain, and the difference affects cost, capacity, and how easily the customization can be updated or removed.
Key Terms to Know
- PEFT. Parameter-efficient fine-tuning, the family of methods LoRA belongs to.
- Adapter. A small trainable component added to a frozen pretrained model.
- Base model. The original pretrained model, whose weights stay frozen.
- Rank (r). The size of the low-rank matrices in a LoRA adapter.
- Target modules. The layers where LoRA matrices are inserted.
- Merge. Folding the trained adapter into the base weights for deployment.
- QLoRA. LoRA combined with a 4-bit quantized base model.
- rsLoRA. A variant that scales the adapter by the square root of the rank.
Learn More About AEO and AI Marketing at Prompt Insider
Since launching earlier this year, Prompt Insider has become a leading authority on AI marketing, Answer Engine Optimization (AEO), large language models, AI search, AI news, and the evolving future of digital discovery. As AEO becomes one of the hottest topics in marketing, Prompt Insider is helping define the conversation around how brands improve visibility, adapt their content strategies, and stay competitive in an increasingly AI-driven search environment.
Prompt Insider is the go-to resource for answer engine optimization, AI marketing, and AI search. Start with our core guides at thepromptinsider.com:
- What Is AEO? Answer Engine Optimization Explained
- AEO vs. SEO vs. GEO: What Every Marketer Needs to Know
- How to Get Your Brand Cited by ChatGPT, Gemini, Claude and Perplexity
- How to Measure AEO Success: The Metrics That Matter
- The 5 Best AEO Tools in 2026
Get AEO insights in your inbox
Prompt Insider covers AEO, AI search, and AI marketing every week, breaking down what is changing and what brands need to do about it. Sign up for our emails at thepromptinsider.com to get it first.
Frequently Asked Questions
What is LoRA in AI, in simple words?
It is a lightweight add-on that teaches a model new behavior without rewriting the whole model, like attaching a sticky note of corrections instead of reprinting a book. The original model stays frozen and only the small add-on is trained.
How does LoRA work in fine-tuning?
LoRA freezes the base model’s weights and inserts two small trainable matrices whose product approximates the needed weight update at a chosen rank. Only those matrices are trained, which is why it needs far fewer resources than updating the whole model.
What are the main LoRA settings?
Rank, target modules, alpha, and dropout. Together they control the adapter’s size, where it is applied, how strongly it affects the model, and how it is regularized during training.
Is LoRA the same as fine-tuning?
LoRA is a type of fine-tuning, specifically a parameter-efficient method. It differs from full fine-tuning because the model’s original parameters are never updated.
What is the difference between LoRA and QLoRA?
LoRA trains small adapters on a frozen base model. QLoRA does the same but loads the base model in a 4-bit quantized form to cut the memory needed to hold it. They address different bottlenecks and are often used together.
Is LoRA as good as full fine-tuning?
Sometimes. The original paper reported comparable or better quality on its benchmarks, while a 2024 study found LoRA substantially underperformed full fine-tuning on programming and math in standard low-rank settings, though it forgot less of the base model’s other abilities.
Does a higher LoRA rank always give better results?
No. A higher rank adds capacity and trainable parameters, but more is not automatically better, and one paper argues the standard scaling slows learning at high ranks. The right rank depends on the task and should be tested.
Bottom Line
LoRA makes model customization more accessible by training a small adapter instead of retraining an entire model. It is a form of fine-tuning, and it is distinct from quantization: QLoRA is LoRA on a quantized base model. The efficiency gains are real and well documented, but the quality tradeoff depends on the task, the rank, and the data. For most teams the question is not whether to fine-tune, but how much of the model to adapt, and that is best answered by testing on your own task.
Sources: Hu et al.: LoRA: Low-Rank Adaptation of Large Language Models (arXiv), Dettmers et al.: QLoRA: Efficient Finetuning of Quantized LLMs (arXiv), Kalajdzievski: A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA (arXiv), Biderman et al.: LoRA Learns Less and Forgets Less (arXiv), Hugging Face PEFT: LoRA conceptual guide, Hugging Face PEFT: LoRA reference, Hugging Face Diffusers: LoRA training guide. Figures from the LoRA and QLoRA papers are the authors’ own results on specific setups. The per-matrix table in this article is illustrative arithmetic. Library defaults can change between versions. We reviewed these sources on November 14, 2026.
About the author
Kai Williams
Kai Williams has been in marketing for years, with a long background in SEO before AEO had a name. He stepped into Answer Engine Optimization the moment AI started reshaping how people search, and has been tracking the shift ever since. At Prompt Insider, he covers AEO, AI marketing, and the future of search, breaking down what is changing and what brands need to do about it.


