Stop 13 · Fine-Tuning & Adapting LLMsTopics · 2/9

LoRA & QLoRA

In short

LoRA is the reason students can fine-tune real LLMs on a single GPU. QLoRA squeezes it further by storing the base model in 4-bit. Together they're the default way to customise open models today.

Low-Rank Adaptation freezes the base model and trains small injected low-rank matrices instead; QLoRA adds quantization to fine-tune large models on consumer GPUs.

The blob clips a small add-on lens onto its phone camera, with spare lenses on the table.

Think of it like this

The clip-on phone lens

You don't buy a new phone to shoot macro photos. You clip a small lens over the camera. The phone stays the same, and you can swap lenses for wide-angle or zoom. LoRA is that clip-on lens for a model. QLoRA also shrinks the phone so it fits in a smaller pocket.

Watch it happen

A QLoRA fine-tuning run

Load 4-bit

Freeze

Add LoRA

Train

Save adapter

1/5Load the base model with weights quantised to 4-bit, so it takes roughly a quarter of the 16-bit memory.

The big ideas

What you need to know

Low-rank update

Instead of a full d x d change, LoRA learns B (d x r) times A (r x d). With small r like 8 or 16, that's tiny.

Key knobs

r sets adapter size, lora_alpha scales its effect, and target_modules picks which layers get adapters.

QLoRA = 4-bit base + LoRA

The 2023 QLoRA paper used a 4-bit NF4 format for the frozen base, cutting memory enough to tune large models on one GPU.

Merge for serving

You can merge adapters into the base weights for faster inference, or keep them separate to swap tasks.

See it in code

Walk through the code

python

1import torch2from transformers import AutoModelForCausalLM, BitsAndBytesConfig3from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training4 5bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",6                         bnb_4bit_compute_dtype=torch.bfloat16)7model = AutoModelForCausalLM.from_pretrained("<open-model>", quantization_config=bnb)8model = prepare_model_for_kbit_training(model)9 10lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,11                  target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],12                  task_type="CAUSAL_LM")13model = get_peft_model(model, lora)14# train with transformers Trainer or TRL's SFTTrainer, then:15model.save_pretrained("my-adapter")

1/4Load the base in 4-bit NF4, computing in bfloat16 for stability.

The QLoRA recipe: 4-bit base via bitsandbytes, then LoRA adapters via PEFT. Needs a CUDA GPU.

Save yourself the pain

Common mistakes

Oops

Setting r=256 thinking bigger is better.

Do this instead

Start with r=8 or 16. Larger ranks add memory and overfitting risk, often for little gain.

Oops

Forgetting which base model an adapter was trained on.

Do this instead

An adapter only works on the exact base it was trained with. Record the base model id with the adapter.

Your mission

Your first adapter

~45 min

Quick check

What does QLoRA add on top of LoRA?

Go deeper

1Fine-tune a small open-weight model with LoRA using the Hugging Face `peft` library.
2Learn the key LoRA hyperparameters: rank, alpha, target modules.
3Try QLoRA on a larger model to see how quantization enables fine-tuning on limited hardware.