Fine-tuning an LLM on a GPU: memory requirements for LoRA and QLoRA

What LoRA and QLoRA change in memory

LoRA keeps the pre-trained weights frozen and trains small low-rank update matrices beside them, so only a small fraction of parameters receives gradients. In Hugging Face PEFT the trainable count depends mainly on the rank r and the shape of each targeted weight matrix. The base model still has to sit in GPU memory, and in plain LoRA it sits there in 16-bit.

QLoRA loads that frozen base in 4-bit through bitsandbytes and trains the LoRA adapters on top. The PEFT documentation says this combination lets you fine-tune a 65B model on a single 48 GB GPU. Unsloth's guide puts the trade at about 4x less VRAM than 16-bit LoRA, for training that is slightly slower and marginally less accurate.

• bnb_4bit_quant_type="nf4": the 4-bit data type from the QLoRA paper; Transformers recommends NF4 for training 4-bit base models.
• bnb_4bit_use_double_quant=True: a second quantization pass that saves about 0.4 bits per parameter.
• bnb_4bit_compute_dtype=torch.bfloat16: computes in bf16 instead of the float32 default.

A 4-bit or 8-bit base trains adapters only. The Transformers docs state that 8 and 4-bit training is supported only for training extra parameters, so full fine-tuning needs the 16-bit weights and far more memory.

GPU memory per model size

Unsloth publishes this table for its own training stack and calls the numbers the absolute minimum; some models need more.

A second published point, from the Transformers bitsandbytes guide: with nested quantization, Llama-13b fine-tunes on a 16 GB T4 at sequence length 1024, batch size 1 and 4 gradient accumulation steps. Change any of those settings and the requirement moves.

The settings that move the figure

• Batch size. Unsloth calls per_device_train_batch_size the primary driver of VRAM use and advises 1, 2 or 3 after an out-of-memory error. Raise gradient_accumulation_steps instead: it simulates a larger batch without the memory. Unsloth's starting point is 2 x 8, an effective batch of 16.
• Sequence length. max_seq_length = 2048 is Unsloth's suggestion for a first test, even on models that support longer context.
• Gradient checkpointing. use_gradient_checkpointing = "unsloth" cuts memory by a further 30%, by Unsloth's figure, and is how it handles long-context runs.
• Rank and targets. A larger r uses more memory; Unsloth suggests 8 or 16 for fast runs, up to 128. It advises keeping all major linear layers targeted, because dropping modules saves little.

Which cards fit which run

Unsloth Core lists CUDA compute capability 7.0 as its minimum (V100, T4, Titan V, RTX 20 series and newer) and says GTX 1070 and 1080 work but slowly. bitsandbytes itself runs NF4 from compute capability 6.0. The table below is our reading of Unsloth's minimums against the memory classes on the GPU price list; treat a fit that only just clears as too tight for real batch sizes.

The /gpu page sorts cards onto three shelves. The compute accelerators carry large memory and ECC and are the shelf it names for fine-tuning and runs that have to finish. The GeForce cards have no ECC, which the page frames as fine where a flipped bit costs a retry; on a multi-hour run, that retry is the whole run.

Unsloth and bitsandbytes both document AMD GPU support through ROCm, and Unsloth ships a separate unsloth/unsloth-rocm image. Check your exact card against their AMD guides before picking an Instinct part.

Why a whole machine suits a training run

• The card is passed through whole. No vGPU profile, no MIG partition, no time-slicing scheduler. A run's memory headroom is the card's, not a slice of it.
• You pin the stack. You install the driver and the CUDA release; Unsloth recommends CUDA toolkit 12.4 or newer, 12.8 for Blackwell. Nothing upgrades it mid-run.
• Nothing counts the bytes. Base models and datasets come in, checkpoints go out, and the port is unmetered in both directions.
• A console when the network is gone. The machine's iLO or iDRAC sits on a separate management network, so a driver install that breaks networking is still recoverable.
• Monthly, no contract. A racked machine is live about 30 minutes after approval; a card fitted to order is a build, 4-24 hours.

Setting up Unsloth on the server

1. Install Ubuntu 20.04 or newer (the release Unsloth names for Linux), then the NVIDIA driver, and confirm the card with nvidia-smi.
2. Install Unsloth Core into a virtual environment with uv, as the Unsloth README gives it (first block below).
3. Load the base model in 4-bit and attach LoRA adapters across all linear layers (second block below).
4. Train with TRL's SFTTrainer, as the Unsloth notebooks do: learning_rate = 2e-4 and 1-3 epochs is Unsloth's starting advice. Watch the loss; one that falls to 0 can mean overfitting.
5. Save the adapter, then run it: GGUF through llama.cpp or Ollama, or with its base model behind a vLLM server for many users.

curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv unsloth_env --python 3.13
source unsloth_env/bin/activate
uv pip install unsloth --torch-backend=auto

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Llama-3.1-8B-Instruct",
    max_seq_length = 2048,
    load_in_4bit = True,   # False = 16-bit LoRA
)
model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",],
    lora_alpha = 16,
    lora_dropout = 0,
    bias = "none",
    use_gradient_checkpointing = "unsloth",
    random_state = 3407,
)

Unsloth's guide says accuracy holds best when you train and serve in the same precision: train in 4-bit to serve in 4-bit.

More than one GPU

Unsloth runs on several GPUs through Accelerate and DeepSpeed. Its simplest mode is Distributed Data Parallel: write the run as train.py and start it with torchrun --nproc_per_node N_GPUS train.py or accelerate launch train.py. DDP puts one copy of the model on each GPU and feeds each copy different samples, so throughput rises with every card while the memory each card needs stays the same.

When the model will not fit on one card, Unsloth splits it across the GPUs with device_map = "balanced" in from_pretrained. Unsloth describes its multi-GPU support as still needing manual setup.

On our side, the largest single card holds 96 GB. Past that, 70B in 16-bit LoRA or 405B in QLoRA, the build is two or more GPUs in one chassis, quoted per card: multi-GPU and multi-node servers compares the PCIe lanes each platform gives a card.

When this is the wrong machine

• Full fine-tuning of a large model: it needs the 16-bit weights plus optimizer state, far past the LoRA column. Prove the run with LoRA first, which is also Unsloth's advice.
• A Pascal-era card for Unsloth: Tesla P4, P40, P100 and the GTX 10 series sit below its 7.0 floor, and Unsloth itself calls the GTX 1070 and 1080 slow.
• A model whose QLoRA minimum is within a gigabyte or two of the card: it may load and still fail on the first real batch.