llama.cpp server on bare metal: llama-server sized and secured

What llama-server needs from the machine

Memory first, then everything else. The model file has to fit somewhere: in GPU memory for full speed, split between GPU and system RAM for a model larger than the card, or entirely in RAM for a CPU-only box. llama.cpp's quantize documentation publishes the sizes for Llama 3.1 8B, and they are the cleanest reference for how bits per weight translate into gigabytes.

Scale by parameter count: the same document lists Llama 3.1 70B at 43.1 GB in Q4_K_M. Then add the KV cache, which grows with context length and with every concurrent slot. A 70B Q4_K_M therefore wants a 48 GB card, not a 40 GB one, and an 8B model at Q8_0 (7.95 GiB, about 8.5 GB) is comfortable on a 12 GB card and does not fit entirely on an 8 GB one: there, drop to Q6_K or lower -ngl.

GPU layers, partial offload and the CPU path

-ngl (--n-gpu-layers) is the maximum number of layers stored in VRAM. Current builds accept a number, auto (the default) or all; 999 works everywhere and means every layer. Set it below the model's layer count and the remaining layers run from system RAM on the CPU. That is how llama.cpp runs a model larger than the card, at a speed set by memory bandwidth on the CPU side.

--fit is on by default and adjusts unset arguments, context size included, to fit device memory. If a log shows the model loading entirely on the CPU, check that the binary was built with CUDA (--list-devices prints the GPUs it sees) before touching flags. For Mixture-of-Experts models, --n-cpu-moe N keeps the expert weights of the first N layers in RAM while attention stays on the GPU.

A CPU-only llama.cpp server is a real option for quantised 7-14B models with one or two users. Our LLM hosting page sizes that machine class: many cores and fast DDR5 rather than a card.

More than one GPU in the chassis

With several GPUs visible, llama-server splits by layer across all of them (-sm layer, the default): each card holds a contiguous block of layers and the KV cache for those layers. -ts 3,1 sends 75% to GPU 0 and 25% to GPU 1, which is how two cards of different sizes share one model. -sm tensor splits every layer across cards for lower latency, but the project marks it experimental, it requires flash attention and an unquantised KV cache, and it does not support every architecture.

Layer split tolerates a slow link between cards, which matters on PCIe 3.0 platforms. A second card in one machine is quoted by hand rather than configured; the multi-GPU page compares the platforms by PCIe lanes.

Build with CUDA and start the server

1. Install the NVIDIA driver and the CUDA toolkit for your card. On a dedicated server the version is yours to pin; nothing upgrades it underneath you.
2. Clone the repository and build with -DGGML_CUDA=ON. The default build targets the GPU present at compile time; add -DGGML_NATIVE=OFF for a binary that runs on any CUDA card. AMD Instinct cards build with -DGGML_HIP=ON against ROCm instead.
3. Put a GGUF file on local disk, or let -hf user/repo:Q4_K_M download one from Hugging Face.
4. Start llama-server bound to localhost, with keys read from a file, and check /health before sending traffic.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j 8

./build/bin/llama-server -m /srv/models/model-Q4_K_M.gguf \
  -ngl 999 -c 32768 -np 4 \
  --host 127.0.0.1 --port 8080 \
  --api-key-file /etc/llama/keys

curl http://127.0.0.1:8080/health

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LLAMA_KEY" \
  -d '{"messages":[{"role":"user","content":"Say hello"}]}'

Pass --port explicitly. The current server README lists 9931 as the default while its Docker examples use 8080, so a client written for one build can miss the other.

Context size and parallel slots

-c sets the context the KV cache is sized for; 0 takes the model's own maximum, which on a long-context model can be far more memory than the card has. -np sets the number of slots, the requests decoded at the same time with continuous batching. Slots draw on the same cache, so raise -c when you raise -np, or cap each slot with --kv-unified-per-slot.

• -ctk q8_0 -ctv q8_0 stores the cache at 8 bits, roughly half of the f16 default. A quantised V cache needs flash attention, which -fa auto enables where supported.
• GET /slots (on by default) shows what each slot is doing; --metrics adds a Prometheus endpoint.
• Router mode: start llama-server with --models-dir and no model, and it loads models on demand by the request's model field, up to four at once by default (--models-max).

Exposing it safely

llama-server listens on 127.0.0.1 unless --host says otherwise, and with no --api-key it accepts any request. For anything reachable from outside, the project's own recommendation is an API key plus a reverse proxy. /health stays public by design. TLS works either at the proxy or natively, by building with -DLLAMA_OPENSSL=ON and passing --ssl-key-file and --ssl-cert-file.

Leave --tools, --agent and the MCP options off on an exposed server: they let the model read and write files and run shell commands, and the documentation says not to enable them in untrusted environments. Run the process as an unprivileged user under systemd, keep model files read-only, and firewall the port to the hosts that call it.

Which of our machines fit

Our GPU dedicated servers are physical machines rented to one account, with the card passed straight through to your operating system: no hypervisor, no vGPU profile, no MIG partition. You install the driver and the CUDA or ROCm release your build was tested against, and you get every core, all the RAM, root and the server's iLO or iDRAC console on a separate management network, so a driver change that drops the network is recoverable. The port is unmetered, so pulling a 43 GB GGUF or serving tokens all month adds nothing to the bill.

• 7-8B at Q4_K_M to Q6_K: the 8 GB cards, older Tesla and Quadro parts included; Q8_0 wants 12-16 GB.
• 13-14B at 4-bit: the 12-16 GB cards. 30-34B at 4-bit (about 19 GB of weights): 24 GB, for example an RTX 3090 or RTX A5000, leaving room for context.
• 70B at Q4_K_M: a 48 GB card such as the L40S, A40 or RTX A6000. Past 96 GB on one card it is a multi-GPU build.
• AMD Instinct MI50 and MI210 run the HIP backend for teams already on ROCm.

The GPU is a separate monthly line beside the machine, so moving from an 8 GB card to a 24 GB one later changes the bill by the difference between the two cards. Servers run in New York, Miami, San Francisco, Amsterdam and Bucharest on our own network, AS19624.

When another engine is the better fit

llama.cpp is at its best with GGUF quantisations, mixed CPU and GPU memory, older or mismatched cards, and one machine serving a handful of users. If you want models pulled and managed by name with fewer flags, read Ollama server hosting. If the job is many concurrent users on a GPU big enough for the model with headroom left, compare it with a vLLM server.