A vLLM server on a whole GPU

Which GPUs run vLLM

vLLM's installation guide sets the floor at CUDA compute capability 7.5. That is Turing and later. Older cards that still carry plenty of memory do not qualify, which matters when you shop by VRAM alone.

The prebuilt wheels target CUDA 12.9 by default, with builds for 12.8 and 13.0. AMD support is limited to the GPUs vLLM lists: Instinct MI200 (such as the MI210), MI300 and MI350, and Radeon RX 7900 and RX 9000, through ROCm 6.3 or later, with prebuilt ROCm wheels for Python 3.12 only. Older Instinct cards such as the MI50 (gfx906) are not on that list.

Sizing VRAM: weights plus KV cache

Weights take parameters times bytes per parameter: about 16 GB for an 8B model at 16-bit, about 5 GB at 4-bit. The rest of the card holds the KV cache, the per-token state of every request in flight. More KV cache means more concurrent requests and longer contexts.

• --gpu-memory-utilization is the fraction of VRAM vLLM claims at start-up, default 0.92. Nothing else should share that GPU.
• --max-model-len caps prompt plus output length. By default it comes from the model config; lower it when the full context will not fit.
• --quantization selects a quantised checkpoint's method; if unset, vLLM reads it from the model config.
• --tensor-parallel-size splits one model across several GPUs in the same machine. The default is 1.

vLLM is built for throughput under concurrency. For a single user and a single GPU, a lighter runtime like the one on our Ollama page is simpler to run.

Installing and serving

1. Install the NVIDIA driver and confirm the card with nvidia-smi.
2. Create a virtual environment with uv and install vLLM with the PyTorch backend chosen automatically, then fetch its kernels.
3. Start the server on localhost with an API key, and point an OpenAI client at http://localhost:8000/v1.

uv venv --python 3.12 ~/vllm && . ~/vllm/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels

export VLLM_API_KEY=$(openssl rand -hex 24)
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --host 127.0.0.1 --port 8000 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 16384

The server speaks the OpenAI Completions, Chat Completions, Responses and Embeddings APIs, plus audio transcription and translation for speech models. Existing OpenAI SDK code needs a new base_url and the key. The example's 7B model at 16-bit is roughly 15 GB of weights, so it wants a 24 GB card; pick a 4-bit checkpoint for a 12-16 GB one.

Containers work too: the official image is vllm/vllm-openai. Run it with --gpus all (or the NVIDIA runtime), mount the Hugging Face cache, publish port 8000, and pass --ipc=host or a larger --shm-size, because PyTorch shares data between processes through shared memory, above all for tensor parallelism. Bind the published port to loopback (-p 127.0.0.1:8000:8000); Docker's port publishing bypasses ufw rules.

What --api-key does not protect

vLLM's documentation states that the key authenticates only routes under /v1, /v2 and /inference. Other endpoints on the same HTTP server, /invocations among them, are not authenticated, and the docs advise against relying on --api-key alone.

• Bind vLLM to 127.0.0.1 and publish it through a reverse proxy with TLS that forwards only the paths you use.
• Authenticate at the proxy as well, and rate-limit per key.
• Firewall everything except SSH and the proxy's port.
• Log usage at the proxy; vLLM's own metrics belong on a private interface.

Why a whole GPU

vLLM assumes it owns the card: it allocates most of the VRAM at start-up and keeps it. On our GPU dedicated servers the GPU is passed straight through to your operating system, with no hypervisor, no vGPU slice and no neighbour on the silicon, so the memory vLLM sees is all of it.

Billing is monthly per machine, with no hourly meter and no egress line: the port is unmetered in both directions, so token traffic never becomes a bandwidth bill. Machines are in five cities on a monthly term. For model-by-model sizing, see LLM hosting; for tensor parallelism across several cards, multi-GPU servers.

When vLLM is not the tool

A Pascal or Volta card is below vLLM's supported floor; llama.cpp or Ollama are the tools there. On CPU-only machines, quantised 7-14B models through llama.cpp are the practical route. vLLM earns its complexity when many requests arrive at once.