DeepSeek self-hosted: the distill that fits your card, and the models that do not

The DeepSeek weights, by size

DeepSeek published six dense distills of R1 in January 2025 and a seventh, R1-0528-Qwen3-8B, with the May 2025 R1 update. Each is an existing Qwen or Llama model fine-tuned on samples generated by R1, so each runs wherever its base architecture runs. Parameter counts below are from the Hugging Face model pages; file sizes are the tags in Ollama's own library.

The R1 model card licenses the code and weights under MIT and states that the Qwen distills derive from Apache 2.0 models and the two Llama distills from the Llama 3.1 and 3.3 licences. Read the base model's licence before you ship a Llama distill in a product.

Why the 671B models do not fit one card

DeepSeek-R1 and DeepSeek-V3 share one architecture: 671B parameters in total, 37B activated per token, a 128K context. The V3 card adds that the Hugging Face upload is 685B, the extra 14B being the Multi-Token Prediction module, and that DeepSeek ships FP8 weights only. The V4 card lists V3.2 at the same 671B total. The V4 previews go both ways: V4-Flash is smaller at 284B total with 13B activated, V4-Pro larger at 1.6T with 49B activated, both with a one-million-token context.

Only 37B parameters do work on each token. Our reading is that a GPU engine has to keep all 671B within reach, so the activated count lowers compute per token, not the memory floor. Our arithmetic: FP8 is one byte per parameter, so the R1 upload needs roughly 685 GB before any cache; Ollama's 4-bit file is 404 GB. V4-Flash, even if every parameter were half a byte, would be at least 142 GB; DeepSeek ships most of its non-expert weights in FP8, so the real files are larger. Every one of those is above 96 GB, the largest card in our GPU catalogue. DeepSeek's own V3 demo launches with torchrun --nnodes 2 --nproc-per-node 8: two machines, eight GPUs each.

Running a full-size DeepSeek model yourself is a multi-GPU or multi-node build, quoted by hand rather than ordered from a list. Multi-GPU and multi-node servers covers PCIe lanes, interconnect and which platform takes how many cards.

Matching a distill to card memory

The file has to fit with room left for the runtime and the key-value cache, which grows with every token of context. The GPU page puts that head-room at a quarter to a half on top of the weights. Applying that rule to the Ollama file sizes, our reading is:

Quality falls as quantization gets coarser, and longer contexts cost memory, so a 32B at q8 on 48 GB often beats a 70B squeezed in at q4. For the arithmetic behind every model family, not just DeepSeek, see LLM hosting.

Serving R1-Distill-Qwen-14B with vLLM

vLLM loads the Hugging Face weights directly and speaks the OpenAI API. In BF16 the 14B distill is about 30 GB of weights by our arithmetic (14.8B parameters at two bytes), which leaves a 48 GB card room for a 32,768-token context, the length DeepSeek's own example uses. Run these as your own user on Ubuntu with the NVIDIA driver already installed; the engine itself, its flags and a systemd unit are on our vLLM server page.

1. Turn on the firewall with SSH only, before anything listens: sudo ufw allow OpenSSH followed by sudo ufw enable. Check that IPV6=yes is set in /etc/default/ufw so the rule covers IPv6 too.
2. Install uv and vLLM, generate a key into a file only you can read, and start the server on loopback:

curl -LsSf https://astral.sh/uv/install.sh | sh
# open a new shell so uv is on PATH, then:
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels

umask 077 && openssl rand -hex 32 > ~/.vllm-key
export VLLM_API_KEY="$(cat ~/.vllm-key)"
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-14B \
  --host 127.0.0.1 --port 8000 \
  --max-model-len 32768 \
  --reasoning-parser deepseek_r1

The key goes in through the VLLM_API_KEY environment variable rather than --api-key on the command line, where ps and shell history would show it. Test from a second shell on the server; the curl below reads the key from the same file, and while it runs the header is visible in ps. --reasoning-parser deepseek_r1 makes vLLM return the model's reasoning in a separate reasoning field instead of mixing it into the answer.

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer $(cat ~/.vllm-key)" \
  -H "Content-Type: application/json" \
  -d '{"model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-14B",
       "temperature": 0.6,
       "messages": [{"role": "user", "content": "Which is larger, 9.11 or 9.8? Reason step by step."}]}'

For a quick start from a pre-quantized file, ollama run deepseek-r1:14b pulls the 9 GB q4 build instead; our Ollama server page covers it. llama.cpp runs the same GGUF files with finer control over offload, and SGLang is one of the engines DeepSeek's V3 card recommends for the full model.

DeepSeek's own sampling rules

• Temperature 0.5 to 0.7, 0.6 recommended. The R1 card says this prevents endless repetition and incoherent output. The January 2025 distills' generation_config.json already sets 0.6 with top_p 0.95; R1-0528-Qwen3-8B ships no such file. Send the temperature explicitly either way.
• No system prompt on the original R1 distills. The card asks for every instruction to sit in the user turn. R1-0528 and its Qwen3-8B distill changed this: their card says system prompts are now supported.
• Maths prompts. DeepSeek suggests asking the model to reason step by step and put the final answer in a boxed expression.
• Thinking can be skipped. The R1 card notes the original models sometimes bypass their reasoning, so DeepSeek advises starting every reply with the opening think tag followed by a newline. The 0528 generation no longer needs this, according to its card.

A chat front end such as Open WebUI can sit on top of the vLLM endpoint; set the temperature in the model's settings there rather than trusting the front end's default.

Keeping the endpoint private

vLLM's security guide warns that --api-key only protects paths under /v1, /v2, /inference and /cohere; endpoints such as /invocations, /pooling and /score answer without it. Treat the key as one layer, not the fence.

• Keep --host 127.0.0.1. Reach the API from your laptop with ssh -N -L 8000:127.0.0.1:8000 you@your-server, logging in as your own user.
• If applications elsewhere must call it, front it with a TLS-terminating reverse proxy that lets through only the /v1 paths you use.
• In Docker, publish as -p 127.0.0.1:8000:8000. A bare -p 8000:8000 listens on every interface, IPv6 included, and Docker's rules bypass ufw.
• Keep ~/.vllm-key at mode 600, and rotate it by restarting with a new value.

What a whole GPU server gives the model

PCIe passthrough puts the card under your own kernel: no hypervisor, no vGPU profile, no MIG slice and no scheduler above you. You install and pin the driver and CUDA release that vLLM was tested against, and nothing upgrades it under you. The machine is single-tenant with every core, all the memory, root, and its iLO or iDRAC console on a separate management network, so a driver change that drops the network can still be undone from the screen.

The port is unmetered. Pulling the 141 GB fp16 70B distill, or trying three quantizations of it, costs nothing in transfer. Servers are rented by the month with no contract, in New York, Miami, San Francisco, Amsterdam and Bucharest, on our own network, AS19624.

When one server is the wrong answer

If the model you need is the full R1, V3.2 or V4, a single-card server cannot hold it at any precision Ollama or DeepSeek publish; plan a multi-GPU build. If you want R1-class answers for a handful of requests a day with no data-handling constraint, DeepSeek's hosted API is less work than a machine. And a 1.5B or 7B distill for light internal use can run on CPU, which LLM hosting prices as a complete machine.