SGLang server on bare metal: install, launch arguments and a locked-down port

What SGLang needs from the machine

The installation page targets Linux with NVIDIA GPUs and a CUDA 13-compatible driver. CUDA 12 images (-cu12 / -cu129) are no longer published; lmsysorg/sglang:v0.5.19-cu129 is the last CUDA 12 tag. The pip route wants Python 3.11 or newer. The docs do not print one minimum compute capability for the whole engine. What they do state: FlashInfer, the default attention kernel backend, "only supports sm75 and above", and the fallback on trouble is --attention-backend triton --sampling-backend pytorch.

The README's NVIDIA row of representative hardware starts at Ampere: A100, RTX 30/40/50, RTX 6000 Ada and PRO 6000, H100 and newer. Our reading, not the docs': treat Ampere and newer as the target (on our list the A100, RTX 30/40/50, the RTX A-series, L4, L40S, H100 and the RTX PRO 6000 Blackwell), and Turing cards (T4, Quadro RTX, RTX 20) as best-effort: they clear FlashInfer's sm75 floor but are not on that list. NVIDIA's own compute-capability table starts at 7.5 (Turing) and moves Volta, Pascal and Maxwell to a legacy list, so a Tesla P40 or V100 is the wrong card for this engine even though its memory looks attractive.

AMD: the SGLang README names Instinct MI300X, MI325X, MI350X and MI355X as representative hardware, and the quantization page's AMD column is headed MI300X, MI325X and MI350X. The MI50 and MI210 we fit are not on those lists; test before you commit a workload to them.

Card memory and --mem-fraction-static

Weights take parameters times bytes per parameter. The arithmetic on our GPU server catalogue gives these floors, before any cache:

SGLang then claims a static pool at startup for the weights plus the KV cache. --mem-fraction-static sets that share of card memory; unset, it is computed from detected memory minus a reserve. nvidia-smi showing a nearly full card is therefore normal. The FAQ's order for out-of-memory errors: lower --chunked-prefill-size to 4096 or 2048 if it fails during prefill, lower --max-running-requests if it fails during decode, or drop --mem-fraction-static to 0.8 or 0.7, which costs peak concurrency.

Quantised checkpoints shrink the weights. The docs recommend offline quantization (a pre-quantized AWQ, GPTQ-Marlin or FP8 checkpoint) over online, and warn not to add --quantization when loading one. Their platform table marks fp8, awq, gptq_marlin, awq_marlin and gguf as supported on NVIDIA GPUs; the ModelOpt FP8 path needs Hopper (SM90) or newer.

Install and first launch

Install the NVIDIA driver for your card first and confirm it with nvidia-smi. Then the uv route, adapted from the install and quickstart pages (only the model, host and prompt are changed):

uv venv --python 3.12
source .venv/bin/activate
uv pip install --prerelease=allow sglang

# default bind is 127.0.0.1; say so explicitly
sglang serve Qwen/Qwen3-0.6B --host 127.0.0.1 --port 30000

# second shell, same host
curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-0.6B", "messages": [{"role": "user", "content": "ping"}]}'

Wait for The server is fired up and ready to roll! before the curl. If uv is older than 0.12.0, keep --prerelease=allow: without it, the docs say, uv silently installs SGLang 0.5.9. A CUDA_HOME error means the toolkit path is unset; export it, typically /usr/local/cuda-VERSION.

The Docker route and the ufw trap

The lmsysorg/sglang image carries Python, SGLang and CUDA 13. It needs Docker Engine and the NVIDIA Container Toolkit, configured with sudo nvidia-ctk runtime configure --runtime=docker and a Docker restart. The docs' example publishes -p 30000:30000, which listens on every interface. Docker's own documentation states that published ports are routed in the nat table before ufw's INPUT chain sees them, so a ufw deny rule does not cover them. Publish on loopback instead:

HF_CACHE_DIR=/srv/hf-cache
docker run -d --name sglang --gpus all --ipc=host --shm-size 32g \
  -p 127.0.0.1:30000:30000 \
  -v "$HF_CACHE_DIR:/root/.cache/huggingface" \
  lmsysorg/sglang:latest \
  sglang serve Qwen/Qwen3-0.6B --host 0.0.0.0 --port 30000

Inside the container the server must bind 0.0.0.0 or Docker cannot reach it; the 127.0.0.1: prefix on -p is what keeps it off the public interface. That holds on Docker Engine 28 or newer; Docker's docs note that older releases let hosts on the same L2 segment reach localhost-published ports, so check docker version. It covers IPv4 loopback only; [::1] is the IPv6 equivalent. Pin a release tag rather than latest once it works: the docs advise a specific release or dated nightly for reproducible deployments.

API key, admin key and a config file

--api-key sets the server's key, which the OpenAI-compatible API also checks; clients send it as their OpenAI API key. --admin-api-key protects the admin/control endpoints, such as weight updates, cache flush and /server_info. A key typed on the command line lands in ps output and shell history, so put it in the YAML file --config reads, owned by the service user and chmod 600:

# /etc/sglang/config.yaml  (chmod 600)
model-path: meta-llama/Meta-Llama-3-8B-Instruct
host: 127.0.0.1
port: 30000
tensor-parallel-size: 2
api-key: replace-with-a-long-random-string
enable-metrics: true

# launch
python -m sglang.launch_server --config /etc/sglang/config.yaml

The docs' config example uses flag names without the leading dashes as keys but does not show api-key itself, so confirm on your version that a request without the key is refused. Publish it to the internet through a TLS reverse proxy on 443, not by opening 30000. If you serve a vision model to untrusted clients, set --allowed-media-domains; without it the docs say HTTP(S) media from any domain is fetched.

Several GPUs in one chassis

--tp 2 (long form --tensor-parallel-size) splits one model across two cards. If it fails with "peer access is not supported between these two devices", add --enable-p2p-check. --dp 2 instead runs whole replicas for throughput when each card holds the model; the docs recommend SGLang Model Gateway for that. Docker runs need a generous --shm-size for inter-process traffic.

The largest single GPU we fit holds 96 GB. Past that, two or more GPUs in one chassis are quoted by hand, and the PCIe lanes each platform gives a card are laid out on multi-GPU and multi-node servers.

SGLang next to vLLM, llama.cpp and Ollama

The SGLang docs publish no head-to-head benchmark against vLLM, and this page quotes none. What SGLang's own docs claim for itself is design, not a number: RadixAttention prefix caching, multi-GPU parallelism, OpenAI- and Anthropic-compatible endpoints, and an Ollama-compatible API so the Ollama CLI can talk to it. Its README also says SGLang reused design and code from vLLM, among other projects. Run your own model and prompt mix on both before deciding; our vLLM server page covers that engine's setup.

• SGLang: Hugging Face checkpoints on Ampere-or-newer NVIDIA cards (Turing best-effort), one or several, many concurrent clients.
• llama.cpp server: GGUF files, partial GPU offload, and machines without a large card.
• Ollama: one command per model and a model library, at the cost of fewer serving knobs.
• For a chat front end on top of any of them, Open WebUI speaks the OpenAI API these servers expose.

Why a whole machine, and which one

SGLang reserves the bulk of GPU memory at launch, so it wants the card to itself. Our GPU dedicated servers pass the whole card through to your OS: no vGPU, no MIG partition, no time-slicing. You install and pin the driver and CUDA release your SGLang version was built against, and nothing upgrades it under you. Every core, the RAM and the disks are yours, with root and the server's iLO or iDRAC console on a separate management network for when a driver change takes the network down. The port is unmetered, so pulling a 140 GB checkpoint or answering clients all day costs nothing extra.

Delivery: about 30 minutes from order approval when the machine is already in a rack, 4-24 hours for a build to order. Monthly term, no contract. For a 4-bit 70B the arithmetic above points at a 48 GB card; for 8-bit, 80 GB.