Triton Inference Server on bare metal: many models, one machine, every card yours

What Triton does that an LLM engine does not

Triton is a general model server. It reads a model repository, a directory with one folder per model, numbered version subfolders and an optional config.pbtxt each, and runs every model through a backend: TensorRT, ONNX Runtime, PyTorch, OpenVINO, Python, RAPIDS FIL and others (README.md). Clients talk to it over HTTP/REST or gRPC, with protocols based on the community KServe v2 protocol. It earns its keep when one box must serve an image classifier, an embedding model, a detector and a little Python pre-processing at once, or chain them as an ensemble.

If the whole job is one chat model, a dedicated LLM engine is less to operate: see vLLM on a dedicated server or llama.cpp's own server. NGC does publish a -vllm-python-py3 Triton image, worth it only when the LLM must sit beside other models.

Which GPUs the current containers run on

NVIDIA's release notes for the 26.10 container list CUDA compute capability 7.5 and later: the Turing, Ampere, Hopper, Ada Lovelace and Blackwell families. The 26.09 image is built on CUDA 13.4.1 according to the frameworks support matrix, so the host driver also has to be recent enough for that CUDA release. An older card does not get a slower Triton from a current tag; it gets none.

Check before you order a second card or move models: nvidia-smi --query-gpu=name,compute_cap --format=csv prints each card's compute capability. A TensorRT plan is tied to the capability it was built for (model_repository.md), so a different card means rebuilt plans, or one plan per capability listed in cc_model_filenames.

Sizing the rest of the machine

Card memory decides how many models load; the rest of the box decides how fast requests reach them. In the default control mode Triton loads every model in the repository at start-up (model_management.md), so the card has to hold all of them at once.

Why a whole machine suits a model server

On our GPU dedicated servers nothing sits between your kernel and the card: no hypervisor, no vGPU profile, no MIG partition. You install the driver and pin the CUDA release, and nothing upgrades it under you. That matters here, because each monthly Triton tag carries a driver floor and you decide when to cross it. Every core, all the memory, both disks and root are yours, and the server's iLO or iDRAC console sits on a separate management network, so a driver change that kills the network still leaves you the screen, virtual media and power.

The port is unmetered in both directions, so pulling the image and model files in costs nothing per byte. The card-taking platforms run from Xeon E5-2600 v1/v2, E5-2600 v3/v4 and Xeon Silver/Gold on PCIe 3.0 to AMD EPYC on PCIe 4.0 with 128 lanes per socket; a chassis carrying a second card or more is quoted by hand on multi-GPU builds. Ready machines are handed over in about 30 minutes once payment is approved; a build to order, such as a card fitted for you, takes 4 to 24 hours. Five cities are listed on our data centre page.

Setting it up

1. Install Ubuntu 24.04, then the NVIDIA driver from Ubuntu's own packages (NVIDIA's container guide recommends the distribution package manager) and Docker Engine from Docker's apt repository. Reboot after the driver.
2. As a sudo user, run the first block. It turns ufw on with SSH only, refuses to go on unless every card reports 7.5 or later, installs the NVIDIA Container Toolkit and proves a container can see the GPUs. We install the toolkit's current release instead of the exact version the guide pins.
3. Only if the first block printed TOOLKIT OK, run the second block. It writes /srv/triton/run.sh, builds a one-model repository from the official densenet example and starts Triton as the image's non-root triton-server user (uid 1000), with the repository mounted read-only and all three listeners on 127.0.0.1.
4. Test a request with the SDK image: sudo docker run --rm --net host nvcr.io/nvidia/tritonserver:26.09-py3-sdk /workspace/install/bin/image_client -m densenet_onnx -c 3 -s INCEPTION /workspace/images/mug.jpg should rank COFFEE MUG first, as the README shows.

set -o pipefail   # stays on in this shell: a failed download stops the chain
sudo sed -i 's/^IPV6=.*/IPV6=yes/' /etc/default/ufw
sudo ufw allow OpenSSH && sudo ufw --force enable
cc=$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader | sort -V | head -1) \
  && [ "$(printf '%s\n' 7.5 "$cc" | sort -V | head -1)" = 7.5 ] && docker --version \
  && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
     | sudo gpg --batch --yes --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
     | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
     | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list >/dev/null \
  && sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit \
  && sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker \
  && sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi \
  && echo "TOOLKIT OK" \
  || echo "STOP - no card at 7.5 or later, no Docker, or a step above did not finish"

sudo install -d -m 755 /srv/triton/models/densenet_onnx/1
sudo tee /srv/triton/run.sh >/dev/null <<'EOF'
TAG=26.09
docker run -d --name triton --restart unless-stopped --gpus all --net host \
  --shm-size=1g --user triton-server -v /srv/triton/models:/models:ro \
  nvcr.io/nvidia/tritonserver:$TAG-py3 tritonserver --model-repository=/models \
  --model-control-mode=none --http-address=127.0.0.1 \
  --grpc-address=127.0.0.1 --metrics-address=127.0.0.1
EOF
B=https://raw.githubusercontent.com/triton-inference-server/server/r26.09/docs/examples/model_repository/densenet_onnx
M=/srv/triton/models/densenet_onnx
sudo wget -qO $M/config.pbtxt $B/config.pbtxt && sudo wget -qO $M/densenet_labels.txt $B/densenet_labels.txt \
  && sudo wget -qO $M/1/model.onnx https://github.com/onnx/models/raw/main/validated/vision/classification/densenet-121/model/densenet-7.onnx \
  && sudo bash /srv/triton/run.sh \
  && for i in $(seq 60); do curl -fs 127.0.0.1:8000/v2/health/ready && break; sleep 5; done \
  && curl -fs 127.0.0.1:8000/v2/health/ready && echo READY \
  || { echo "STOP - Triton is not ready; its last log lines follow"; sudo docker logs --tail 30 triton; }

Host networking means Docker publishes nothing, so ufw filters these ports like any other process; with the 127.0.0.1 bindings that is two fences. The wget lines overwrite and are safe to repeat. A second run refuses to create another container named triton: after a failed start, read the log, fix the cause, run sudo docker rm -f triton, then paste the block again. Pasting it while Triton is already running stops at the container name and prints STOP; that is expected, and curl -fs 127.0.0.1:8000/v2/health/ready && echo up confirms the running copy.

Batching and spreading instances across cards

Dynamic batching merges separate requests into one batch on the server, which usually raises throughput. It only works when the model's first dimension is the batch and max_batch_size is 1 or more; the densenet example is set to 0, so it never batches. For your own ONNX model, a config like the one below (ONNX inputs and outputs are filled in automatically) puts two instances on GPU 0 and one on GPU 1 and lets requests wait up to 100 microseconds for company. The values come from the docs' examples; tune them with your own traffic.

# /srv/triton/models/my_model/config.pbtxt, next to 1/model.onnx
platform : "onnxruntime_onnx"
max_batch_size : 16
dynamic_batching {
  max_queue_delay_microseconds : 100
}
instance_group [
  {
    count : 2
    kind : KIND_GPU
    gpus : [ 0 ]
  },
  {
    count : 1
    kind : KIND_GPU
    gpus : [ 1 ]
  }
]

The default control mode ignores repository changes while running, so load the new model with sudo docker restart triton. Then measure from the SDK image with sudo docker run --rm --net host nvcr.io/nvidia/tritonserver:26.09-py3-sdk perf_analyzer -m my_model --concurrency-range 1:8 and watch the metrics: dividing the inference count by the execution count gives the average batch size Triton actually formed (metrics.md). Instance groups place whole copies of a model; they do not split one model across cards.

Locking it down and watching it

• No login of its own. NVIDIA's secure deployment guide expects Triton behind a gateway or proxy that handles authorization and encryption. Reach the loopback ports from your workstation with ssh -N -L 8000:127.0.0.1:8000 -L 8001:127.0.0.1:8001 -L 8002:127.0.0.1:8002 you@server.
• Application servers elsewhere go through an authenticating proxy on this host (TLS with client certificates or a token), opened in ufw to their addresses only. The alternative is Triton's own mutual TLS on gRPC: in run.sh, change --grpc-address to the server's address and add --grpc-use-ssl-mutual=true with --grpc-server-cert, --grpc-server-key and --grpc-root-cert, then open 8001 in ufw only to the application servers' addresses. Put the certificates in /srv/triton/tls. Create the folder (sudo install -d -m 755 /srv/triton/tls), then an empty key file, sudo install -m 600 -o 1000 -g 1000 /dev/null /srv/triton/tls/server.key, before writing the key into it. Mount the folder with -v /srv/triton/tls:/tls:ro in run.sh, and point the three flags at /tls/... paths. Your backup archive then holds the private key, so store it as a secret.
• Keep model control at none. The guide warns that loading models through the API or by polling the repository can lead to arbitrary code execution, and that backends such as Python run code from the repository with no sandbox. Keep /srv/triton/models root-owned and mounted read-only.
• Metrics are Prometheus text at 127.0.0.1:8002/metrics, pull only. Watch nv_gpu_utilization, nv_gpu_memory_used_bytes and nv_inference_queue_duration_us, scraped by a Prometheus on the same box or over the tunnel.

Updates, backups and honest limits

• Updates. Tags are named YY.MM. Use a tag only once NGC lists it. Read its release notes for CUDA and driver needs, then run NEW=YY.MM; sudo docker pull nvcr.io/nvidia/tritonserver:$NEW-py3 && sudo sed -i "s/^TAG=.*/TAG=$NEW/" /srv/triton/run.sh && sudo docker rm -f triton && sudo bash /srv/triton/run.sh, with YY.MM replaced by the new tag. Rolling back is the same line with the old tag, which stays on disk until you prune it.
• Start-up errors stop everything. With the default --exit-on-error=true, a failure during initialization exits the server, so one broken config.pbtxt takes every model down until it is fixed. Read sudo docker logs triton after each change.
• Backups. The only state is /srv/triton: the repository and run.sh. Archive it with sudo tar -czf /root/triton-$(date +%F-%H%M%S).tgz -C /srv triton, check it with sudo tar -tzf on that file, and copy it off the machine yourself.
• Restore, as a sudo user. Set f to the archive's path, then run `sudo test -f "$f" && [ ! -e /srv/triton.before-restore ] && [ ! -e /srv/triton.failed-restore ] && sudo tar -tzf "$f" >/dev/null && { sudo docker rm -f triton; true; } && sudo mv /srv/triton /srv/triton.before-restore && sudo test -f /srv/triton.before-restore/run.sh && sudo tar -xzf "$f" -C /srv && sudo bash /srv/triton/run.sh || echo "STOP - restore halted at the step that failed". It refuses if a set-aside folder from an earlier restore still exists. Then wait until curl -fs 127.0.0.1:8000/v2/health/ready && echo READY` prints READY.
• Rollback if it never does: [ -d /srv/triton.before-restore ] && [ ! -e /srv/triton.failed-restore ] && { sudo docker rm -f triton; [ ! -e /srv/triton ] || sudo mv /srv/triton /srv/triton.failed-restore; } && sudo mv /srv/triton.before-restore /srv/triton && sudo bash /srv/triton/run.sh. If the restore stopped before the move, the old tree is untouched: sudo bash /srv/triton/run.sh brings it back. Once the restored copy serves, sudo rm -rf /srv/triton.before-restore. After a rollback, read what you need from /srv/triton.failed-restore and then remove it. Either folder left in place makes the next restore stop.
• Limits. One box, no autoscaling; for a fleet the README points to Kubernetes and Helm examples. A single model larger than 96 GB of card memory needs sharding across cards and a quoted multi-GPU build.
• Licence. The server's source is BSD-3-Clause (LICENSE): keep the copyright notice and licence text when you pass it on; there is no duty to publish source. The NGC image bundles other components under their own terms. Our general reading, not legal advice.