FFmpeg GPU transcoding on a dedicated server: NVENC, Quick Sync and a batch queue

What the hardware path trades

A software encode with libx264 or libx265 spends CPU cores on every frame and gives the best quality per bit. A hardware encode moves that work to a fixed-function block. The FFmpeg wiki is blunt about the result: hardware encoders "typically generate output of significantly lower quality than good software encoders like x264", so they need a higher bitrate for the same perceived quality, in exchange for speed and a nearly idle CPU.

For a VOD library the trade usually lands on hardware: the job is throughput across thousands of files, and a higher bitrate per rendition usually costs less than the CPU cores a software encode would need. Masters you will keep for years are the exception; encode those in software.

Which cards carry an encoder

NVIDIA publishes the encoder count and the session ceiling per board in its Video Encode and Decode Support Matrix. The figures below were read from it on 22 October 2026, for cards on our GPU dedicated server list. The last column is the matrix's own "Max # of concurrent sessions" column.

The A100 and H100 have no video encoder. They decode, but h264_nvenc and the other NVENC encoders will not run on them; any encode has to use a software encoder on the CPU. Do not rent one for transcoding.

Sessions and engines are different limits. A GeForce card stops at 12 simultaneous encodes whatever its engine count; a card with more engines finishes each encode sooner. For a batch queue that runs a handful of jobs at once, the GeForce ceiling rarely binds. For many simultaneous renditions it does, and the Quadro, Tesla, L4 and L40S rows are the way past it.

Setting up the NVIDIA path

1. Install the operating system and the NVIDIA driver you want to pin. The card is passed straight through to your OS, so the driver version is yours to choose and nothing upgrades it under you.
2. Check that your FFmpeg build has the CUDA hwaccel and the NVENC encoders. A distribution package may or may not; NVIDIA's guide builds FFmpeg from source against nv-codec-headers when it does not.
3. Note which Video Codec SDK your FFmpeg build expects and install at least the driver that SDK version requires, as NVIDIA's guide recommends.
4. Run one file end to end on the GPU before scripting anything.

# what this build can use
ffmpeg -hide_banner -hwaccels
ffmpeg -hide_banner -encoders | grep nvenc

# one file, decode + encode on the GPU, frames never leave card memory
ffmpeg -n -hwaccel cuda -hwaccel_output_format cuda -i input.mp4 \
  -sn -vf scale_cuda=w=iw:h=ih:format=yuv420p -c:a copy -c:v h264_nvenc -preset p6 -tune hq \
  -b:v 5M -maxrate 10M -bufsize 5M -bf 3 -rc-lookahead 20 output.mp4

# 1080p and 720p renditions from one decode, scaled on the GPU
ffmpeg -n -hwaccel cuda -hwaccel_output_format cuda -i input.mp4 \
  -map 0:v -map 0:a? -sn -vf scale_cuda=1920:1080:format=yuv420p -c:v h264_nvenc -preset p6 -b:v 5M -c:a copy out_1080.mp4 \
  -map 0:v -map 0:a? -sn -vf scale_cuda=1280:720:format=yuv420p -c:v h264_nvenc -preset p6 -b:v 3M -c:a copy out_720.mp4

The settings follow NVIDIA's latency-tolerant, high-quality profile: B-frames and lookahead are allowed because nobody is watching live. NVIDIA suggests 10 to 20 lookahead frames, and at least the B-frame count plus one. Swap h264_nvenc for hevc_nvenc, or for av1_nvenc on a card whose matrix row says AV1. -n makes FFmpeg refuse to overwrite an existing output instead of asking.

Quick Sync instead of a card

Quick Sync lives in the graphics on the processor, so it needs no add-in card. On our Intel processor list only five parts have it: Xeon E3-1284L v3, E3-1284L v4, E3-1585L v5, Xeon E-2286M and Core i5-9500. No Xeon E5, Xeon D, Atom or Xeon Scalable processor has it. Our GPUs go in Xeon E5, Xeon Scalable and AMD EPYC machines, so none of them has Quick Sync. Pick the path first, then the machine.

The five do not encode the same formats. The E3-1284L v3 is Haswell, older than iHD supports: on it use the legacy i965 driver (i965-va-driver) and the VAAPI path, H.264 only. The E3-1284L v4 runs on Intel's iHD driver, which starts at Broadwell, and also encodes H.264 only. The E3-1585L v5 is Skylake and adds 8-bit HEVC; the E-2286M and Core i5-9500 are Coffee Lake and add 10-bit HEVC as well, in line with the FFmpeg wiki's generation table. For iHD on Ubuntu and Debian install the non-free package, Intel's full-feature build; Intel's own table shows several encoders on these older generations only in that build.

sudo apt install intel-media-va-driver-non-free   # E3-1284L v3: i965-va-driver instead

# VAAPI: hardware decode, scale to 720p, H.264 encode at 5 Mbit/s
ffmpeg -n -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi \
  -i input.mp4 -sn -vf scale_vaapi=w=1280:h=720:format=nv12 -c:v h264_vaapi -b:v 5M -c:a copy out_720.mp4

# libvpl/libmfx: H.264 in, constant-quality (ICQ) H.264 out
ffmpeg -n -hwaccel qsv -c:v h264_qsv -i input.mp4 \
  -c:v h264_qsv -global_quality 25 -c:a copy output.mp4

Running a library as a batch queue

NVIDIA's guide says a single FFmpeg process does not saturate the encoder: start-up time is hidden only when several sessions run in parallel, and its own measurement example runs four at once. It also advises fewer, larger processes when chunks are short, since each launch carries initialisation cost. A queue that runs a fixed number of jobs side by side follows both rules.

#!/bin/sh
# /usr/local/bin/vod-batch: every .mov in /srv/in -> H.264 MP4 in /srv/out, 4 at a time
find /srv/in -maxdepth 1 -name '*.mov' -print0 |
  xargs -0 -P 4 -I{} sh -c '
    in="${1}"; out="/srv/out/$(basename "$in" .mov).mp4"
    [ -e "$out" ] && exit 0
    ffmpeg -nostdin -hide_banner -loglevel error -y \
      -hwaccel cuda -hwaccel_output_format cuda -i "$in" -sn \
      -vf scale_cuda=w=iw:h=ih:format=yuv420p -c:v h264_nvenc -preset p6 -b:v 5M -c:a aac -b:a 128k \
      "$out.part.mp4" && mv "$out.part.mp4" "$out"
  ' _ {}

• Each job writes to a .part.mp4 name and is renamed only on success, so a crash never leaves a half file that looks finished, and a rerun skips what is done.
• -nostdin stops FFmpeg reading the terminal, which the FFmpeg manual calls useful when it runs in the background. Start it from cron or a systemd timer as flock -n /tmp/vod-batch.lock /usr/local/bin/vod-batch, so a slow run is never joined by a second copy.
• On a GeForce card keep -P times renditions per job at 12 or fewer. Each rendition is a session.
• Keep sources and outputs on local disks. NVIDIA notes that raw input from disk can become the bottleneck and recommends SSD for it.

Why a whole machine for it

Batch transcoding is long, steady and heavy on transfer. On a GPU dedicated server the card sits in a PCIe slot of a machine rented to one account and goes straight to the OS you install: no vGPU slice, no MIG partition, no scheduler above you, so a queue that runs all night gets the same encoder at 4 a.m. as at noon. All cores, the RAM, the drives and root are yours, and the iLO or iDRAC console on a separate management network gets you back in if a driver change takes the network down.

The port is unmetered in both directions and inbound transfer is not billed, so uploading a source library and shipping renditions to a CDN add nothing to the invoice. The GPU is priced as its own line beside the machine, so moving from a GeForce card to a data-centre card changes the bill by the difference between the two. With the card already racked in a machine, delivery is about 30 minutes from order approval; a card fitted to order takes 4 to 24 hours.

When this is the wrong tool

• Live channels, IPTV and adaptive bitrate in real time belong on a live GPU transcoding server running a streaming engine, not a file queue.
• Taking an RTMP feed in and packaging HLS is an ingest problem; see the nginx RTMP server walkthrough.
• If the files are for your own household viewing, Jellyfin transcodes on demand as people press play; the Jellyfin server page covers that.
• If you only need lessons played in a browser, start with course video hosting, which may not need a GPU at all.
• If quality per bit matters more than hours, encode in software with libx265 on a many-core machine.