A self-hosted Whisper server on a whole GPU

VRAM per model

VRAM decides which model loads. OpenAI's README gives these approximate figures for its reference implementation, with speed relative to large:

The reference code is not the efficient way to serve it. faster-whisper re-implements Whisper on CTranslate2; its own benchmark, 13 minutes of audio with large-v2 on an 8 GB RTX 3070 Ti, used 4,525 MB in fp16 and 2,926 MB in int8, against 4,708 MB for the reference code in fp16. Batching (batch_size=8) cut the run from about a minute to 16-17 seconds and raised memory to 4.5-6.1 GB.

turbo is an optimized version of large-v3 for fast transcription. The README states it is not trained for translation; keep a multilingual model such as large for translating speech into English.

GPU or CPU

A CPU runs the small models acceptably and the large ones slowly. In the same benchmark, the small model running with 8 threads on a Core i7-12700K took 1m42s for 13 minutes of audio with faster-whisper in int8; large-v2 on the GPU took 59 seconds in int8. Throughput on a different card or processor differs, so time a sample of your own audio before you commit to a model size.

For a queue of recordings, a meeting archive or subtitles for a video library, the GPU is what keeps the backlog short. For a few minutes of audio a day, a CPU-only machine with small or base is enough.

The machine

The RTX 3050 dedicated server is a whole Intel Core i5-9500 desktop with an NVIDIA GeForce RTX 3050 6GB in a PCIe slot, 32 GB of DDR4 and a 256 GB NVMe SSD, on an unmetered 1 Gbps port in Bucharest. The card is passed straight to your operating system: no vGPU, no time-slicing, nobody else on its 6 GB of GDDR6.

• Fits: large-v3 in int8 or fp16 through faster-whisper; medium and smaller in any implementation.
• Tight: turbo in the reference code, at about 6 GB. Uncertain: large in the reference code. OpenAI's table says about 10 GB, while faster-whisper's benchmark measured it at 4.7 GB for large-v2 in fp16; faster-whisper is the safer route for large-v3.
• Shares badly: an 8B LLM quantised to 4 bits is 4.92 GB of weights, so Whisper and an LLM server on one 6 GB card compete for memory. Run one, or move up to a bigger card from the GPU server list.

Billing is monthly, not per minute of audio, and the port does not count traffic, so uploading hours of recordings costs nothing extra. The machine is online about 30 minutes after the order is approved.

Installing faster-whisper on Ubuntu

1. Install the NVIDIA driver from your distribution and confirm the card with nvidia-smi.
2. Create a Python virtual environment and install faster-whisper. GPU execution needs cuBLAS for CUDA 12 and cuDNN 9; on Linux the README installs both as pip packages.
3. Export LD_LIBRARY_PATH to the libraries' directories before starting Python, as the README requires.
4. Load the model with device="cuda" and compute_type="int8_float16" or "float16", and transcribe. Segments are a generator: work starts when you iterate.

sudo apt install -y python3-venv
python3 -m venv ~/whisper && . ~/whisper/bin/activate
pip install faster-whisper
pip install nvidia-cublas-cu12 nvidia-cudnn-cu12==9.*
export LD_LIBRARY_PATH=$(python3 -c 'import os, nvidia.cublas, nvidia.cudnn; print(os.path.dirname(nvidia.cublas.__path__[0]) + "/cublas/lib:" + os.path.dirname(nvidia.cudnn.__path__[0]) + "/cudnn/lib")')

from faster_whisper import WhisperModel

model = WhisperModel("large-v3", device="cuda", compute_type="int8_float16")
segments, info = model.transcribe("meeting.mp3", vad_filter=True)
print(info.language, info.language_probability)
for s in segments:
    print("[%.2fs -> %.2fs] %s" % (s.start, s.end, s.text))

vad_filter=True runs Silero VAD first and drops silence longer than two seconds by default, which saves GPU time on recordings with long pauses. The README also documents a Docker route on the nvidia/cuda:12.3.2-cudnn9-runtime-ubuntu22.04 image.

Putting an API in front

faster-whisper is a library, not a server. Put an OpenAI-compatible endpoint in front of it, so existing OpenAI SDK and CLI clients only need a new base URL: speaches, listed in the faster-whisper README, is one such server and deploys with Docker. Or write a small HTTP service that accepts an upload and returns text or SRT. Either way, keep the model loaded between requests and run one job at a time per card unless you have measured batching.

• Bind the service to localhost and publish it through a reverse proxy with TLS.
• Require an API key; an open transcription endpoint is free GPU time for anyone who finds it.
• Put uploads in a queue with a size limit, so one long file cannot starve the rest.
• Delete audio after transcription unless you have a reason to keep it.

Operating it

Pin the faster-whisper and ctranslate2 versions: the README notes that the latest ctranslate2 supports only CUDA 12 and cuDNN 9, and older stacks need older releases. The machine has one NVMe drive and no RAID, so keep transcripts somewhere else as well, or add the nightly backup offered on the order form.

These desktops have no IPMI or iLO. Power-on is remote; if the system hangs, staff in Bucharest reset it. Run the service under systemd so it starts again on boot, and watch nvidia-smi for memory headroom when you change models.

When this is the wrong machine

Training is out of scope: 6 GB is for inference, encoding and development, not for fine-tuning a model. Real-time captioning for listeners on another continent is better served closer to them; these machines are only in Bucharest, and other cities have GPU servers on the full list. For an occasional file, a laptop running the small model is enough and costs nothing.