Ollama server hosting, sized from the model down

What a model needs in memory

When a model's weights fit on one GPU, Ollama loads them there; when they do not, it spreads them across the GPUs in the machine. The weights weigh the parameter count times the bytes per parameter: two bytes at 16-bit, a little over one at 8-bit and a little over half a byte at 4-bit, as the table shows. It lists the files Ollama actually downloads for four common models; the 4-bit column is each model's default tag.

Those figures are the floor, not the requirement. Every token in the context window is held in memory as well, and Ollama picks its default window from the GPU memory it finds: 4k tokens under 24 GiB, 32k from 24 to 48 GiB, 256k at 48 GiB and above. OLLAMA_CONTEXT_LENGTH overrides it. Parallel requests multiply it: the documentation's own example turns a 2K context with 4 parallel requests into an 8K context, with the memory to match.

A model whose weights only just fit leaves no room for its context. Ollama then loads part of it into system memory, ollama ps shows a split such as 48%/52% CPU/GPU instead of 100% GPU, and every answer slows down. Size for the biggest model at the precision you will load it, then leave room for the context. The GPU builds have 16 GB of system memory, so there is little room to spill into: set the context so ollama ps reads 100% GPU.

CPU or GPU for Ollama

Ollama runs without a graphics card, from system memory on the processor. A quantized 7-8B model answering one person or a small team is a comfortable CPU job, and it is the job the CPU inference build on the LLM hosting page was specified for: an AMD Ryzen AI 9 HX 370 with 128 GB of DDR5 that serves quantized models up to 14B. What a processor will not do is serve a 70B model to many users at once. That takes card memory.

On the card side, Ollama supports NVIDIA GPUs of compute capability 5.0 and newer with driver 550 or later; cards of compute capability 5.0 to 6.2 need driver 570. AMD cards run through ROCm 7, and the Instinct MI210 is on Ollama's list. Every card we fit, with its memory, is on the GPU servers page. The Ryzen AI machines have a page of their own.

The builds on the LLM hosting page

• CPU inference. Ryzen AI 9 HX 370, 12 cores, 128 GB DDR5, 1 TB NVMe. Quantized 7-14B models from system memory. Built to order in 4-24 hours, in all five cities.
• GPU serving, 48 GB. An NVIDIA L40S with 48 GB of ECC GDDR6 in a Xeon Silver 4110 chassis with 16 GB of DDR4 and two 240 GB SATA SSDs in RAID 1: 13B models at speed, and a 70B at 4-bit once the context window is set short enough to leave the model on the card.
• GPU serving, 80 GB. The same chassis with an A100 80GB: a 70B at 8-bit, again with a modest context window.

Each build is priced complete on the LLM hosting page, card and machine together. More than one card's worth of weights, a 70B at 16-bit for one, is a multi-GPU build, quoted with the parts named.

Why a whole machine suits a model server

A model server holds its weights in memory around the clock and answers whenever a client calls. That wants hardware nobody else touches. Each machine here has one tenant: no hypervisor underneath, no shared card, no time-slicing, so the memory you sized for is the memory you get.

Prompts, retrieved documents and generated tokens travel between your users and your machine, with no third party in the path to log them or train on them. Amsterdam and Bucharest keep the data inside the EU; New York, Miami and San Francisco keep it in the US.

Nothing counts the traffic on the port, in either direction, so pulling a model again after a rebuild or serving a busy team adds no line to the invoice. On the CPU and GPU builds above you get root, a clean OS of your choice and an out-of-band console (iLO, iDRAC or IPMI) that answers when the OS does not. The machines and the network are ours, announced by our own autonomous system, AS19624, which anyone can look up.

Setting up Ollama on Linux

1. Order the machine with Ubuntu or another Linux. On a GPU build, put the driver version you want on the order, or install NVIDIA's driver and CUDA yourself and confirm the card with nvidia-smi.
2. Run the install script from Ollama's Linux documentation. It creates an ollama user and a systemd service that runs ollama serve and restarts it if it stops.
3. Pull a model and ask it something from the shell.
4. While the model is loaded, run ollama ps. On a GPU build the PROCESSOR column should read 100% GPU, and CONTEXT shows the window the model was given.

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl status ollama
ollama pull llama3.1:8b
ollama run llama3.1:8b "Reply with one word: ready?"
ollama ps

To install the exact release your code was tested against, set OLLAMA_VERSION when you run the script; the documentation's example is curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.5.7 sh, with your version in place of 0.5.7. The service logs to the journal: journalctl -e -u ollama.

Keeping port 11434 private

Ollama binds 127.0.0.1, port 11434, so nothing outside the machine can reach it until you change that. Keep it that way if you can. Its OpenAI-compatible API needs a key in the client, but Ollama ignores the key, so anyone who can reach the port can run your models.

From your own computer, the simplest route is an SSH tunnel: the server's port appears on your laptop's localhost and the traffic rides your SSH login, encrypted. Set up SSH keys first. Any OpenAI client then works against http://localhost:11434/v1/ with any key string.

# on your laptop: the server's port 11434 on your own localhost
ssh -N -L 11434:127.0.0.1:11434 you@your-server

When application servers elsewhere must call it, let the firewall admit only those servers first, then bind Ollama to every address with a systemd override. The firewall lines below are Ubuntu's ufw, and the address is an example; use your own. The firewall decides who may connect, not who can read: Ollama speaks plain HTTP, so this traffic crosses the network unencrypted. When the prompts matter, carry it over an SSH or WireGuard tunnel, or put a reverse proxy that terminates TLS in front.

sudo ufw allow 22/tcp
sudo ufw allow from 203.0.113.10 to any port 11434
sudo ufw enable
sudo ufw status

sudo systemctl edit ollama.service
# add these lines, then save:
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

sudo systemctl daemon-reload
sudo systemctl restart ollama

Allow SSH before you enable the firewall. If you forget, the session you are in normally stays up, so add the rule from it before you log out. Otherwise the next login is refused, and the out-of-band console on these builds, which answers when the OS does not, still reaches the machine. For a public endpoint, keep Ollama on 127.0.0.1 behind a reverse proxy that checks credentials and serves TLS.

Running it day to day

• Keep the model loaded. Ollama unloads a model 5 minutes after its last request, and the next caller waits while it loads again. A second line in the same override, Environment="OLLAMA_KEEP_ALIVE=-1", keeps it in memory; a duration such as 24h works too.
• Choose the concurrency. OLLAMA_NUM_PARALLEL defaults to 1, so requests wait in a queue that holds 512 by default (OLLAMA_MAX_QUEUE). Raising it serves more people at once and multiplies the context memory by the same factor.
• Put the models on the disk you want. They live in /usr/share/ollama/.ollama/models. A line Environment="OLLAMA_MODELS=/data/ollama" in the same override points Ollama at that directory from then on. Models you have already pulled stay in the old directory, so move them across or pull them again. Create the directory and give it to the ollama user first: sudo mkdir -p /data/ollama && sudo chown -R ollama:ollama /data/ollama.
• Update on purpose. Running the install script again moves Ollama to the latest release, and with OLLAMA_VERSION to the release you name. Nothing on our side changes your versions.
• Back up what a pull cannot restore. Models come back with ollama pull; your Modelfiles, system prompts and application data do not, so copy those off the machine.

When this is the wrong place to run it

A per-token API is the better tool when your traffic arrives in bursts or when the job needs a frontier model's quality. A machine billed by the month wins when the volume is steady, when the data has to stay in your custody, or when an open model already does the work. Plenty of setups mix the two: the hardest requests go to an API and the rest stay on the machine.

One machine also has a ceiling. A 70B model at full precision is a 141 GB download, more than any single card we fit holds. And when many users hit one model at once, vLLM, which batches requests on the card, is the usual next step on the same hardware.