MLflow tracking server on an NVMe dedicated server

Three parts behind one URL

MLflow's self-hosting overview splits a deployment into the tracking server, a backend store for metadata and an artifact store for files, each replaceable on its own.

The tracking server page calls SQLite good for most use cases and suggests PostgreSQL or MySQL for production deployments with high concurrency. Proxied artifact access is on by default (--serve-artifacts): clients upload through the server and never hold storage credentials. The docs add the other side of that: anyone who can use the server reaches every artifact the server can reach.

Sizing the machine

The self-hosting pages we read give no CPU or memory minimums. Training does not run on the tracking server; it runs on your GPU boxes and notebooks, which only send HTTP requests. What grows on this machine is data:

• Metric and param rows land in PostgreSQL as many small writes, the random I/O NVMe handles well.
• Artifacts fill the drive. Checkpoints logged from LLM fine-tuning runs are the usual reason; size the drive for the models you keep, not the code.
• Deleted runs keep their disk. Per the backend store page, deleting a run removes neither its metadata nor its artifacts until you run mlflow gc.

The cartridge to run it on

A tracking server is a database plus a file tree, and the NVMe dedicated server lines are a whole HPE Moonshot cartridge with its NVMe drives on PCIe, root access and a 1 Gbps unmetered port.

• Xeon D-1500 (D-1548 with 8 cores or D-1587 with 16, 32 GB to 128 GB): room for PostgreSQL's cache and a team's worth of requests.
• Xeon E3-1585L v5 (4 cores, 32 GB to 64 GB): enough for a small group.
• Drives: 240 GB to 3.84 TB each; two in RAID 1 survive a failed drive, but a mirror is not a backup.
• Delivery: a racked build is handed over about 30 minutes once payment is approved; a build made to order takes 4 to 24 hours.

Past 3.84 TB of artifacts, keep the tracking server here and point --artifacts-destination at an S3 bucket on a MinIO storage server; the artifact store page covers S3-compatible stores through MLFLOW_S3_ENDPOINT_URL. Cartridges and network are ours, on AS19624, in New York, Miami, San Francisco, Amsterdam and Bucharest.

Installing MLflow and PostgreSQL

Run this as root on a fresh Ubuntu 24.04 install. It enables ufw with SSH only, installs PostgreSQL, a system user and a virtualenv in /opt/mlflow, and writes the generated database password, CSRF key and admin password to two files that only root and the mlflow group may open. The env file is moved into place last, and the first line refuses a second run once it exists.

cd /tmp && E=/etc/mlflow && { [ ! -e $E/mlflow.env ] || { echo "STOP - setup already ran"; false; }; } &&
sed -i 's/^IPV6=.*/IPV6=yes/' /etc/default/ufw && ufw allow OpenSSH && ufw --force enable &&
apt-get update && apt-get install -y python3-venv postgresql &&
{ id mlflow >/dev/null 2>&1 || useradd -r -m -d /var/lib/mlflow -s /usr/sbin/nologin mlflow; } &&
install -d -o mlflow -g mlflow -m 750 /srv/mlflow/artifacts &&
python3 -m venv /opt/mlflow && /opt/mlflow/bin/pip install 'mlflow[auth]==3.17.2' psycopg2-binary &&
PG=$(openssl rand -hex 24) && KEY=$(openssl rand -hex 32) && ADM=$(openssl rand -hex 16) &&
[ ${#PG} = 48 ] && [ ${#KEY} = 64 ] && [ ${#ADM} = 32 ] &&
printf "CREATE ROLE mlflow LOGIN PASSWORD '%s';\n" "$PG" | sudo -u postgres psql -q -v ON_ERROR_STOP=1 &&
sudo -u postgres createdb -O mlflow mlflow &&
install -d -m 750 -g mlflow $E && install -m 640 -g mlflow /dev/null $E/basic_auth.ini &&
install -m 640 -g mlflow /dev/null $E/mlflow.env.new &&
DB="postgresql://mlflow:$PG@127.0.0.1:5432/mlflow" &&
printf '%s\n' '[mlflow]' 'default_permission = READ' "database_uri = $DB" 'admin_username = admin' > $E/basic_auth.ini &&
printf '%s\n' "MLFLOW_BACKEND_STORE_URI=$DB" "MLFLOW_FLASK_SERVER_SECRET_KEY=$KEY" \
  "MLFLOW_AUTH_CONFIG_PATH=$E/basic_auth.ini" "MLFLOW_AUTH_ADMIN_PASSWORD=$ADM" > $E/mlflow.env.new &&
mv $E/mlflow.env.new $E/mlflow.env && echo "OK - admin password is $ADM" || echo 'STOP - see above'

The admin password is printed once: store it now. If the block stops halfway, nothing has used the secrets yet; clear the partial state with sudo -u postgres dropdb --if-exists mlflow && sudo -u postgres dropuser --if-exists mlflow and paste it again.

A systemd unit with login turned on

--app-name basic-auth starts the auth app. In 3.17.2 it ships no default admin password: on the first start it creates admin from MLFLOW_AUTH_ADMIN_PASSWORD and refuses to start without one, or without MLFLOW_FLASK_SERVER_SECRET_KEY (basic_auth.ini, mlflow/server/__init__.py, mlflow/server/auth/__init__.py). Passwords must be at least 12 characters.

printf '%s\n' '[Unit]' 'Description=MLflow tracking server' 'Requires=postgresql.service' \
  'After=postgresql.service network-online.target' '[Service]' 'User=mlflow' 'Group=mlflow' \
  'WorkingDirectory=/var/lib/mlflow' 'EnvironmentFile=/etc/mlflow/mlflow.env' \
  'ExecStart=/opt/mlflow/bin/mlflow server --app-name basic-auth --host 127.0.0.1 --port 5000 --artifacts-destination /srv/mlflow/artifacts' \
  'Restart=on-failure' '[Install]' 'WantedBy=multi-user.target' > /etc/systemd/system/mlflow.service &&
systemctl daemon-reload && systemctl enable --now mlflow &&
for i in $(seq 60); do curl -fs http://127.0.0.1:5000/health >/dev/null && break; sleep 2; done &&
curl -fsS http://127.0.0.1:5000/health && echo &&
[ "$(curl -s -o /dev/null -w '%{http_code}' 'http://127.0.0.1:5000/api/2.0/mlflow/experiments/search?max_results=1')" = 401 ] &&
sed -i '/^MLFLOW_AUTH_ADMIN_PASSWORD=/d' /etc/mlflow/mlflow.env &&
! grep -q '^MLFLOW_AUTH_ADMIN_PASSWORD' /etc/mlflow/mlflow.env &&
systemctl restart mlflow && sleep 5 &&
for i in $(seq 60); do curl -fs http://127.0.0.1:5000/health >/dev/null && break; sleep 2; done &&
curl -fs http://127.0.0.1:5000/health >/dev/null && echo "OK - login required, bootstrap password removed" ||
echo 'STOP - check journalctl -u mlflow -n 50'

The block waits for /health, which the auth app leaves open, then checks that the API answers 401 without credentials. Only then does it delete the bootstrap password. The docs say to unset it after the first successful start so the secret does not linger in the server's environment; once admin exists the setting is ignored. Leave the secret key in place; the docs warn that servers holding different keys hit validation errors.

Reaching the UI: tunnel first, TLS second

The CLI reference is blunt that --host controls binding and is not a security setting. Bound to 127.0.0.1, the server is reachable only from the machine, so the first login goes through your own SSH account: ssh -N -L 5000:127.0.0.1:5000 you@your-server, and the UI answers at http://localhost:5000; sign in as admin. As admin, add users at /signup or with create_user on the auth client, and rotate the admin password with update_user_password.

Every user starts with READ on every experiment (default_permission). For private-by-default experiments, set default_permission = NO_PERMISSIONS in /etc/mlflow/basic_auth.ini and restart. Scripts authenticate with MLFLOW_TRACKING_USERNAME and MLFLOW_TRACKING_PASSWORD; both must be set.

1. Point an A record (and AAAA, if you use IPv6) for mlflow.example.com at the server.
2. apt-get install -y caddy (in Ubuntu 24.04's universe repository), then ufw allow 80,443/tcp before any certificate is requested.
3. printf '%s\n' 'mlflow.example.com {' ' reverse_proxy 127.0.0.1:5000' '}' > /etc/caddy/Caddyfile && systemctl reload caddy
4. Caddy passes the public Host header through, and the security middleware accepts only localhost and private IPs by default, so allow your name, and allow its HTTPS origin too, since browsers send an Origin header on the UI's own POSTs and the CORS guard blocks writes from any origin it does not know. Set N to your name, then run `N=mlflow.example.com; [ "$N" != mlflow.example.com ] && printf '%s\n' "MLFLOW_SERVER_ALLOWED_HOSTS=$N,localhost:*,127.0.0.1:*" "MLFLOW_SERVER_CORS_ALLOWED_ORIGINS=https://$N" >> /etc/mlflow/mlflow.env && systemctl restart mlflow || echo 'STOP - put your own name in N'`.
5. Set MLFLOW_TRACKING_URI=https://mlflow.example.com on clients. Basic auth sends the password with every request; never open port 5000 itself.

Backups and a restore that keeps the old copy

Two things hold state: the PostgreSQL database (runs, registry, users) and the artifact folder. The script dumps the database first and archives files second, so every run in the dump has its artifacts in the same set (our ordering choice). Each file is verified, and old sets are pruned only after a good one exists. GNU tar exits 1 when a file changes while it reads, which an active upload can cause; the script accepts that and still lists the archive.

install -d -m 700 /var/backups/mlflow &&
cat > /usr/local/sbin/mlflow-backup.sh <<'EOF'
D=/var/backups/mlflow; T=$(date +%F-%H%M%S)
sudo -u postgres pg_dump -Fc mlflow > "$D/db-$T.dump" &&
  pg_restore --list "$D/db-$T.dump" > /dev/null || { echo "STOP - database dump bad, see $D"; exit 1; }
tar -czf "$D/files-$T.tar.gz" -C / srv/mlflow/artifacts etc/mlflow etc/systemd/system/mlflow.service
[ $? -le 1 ] && tar -tzf "$D/files-$T.tar.gz" > /dev/null || { echo "STOP - file archive bad, see $D"; exit 1; }
find "$D" -name 'db-*.dump' -mtime +14 -delete
find "$D" -name 'files-*.tar.gz' -mtime +14 -delete
echo "backup ok $T"
EOF
printf '%s\n' '15 3 * * * root umask 077; bash /usr/local/sbin/mlflow-backup.sh >> /var/log/mlflow-backup.log 2>&1' > /etc/cron.d/mlflow-backup &&
(umask 077; bash /usr/local/sbin/mlflow-backup.sh)

Get /var/backups/mlflow off the box: a second host pulls it using rsync -a over SSH with a key allowed to read that root-only directory (no --delete, so a wiped source cannot wipe the copies). To restore a set with timestamp STAMP, as root:

1. Verify the set and look for old leftovers while the service still runs, then stop it and set the current state aside: `d=/var/backups/mlflow; tar -tzf $d/files-STAMP.tar.gz srv/mlflow/artifacts >/dev/null && pg_restore -l $d/db-STAMP.dump >/dev/null && [ ! -e /srv/mlflow/artifacts.before ] && ! sudo -u postgres psql -lqt | cut -d'|' -f1 | grep -qw mlflow_before_restore && systemctl stop mlflow && sudo -u postgres psql -c 'ALTER DATABASE mlflow RENAME TO mlflow_before_restore' && mv /srv/mlflow/artifacts /srv/mlflow/artifacts.before && [ -d /srv/mlflow/artifacts.before ] && sudo -u postgres psql -lqt | cut -d'|' -f1 | grep -qw mlflow_before_restore && echo ok. A damaged set or a previous restore's copies fail a check before anything is stopped; if the stop happened but ok is missing, MLflow stays down until you fix the cause. Go on only after ok`.
2. sudo -u postgres createdb -O mlflow mlflow && sudo -u postgres pg_restore -d mlflow --no-owner --role=mlflow --exit-on-error < /var/backups/mlflow/db-STAMP.dump
3. tar -xzf /var/backups/mlflow/files-STAMP.tar.gz -C / srv/mlflow/artifacts && chown -R mlflow:mlflow /srv/mlflow/artifacts && systemctl start mlflow, then check /health and log in.
4. To undo the restore, run `[ -d /srv/mlflow/artifacts.before ] && [ ! -e /srv/mlflow/artifacts.failed ] && sudo -u postgres psql -lqt | cut -d'|' -f1 | grep -qw mlflow_before_restore && systemctl stop mlflow && sudo -u postgres dropdb --if-exists mlflow && sudo -u postgres psql -c 'ALTER DATABASE mlflow_before_restore RENAME TO mlflow' && { [ ! -e /srv/mlflow/artifacts ] || mv /srv/mlflow/artifacts /srv/mlflow/artifacts.failed; } && mv /srv/mlflow/artifacts.before /srv/mlflow/artifacts && systemctl start mlflow && systemctl is-active mlflow, which ends by printing active; afterwards delete artifacts.failed. If the restore is good instead, drop mlflow_before_restore with dropdb as postgres and delete artifacts.before`.
5. On a replacement server, install with the first two blocks, then work through steps 1 to 3. Extracting only srv/mlflow/artifacts keeps that server's new database password in place. Accounts come back with the database, so use the old server's users to sign in; the admin password printed during the new server's setup stops working.

Upgrades and schema migrations

Per the backend store page, mlflow server fails against a database with an out-of-date schema, so an upgrade is a short, ordered outage. The CLI reference warns that migrations can be slow and are not guaranteed to be transactional.

1. bash /usr/local/sbin/mlflow-backup.sh; it must end with backup ok.
2. systemctl stop mlflow && /opt/mlflow/bin/pip install 'mlflow[auth]==NEW_VERSION' after reading that release's notes.
3. sudo -u mlflow bash -c 'set -a; . /etc/mlflow/mlflow.env; exec /opt/mlflow/bin/mlflow db upgrade "$MLFLOW_BACKEND_STORE_URI"'. The URI is a required argument, so the password shows in ps while it runs.
4. systemctl start mlflow. The auth tables migrate themselves on start (init_db in sqlalchemy_store.py); the docs also give python -m mlflow.server.auth db upgrade --url for doing it by hand.
5. If it goes wrong, reinstall the old version (/opt/mlflow/bin/pip install 'mlflow[auth]==OLD_VERSION') and restore the set from step 1 with the restore steps above.

Licence and limits

MLflow is Apache License 2.0 (LICENSE.txt). Our general reading, not legal advice: running it, changing it and using it commercially are allowed; if you distribute copies, keep the licence text and notices and mark files you changed. The licence carries no duty to publish your source.

• A single cartridge is a single point of failure; there is no failover in this design.
• The repo's docker-compose/ bundle (MLflow, PostgreSQL, RustFS) publishes all of their ports on every host address with no login app, per docker-compose.yml. Fine on a laptop; on a public server it needs the same loopback and auth treatment as above.
• Training compute belongs elsewhere: a Jupyter GPU server logs to this one over HTTPS. If PostgreSQL itself becomes the bottleneck, a dedicated PostgreSQL server can take the backend store.