LibreChat self-hosted on a GPU server: Compose, secrets, Ollama, nginx

What the stack runs and what it needs

The default docker-compose.yml in the repository starts five containers: api (LibreChat on port 3080), mongodb (users, conversations, settings), meilisearch (full-text search over past conversations), rag_api and vectordb (file search on a pgvector database). An admin-panel service is also defined; without ADMIN_PANEL_SESSION_SECRET in .env it refuses to start, and restart: always keeps it retrying.

The front end is small. The engine sets the hardware: the weights figures are a floor, and the key-value cache grows with context on the same card, so add a quarter to a half. LLM hosting walks through the arithmetic per model size.

Where the GPU earns its place

Pair LibreChat with Ollama on the same host and conversations stay on hardware you control, while hosted providers can still sit beside it as extra endpoints. On a GPU dedicated server the card sits in a PCIe slot and is passed straight to your own OS: no hypervisor, no vGPU profile, no MIG partition, no time-slicing. You pin the CUDA or ROCm release Ollama was tested against and nothing upgrades it under you.

Everything else in the chassis is yours too: every core, all the memory, both disks, root, and the server's own iLO or iDRAC console on a separate management network, so a broken network config still leaves you a screen and power control. The port is unmetered in both directions, the term is monthly with no contract, and the machines sit in five cities.

Firewall, clone and secrets

Install Docker Engine and the Compose plugin (docker-ce, docker-compose-plugin) from Docker's apt repository, as the project's Ubuntu guide does, and add your user to the docker group so the commands below run without sudo. Then close the box before anything listens. IPV6=yes goes first because ufw writes IPv6 copies only of rules added while it is set; if you add IPv6 at checkout, the same default-deny covers it. The 11434 rule lets containers on Docker's private ranges reach Ollama on the host and nothing else. The docs also offer a Credentials Generator in the browser; generating on the server keeps the values off your workstation. Left blank, the four credentials are generated into /app/data/.env.temp, which the docs call a bootstrap convenience, not a production setup.

sudo sed -i 's/^IPV6=.*/IPV6=yes/' /etc/default/ufw
sudo ufw default deny incoming
sudo ufw allow 22/tcp
sudo ufw allow 80,443/tcp
sudo ufw allow from 172.16.0.0/12 to any port 11434 proto tcp
sudo ufw --force enable

git clone https://github.com/LibreChat-AI/LibreChat.git ~/LibreChat
cd ~/LibreChat
cp .env.example .env
chmod 600 .env
sed -i -E '/^(CREDS_KEY|CREDS_IV|JWT_SECRET|JWT_REFRESH_SECRET|MEILI_MASTER_KEY|SEARCH)=/d' .env
{
  printf 'CREDS_KEY=%s\n' "$(openssl rand -hex 32)"
  printf 'CREDS_IV=%s\n' "$(openssl rand -hex 16)"
  printf 'JWT_SECRET=%s\n' "$(openssl rand -hex 32)"
  printf 'JWT_REFRESH_SECRET=%s\n' "$(openssl rand -hex 32)"
  printf 'MEILI_MASTER_KEY=%s\n' "$(openssl rand -hex 32)"
  printf 'SEARCH=true\n'
  printf 'UID=%s\nGID=%s\n' "$(id -u)" "$(id -g)"
} >> .env
mkdir -p data-node uploads images logs "$(grep -o 'meili_data_v[0-9.]*' docker-compose.yml | head -n1)"

Lengths per the .env reference: CREDS_KEY 32 bytes (64 hex), CREDS_IV 16 bytes (32 hex), each JWT secret at least 32 bytes, the Meilisearch key at least 16. printf is a shell builtin, so no secret appears in ps or in shell history. UID and GID make the containers run as you, so they can read the private .env and write their data folders, which the mkdir creates as yours. Rotating CREDS_KEY or CREDS_IV does not re-encrypt existing records.

Compose override and the Ollama endpoint

The stock file publishes ${PORT}:${PORT} on every interface. Docker inserts its own iptables rules for published ports, so ufw never sees that traffic. Compose merges ports lists from an override rather than replacing them, so the override below uses the !override tag (it needs a recent Docker Compose that supports it) to bind 3080 and the admin panel's 3000 to IPv4 loopback only. It also switches the image to librechat:latest, which follows tagged releases instead of the dev build of the main branch (a release tag such as :v0.8.8 holds one version until you change it), and mounts librechat.yaml. The stock file already hands Meilisearch the key from .env. Create both files in ~/LibreChat before the first start.

# docker-compose.override.yml
services:
  api:
    image: registry.librechat.ai/librechat-ai/librechat:latest
    ports: !override
      - "127.0.0.1:3080:3080"
    volumes:
      - type: bind
        source: ./librechat.yaml
        target: /app/librechat.yaml
  admin-panel:
    ports: !override
      - "127.0.0.1:3000:3000"

# librechat.yaml
version: 1.3.18
cache: true
endpoints:
  custom:
    - name: "Ollama"
      apiKey: "ollama"
      baseURL: "http://host.docker.internal:11434/v1/"
      models:
        default: ["llama3:latest"]
        fetch: true
      titleConvo: true
      titleModel: "current_model"
      modelDisplayLabel: "Ollama"

The api service already maps host.docker.internal to the host gateway. Ollama listens on 127.0.0.1 by default, which a container cannot reach, so set OLLAMA_HOST=0.0.0.0 in its service environment (sudo systemctl edit ollama, then restart it). Ollama's API has no authentication: with ufw on and only the 172.16.0.0/12 rule for 11434, the internet cannot reach it. The docs prefer binding to the compose network's gateway address instead; either way the firewall rule stays. apiKey must be non-empty even though Ollama ignores it, and fetch: true pulls the model list from the server. Hosted providers go in the same custom list, with the key in .env and referenced as ${VAR_NAME}, or as user_provided so each user enters their own. Run docker compose up -d; when an endpoint fails to appear, docker compose logs api usually says why.

First account, then close registration

1. From your workstation: ssh -L 3080:127.0.0.1:3080 user@server-ip, and browse to http://localhost:3080.
2. Click Register. In a single-tenant deployment the first registered account is made administrator; there are no default credentials.
3. Turn sign-ups off: sed -i 's/^ALLOW_REGISTRATION=.*/ALLOW_REGISTRATION=false/' .env, then docker compose up -d --force-recreate api. docker compose restart would miss it: .env is bind-mounted as a single file and sed -i replaces it, so only a recreated container sees the new copy.
4. Add colleagues with docker compose exec api npm run create-user (it prompts for email and password, and works with registration disabled); remove one with docker compose exec api npm run delete-user email@domain.com.
5. Only now publish the app through the proxy.

The docs document no manual ban command. The automated moderation system, on by default, scores excessive logins, registrations and messages and temporarily bans the user and IP at a threshold (BAN_VIOLATIONS, BAN_DURATION, BAN_INTERVAL). ENFORCE_TWO_FACTOR_AUTHENTICATION=true makes local accounts enrol in TOTP before they get a session.

nginx and TLS

# /etc/nginx/sites-available/librechat
server {
    listen 80;
    listen [::]:80;
    server_name chat.example.com;
    client_max_body_size 25M;

    location / {
        proxy_pass http://127.0.0.1:3080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_cache_bypass $http_upgrade;
    }
}

Link it into sites-enabled, run sudo nginx -t, reload, then sudo certbot --nginx -d chat.example.com for the certificate. The proxy block is the docs' host-based example plus two forwarding headers: TRUST_PROXY defaults to 1, so LibreChat takes the client address from X-Forwarded-For, and the moderation bans key on it. The 25M body limit is the one in the project's own bundled nginx config; nginx's default would refuse larger uploads. Finally point DOMAIN_CLIENT and DOMAIN_SERVER at https://chat.example.com and recreate api the same way.

Updates and backups

The documented update for the default stack: docker compose down, git pull, docker compose pull, docker compose up -d. The docs also remove the old LibreChat images first to reclaim disk. git pull brings any change to the compose files with it, which is why local changes belong in the override and never in docker-compose.yml. Read the release notes before pulling: some releases reindex Meilisearch on first sync or ship migration scripts.

State lives in the project directory: data-node (MongoDB), the meili_data_* folder, uploads, images, .env and librechat.yaml, plus the librechat-data and pgdata2 volumes. Back them up before every update with the stack stopped, and keep the archive private: it holds the encryption keys. Backing up this data is your job.

When a GPU box is the wrong buy

• Only hosted APIs behind it: LibreChat needs no card, and a plain dedicated server runs the stack.
• You want one container with the engine inside and settings edited in the browser: Open WebUI is built that way.
• The model needs more memory than any single card holds: that is a multi-card build, quoted by hand.