AnythingLLM self-hosted: Docker, built-in engine, multi-user, TLS

What the Docker edition runs

The image starts two Node.js processes: the API and web server on port 3001, and the collector that parses uploaded files and scraped pages (both launched by docker-entrypoint.sh). Users, workspaces and chat history go into a SQLite file, anythingllm.db. Vectors go into LanceDB, which is built in, and text is embedded on the CPU inside the container unless you pick an external embedder. All of it lives under STORAGE_DIR, so the host folder you mount there is the whole instance.

The project's overview points single-person use at the Desktop app. The Docker edition is the one with multi-user accounts, per-workspace access, password protection, embeddable chat widgets and white-labelling. Sessions last 30 days by default (JWT_EXPIRY changes that). Because the database is SQLite, the cloud deployment guide advises against running several containers against one instance, and points to a separate PostgreSQL image with PGVector when you need that.

Licence: MIT. Our general reading, not legal advice: you may run, change and redistribute it, and copies you pass on must keep the copyright notice and licence text. No source-code duty attaches.

Hardware: check AVX2 before the card

LanceDB is compiled for Intel Haswell (2013) and newer. On a processor without AVX2 the server dies with an Illegal instruction message as soon as it loads, and the only fix the docs give is a different vector database. That matters when choosing the host: of the four platforms our GPU catalogue fits cards into, Xeon E5-2600 v1/v2 predates Haswell, while E5-2600 v3/v4, Xeon Silver/Gold and AMD EPYC do not. Many rows on the GPU list show the cheaper E5-2600 v1/v2 host by default. Some cards fit more than one chassis, and the configurator shows which hosts each one comes on: pick it on one of the latter three, and if a card is offered only on E5-2600 v1/v2, choose a different card. Still run lscpu | grep -o avx2 on the delivered machine; the install block below refuses to continue without it.

With the built-in engine on, the model runs inside the AnythingLLM container, so the card's memory sets which models fit. Onboarding suggests models that fit your GPU memory. For sizing by parameter count, see LLM hosting.

Install with the built-in engine on the GPU

Start from Ubuntu 24.04 with the NVIDIA driver, Docker Engine from Docker's own repository, and the NVIDIA Container Toolkit registered with nvidia-ctk runtime configure --runtime=docker followed by a Docker restart. Then, as root:

sed -i 's/^IPV6=.*/IPV6=yes/' /etc/default/ufw && ufw allow OpenSSH && ufw --force enable
D=/srv/anythingllm
v=$(docker info 2>/dev/null | sed -n 's/^ *Server Version: *\([0-9]*\).*/\1/p')
if ! lscpu | grep -qw avx2; then echo "STOP: this CPU lacks AVX2"
elif ! [ "${v:-0}" -ge 28 ] 2>/dev/null; then echo "STOP: Docker Engine 28+ needed"
elif ! docker info 2>/dev/null | grep -qi 'runtimes:.*nvidia'; then echo "STOP: NVIDIA runtime missing"
elif [ -e $D/storage/.env ]; then echo "STOP: $D is already set up"
else
mkdir -p $D/storage && install -m 600 /dev/null $D/storage/.env &&
echo 'DISABLE_TELEMETRY="true"' > $D/storage/.env && chown -R 1000:1000 $D/storage &&
cat > $D/compose.yaml <<'EOF' && docker compose -f $D/compose.yaml up -d
services:
  anythingllm:
    image: mintplexlabs/anythingllm:1.19.0
    container_name: anythingllm
    restart: unless-stopped
    ports: ["127.0.0.1:3001:3001"]
    cap_add: [SYS_ADMIN]
    environment: [STORAGE_DIR=/app/server/storage, ENABLE_BUILT_IN_LLM=true]
    volumes:
      - /srv/anythingllm/storage:/app/server/storage
      - /srv/anythingllm/storage/.env:/app/server/.env
    extra_hosts: ["host.docker.internal:host-gateway"]
    deploy:
      resources:
        reservations:
          devices: [{driver: nvidia, count: all, capabilities: [gpu]}]
EOF
fi

Our changes to the official commands, in full: host port bound to 127.0.0.1; image pinned to one release instead of latest; restart: unless-stopped instead of --rm; a fixed container name; the .env created empty at mode 600 with telemetry off (the README's opt-out); and the host.docker.internal mapping the docs add for Linux. The NVIDIA reservation is the commented option from HOW_TO_USE_DOCKER.md. cap_add: SYS_ADMIN is kept: the cloud guide says the scraper's sandboxed Chromium needs it.

That .env does not stay empty. On first start the app writes its encryption key and salt there (server/utils/EncryptionManager/index.js), and a JWT secret when you enable multi-user mode (server/endpoints/system.js). Never delete it or start over with a new one; stored secrets would no longer decrypt.

Why 127.0.0.1: a port mapping with no address is published on 0.0.0.0 and [::], and Docker's published traffic skips ufw on IPv4 (on IPv6 too when Docker's own IPv6 networking is on). Publishing on 127.0.0.1 keeps it local only on Engine 28 and later, hence the version check. The storage folder belongs to UID 1000 because the image always runs as that user; the docs warn against chmod 777.

Lock the instance before it is public

A fresh instance is in single-user mode, where a password is optional and whoever can open the page can change every setting and read every chat. Close that before any proxy exists.

1. Wait until docker ps shows the container as (healthy); the image's health check calls /api/ping once a minute.
2. Tunnel from your workstation with ssh -L 3001:127.0.0.1:3001 you@your-server and browse to http://localhost:3001 there.
3. In onboarding, pick Built-in as the LLM provider and download a model from its table. The first chat also fetches the llama.cpp runtime for your card, so it is slower than later ones.
4. Turn on Enable multi-user mode and set the first admin's username and password. You are logged out; log back in with them. The docs warn this cannot be reverted to single-user mode.
5. Add people as Admin, Manager (sees and edits every workspace; the LLM, embedder and vector store settings stay with admins) or Default (chat only in workspaces they are added to).

To confirm the card is used, the Docker guide's health query against the engine (127.0.0.1:13355 inside the container) should report cuda after your first chat. If it reports something else, EMBEDDED_LEMONADE_LLAMACPP_BACKEND=cuda forces it.

TLS in front with Caddy

Create an A record for the name that points here, and an AAAA too, but only if you added IPv6 at checkout (no default build includes it). Ubuntu 24.04 ships Caddy as a regular package, and it obtains the certificate on its own:

apt-get update && apt-get install -y caddy &&
cat > /etc/caddy/Caddyfile <<'EOF' &&
llm.example.com {
	reverse_proxy 127.0.0.1:3001 {
		flush_interval -1
	}
}
EOF
caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile &&
ufw allow 80,443/tcp && systemctl reload caddy

The project publishes an nginx example instead; this is our translation of it. Their config upgrades WebSockets for the agent route (/api/agent-invocation/), which Caddy's reverse proxy does with no extra directive, and turns response buffering off, which flush_interval -1 matches. Caddy runs on the host as an ordinary process, so it is unreachable until ufw allows 80/443, on IPv4 and, if the server has addresses there, on IPv6; port 3001 stays closed.

Pointing it at an engine you already run

If Ollama or llama.cpp's server already runs on the host, leave the built-in engine off and choose that provider in settings. An engine bound to 127.0.0.1 is invisible from the container. On Linux the Docker guide gives http://172.17.0.1:11434 (the host's Docker bridge address) for Ollama, so make the engine listen there (OLLAMA_HOST=172.17.0.1 for Ollama) and allow its port only from Docker's networks, for example ufw allow from 172.16.0.0/12 to any port 11434 proto tcp. If your team also wants a plain chat front end without workspaces, Open WebUI can share the same engine.

Backups, restores and updates

Stop the container so the SQLite and LanceDB files are quiet, archive the whole folder (database, documents, vectors, models, .env, compose file), start it again:

D=/srv/anythingllm; B=/root/backup/anythingllm-$(date +%F-%H%M%S).tar.gz
mkdir -p -m 700 /root/backup &&
docker compose -f $D/compose.yaml stop &&
(umask 077; tar -czf "$B" -C /srv anythingllm); rc=$?
docker compose -f $D/compose.yaml start
[ "$rc" -eq 0 ] && tar -tzf "$B" > /dev/null && echo "ok: $B"

Only trust an archive that printed ok, and copy it off the machine (rsync -a or scp to a box of yours; never --delete). The archive holds your encryption key, so keep the copy private. Built-in models make it large; they can be downloaded again, but a full copy restores without waiting.

1. Restore, as root. First test the archive: tar -tzf /root/backup/anythingllm-STAMP.tar.gz > /dev/null && echo ok, where STAMP is the date-and-time part of its name. No ok, stop: that archive will not restore. Then docker compose -f /srv/anythingllm/compose.yaml down.
2. Stop here if /srv/anythingllm.previous exists; otherwise mv /srv/anythingllm /srv/anythingllm.previous keeps the current state whole as your way back.
3. tar -xzf /root/backup/anythingllm-STAMP.tar.gz -C /srv, check /srv/anythingllm/storage/anythingllm.db exists, then chown -R 1000:1000 /srv/anythingllm/storage.
4. docker compose -f /srv/anythingllm/compose.yaml up -d, wait for healthy, log in and check your workspaces.
5. Only then remove /srv/anythingllm.previous. To roll back instead: bring it down, mv /srv/anythingllm /srv/anythingllm.failed, mv /srv/anythingllm.previous /srv/anythingllm, bring it up.

Updating is a backup first, then a new release number in the image: line of compose.yaml, then docker compose -f /srv/anythingllm/compose.yaml pull and up -d. The entrypoint applies database migrations on every start (npx prisma migrate deploy in docker-entrypoint.sh). Our inference: an older image may not match a migrated database, so roll back by restoring the pre-update backup, which restores the old data together with the old compose.yaml and its tag; never revert only the tag.

The machine underneath

A GPU dedicated server puts the whole card in a PCIe slot handed straight to the OS you install: no vGPU profile, no MIG slice, no time-slicing, and you choose and pin the driver and CUDA release. The host is not shared either: its cores, RAM and drives serve this one instance. Nothing on the port counts bytes, so model pulls and uploaded document sets cost no transfer, and the machine's own iLO or iDRAC console sits on a separate management network, still reachable after a bad kernel module upgrade drops the NIC.

• Host platform: E5-2600 v3/v4, Xeon Silver/Gold or EPYC, for AVX2; where a card also fits another chassis, the configurator lists it; a card offered only on E5-2600 v1/v2 is the wrong card here.
• Card: Turing or newer NVIDIA (RTX 20-series, Quadro RTX, Tesla T4 and later) for the built-in engine on CUDA. The Pascal, Volta, Maxwell and older cards on the list (GTX 10-series, Tesla P4/P40/P100, Quadro P5000/P6000, TITAN V, Tesla V100, Tesla M6/M60, the Quadro mobile parts) are older than that.
• Ready machines go live about 30 minutes after the order is approved; a build to order takes 4-24 hours.
• Not a fit: if every chat goes to a hosted API and embedding stays on the CPU, the GPU sits idle; a plain dedicated server is enough, provided its line has AVX2, which rules out Atom C2000, the Opterons and Xeon E5-2600 and E5-4600 v1/v2.