Prometheus Grafana server: size the disk, install, fence the ports

What the stack asks of the machine

Grafana's sizing covers the Grafana process only: the UI, the data source proxy, the alert engine and the image renderer. The metric store is sized separately, and for a self-hosted stack that store is Prometheus. Its documentation publishes no memory or core figure, so treat RAM as something you measure once real targets are scraped, and buy a line whose memory you can raise later.

Sizing the TSDB disk

The Prometheus storage documentation gives the rough formula needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample, with an average of 1-2 bytes per sample. One series scraped every 15 seconds produces one sample per 15 seconds, so the ingest rate is your active series divided by the scrape interval.

Our own worked example, not a documented figure: 20,000 active series at a 15-second interval is about 1,333 samples a second. Ninety days is 7,776,000 seconds. At 2 bytes a sample that comes to roughly 20.7 GB of blocks. Count your series before you trust any example, including this one.

• Retention is set in prometheus.yml under storage.tsdb.retention: time is how long samples are kept, 15d when neither time nor size is set. The old flags --storage.tsdb.retention.time and --storage.tsdb.retention.size still appear in many guides; the current command-line reference marks both deprecated.
• size caps the bytes of blocks kept and is off unless set. Its units are powers of two, so 400GB means 400 GiB. The WAL and head chunks count toward it, and if both limits are set, whichever triggers first wins.
• The docs recommend setting the size cap to at most 80-85% of the disk you give Prometheus; the rest covers compactions in progress.
• Expired blocks can take up to two hours to be removed.
• To shrink the ingest rate, the docs say cutting the number of series is likely more effective than lengthening the scrape interval, because samples within a series compress well.

Why a whole machine, and a separate one

A monitoring host that shares hardware with what it watches goes dark at the moment you need it. The Prometheus docs also rule out network file systems for the TSDB and strongly recommend a local filesystem, which is what a dedicated server gives you: the whole machine, its own disks, root access, and the hardware to yourself.

Scrape traffic runs inbound to the monitoring host all day. Every line marked Unmetered has no transfer allowance and no overage fee, so the scrape interval is a storage decision, not a billing one. Every racked single-, dual- and quad-socket server has an out-of-band console (HPE iLO, Dell iDRAC or a Supermicro BMC), so when a bad firewall rule cuts off SSH you repair it from the console, not a ticket queue.

Which lines fit

• Moonshot cartridges: one service per box, for a small fleet whose disk figure fits the cartridge's drive. Of the cartridges, just the m510 and m710x carry a console; the others offer power control alone.
• Intel Xeon D-1587: a 1.92 TB NVMe SSD for a TSDB that writes all day, with 16 cores and 16-128 GB of memory. See NVMe dedicated servers.
• Single socket: the usual base for an application host with its database, here Grafana with an external PostgreSQL on the same box.
• Dual socket: long retention over many series, or Prometheus, Grafana and Alertmanager side by side with room to grow.

Put the box in the city closest to the hosts it scrapes; the five sites and their test IPs are on our data center page. A racked machine is handed over in about 30 minutes once payment is approved; a built-to-order line takes 4 to 24 hours. Linux images are free.

Installing Prometheus

The docs ship Prometheus as a precompiled tarball and leave the service unit to you. The unit below is ours: it runs Prometheus as an unprivileged user and binds the HTTP port to loopback so only Grafana on the same machine can reach it; both retention limits are set in prometheus.yml in the next step. Version 3.15.1 was current on 22 October 2026; take the current one and its SHA256 from the downloads page.

cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v3.15.1/prometheus-3.15.1.linux-amd64.tar.gz
sha256sum prometheus-3.15.1.linux-amd64.tar.gz   # compare with prometheus.io/download
tar xvf prometheus-3.15.1.linux-amd64.tar.gz
sudo install -m 0755 prometheus-3.15.1.linux-amd64/prometheus prometheus-3.15.1.linux-amd64/promtool /usr/local/bin/
sudo useradd --system --no-create-home --shell /usr/sbin/nologin prometheus
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus
sudo tee /etc/systemd/system/prometheus.service >/dev/null <<'EOF'
[Unit]
Description=Prometheus
After=network-online.target
Wants=network-online.target

[Service]
User=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.listen-address=127.0.0.1:9090
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

On every host to be watched, this one included, install node_exporter the same way from the same downloads page (1.12.1 on that date) and run it as its own unprivileged user under systemd. It listens on port 9100. Then write the scrape configuration, check it with promtool, and start the service:

sudo tee /etc/prometheus/prometheus.yml >/dev/null <<'EOF'
global:
  scrape_interval: 15s

storage:
  tsdb:
    retention:
      time: 90d
      size: 400GB   # 400 GiB: 80% of a 500 GiB volume

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
  - job_name: node
    static_configs:
      - targets: ['localhost:9100', '203.0.113.21:9100', '203.0.113.22:9100']
EOF
promtool check config /etc/prometheus/prometheus.yml
sudo systemctl daemon-reload
sudo systemctl enable --now prometheus

Grafana from the APT repository

Turn the firewall on first, with SSH as the only open port: run sudo ufw allow OpenSSH, and after it sudo ufw enable. Grafana binds to all interfaces on port 3000 by default and its first account is admin with the password admin, so it must not start on an open port. Then install the OSS package from Grafana's repository, which also keeps it updated through apt:

sudo apt-get install -y apt-transport-https wget gnupg
sudo mkdir -p /etc/apt/keyrings
sudo wget -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
sudo chmod 644 /etc/apt/keyrings/grafana.asc
echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install grafana
sudo systemctl daemon-reload
sudo systemctl enable --now grafana-server

1. From your workstation, open a tunnel through your own login: ssh -L 3000:localhost:3000 you@203.0.113.10.
2. Browse to http://localhost:3000, sign in as admin/admin, and set a strong password at the prompt.
3. Under Connections, choose Add new connection, search for Prometheus, add the data source, and set the server URL to http://localhost:9090.
4. Only then decide how the team reaches Grafana: set http_addr = 127.0.0.1 in the server section of /etc/grafana/grafana.ini behind a TLS reverse proxy, or allow port 3000 from your office address alone.

Fencing 9090 and 9100

The Prometheus security model presumes that untrusted users can reach the HTTP endpoint, and anyone who can reaches every time series in the database. Basic auth and TLS exist only through a web configuration file passed with --web.config.file, which the docs still call experimental. Node_exporter takes the same kind of file. Without one, neither asks for a login.

• Prometheus: bound to 127.0.0.1 in the unit above, so nothing outside the box reaches 9090. Its stock default is 0.0.0.0:9090.
• node_exporter on each watched host: ufw has to be on there as well, or an allow rule fences nothing. If it is off, allow SSH and every port the host already serves before sudo ufw enable applies its default deny to incoming traffic; then allow 9100 from the monitoring server's address only, for example sudo ufw allow from 203.0.113.10 to any port 9100 proto tcp.
• Check that IPV6=yes is set in /etc/default/ufw, so the same rules cover IPv6; node_exporter listens on :9100, every address.
• Leave --web.enable-admin-api and --web.enable-lifecycle off unless 9090 is on loopback: they expose series deletion, reloads and shutdown over HTTP.

Backups, upgrades and honest limits

The docs recommend snapshots for backups. The snapshot endpoint, POST /api/v1/admin/tsdb/snapshot, needs the admin API, so enable it only while 9090 is bound to loopback. A copy taken without a snapshot can lose the samples recorded since the last block was written, which can be up to about the last three hours. Grafana's dashboards live in its own database: SQLite works for evaluation, but Grafana recommends an external MySQL or PostgreSQL for production, and that database needs its own backup.

Prometheus local storage is neither clustered nor replicated, so one disk or node failure loses what was on it. And a monitoring box cannot report its own outage: pair it with an outside check such as a self-hosted Uptime Kuma instance on a different machine.