A Frigate NVR server on bare metal: detector, disks and the uplink

What Frigate asks of the machine

Frigate splits the work in two. The CPU decodes the video and looks for motion; regions with motion go to a detector, which says whether there is a person, a car or a dog in them. Frigate is designed around a detector; its plain CPU detector is listed for testing only.

Detection runs on a lower-resolution substream (Frigate's example uses 1280x720): Frigate crops the moving region and resizes it to the model's 320x320 input, so the detect resolution only needs to be high enough for the objects you want to catch, and anything above that mostly costs CPU. Recording takes the camera's main stream and writes it to disk without re-encoding. The default detection rate is 5 fps, and Frigate's docs say it rarely needs raising.

Choosing the detector

For a new build, take the Hailo-8. Frigate now advises against the Coral for new installations except where power is very tight or nothing else fits, and keeps supporting it for existing ones. Google has also archived the Coral software, so its PCIe driver comes from community forks; Frigate's docs point to jnicolson/gasket-builder. Keep the Coral if your models already run on an Edge TPU.

A Coral's ceiling is easy to compute: Frigate's docs give 1000 divided by the inference time in milliseconds, so 10 ms means about 100 detections per second. Frigate's planning guide puts 1-6 cameras with occasional motion on one entry-level accelerator, 12 or more busy cameras on multiple accelerators or one discrete GPU, and 40-plus cameras on a modern discrete GPU. That last tier belongs on GPU dedicated servers, with the card in a rack machine.

The two TPU cartridges

Our Coral and Hailo-8 TPU servers are HPE Moonshot cartridges with the module on the M.2 slot: the m710x with a quad-core Xeon E3-1585L v5, and the m510 with an 8- or 16-core Xeon D. The entry build of either has 32 GB DDR4, one 240 GB NVMe drive and an unmetered 1 Gbps port. You get root and the remote console, and no other customer's workload runs on the cartridge.

• m710x: its Iris Pro P580 graphics decode video in hardware, which is the job Frigate wants off the CPU. The E3-1585L v5 is a 6th-generation (Skylake) part, so Frigate's table gives preset-vaapi for it.
• m510: more cores, but no Xeon D has Quick Sync, so the cores decode every stream themselves. Pick it when the camera count is small and you want cores for other services.
• Disks: 240 GB holds little video. Disks are set in the same configurator as the module; size them for the retention below, or keep continuous footage short.
• Lead time: a module makes the order a custom build, so it is built to order in 4-24 hours rather than taken off the shelf.

Sizing recording storage

Frigate stores the record stream as the camera encodes it, so disk use is the camera's bitrate multiplied by the time kept. Our arithmetic, not Frigate's: one camera at 1 Mbit/s fills 10.8 GB a day. Read the bitrate from the camera's encoder settings and scale.

Retention is what brings that down. Frigate holds each class of footage (continuous, motion, alerts, detections) for its own number of days, and the most generous matching rule wins. Continuous recording is off by default; keeping motion for a week and alerts for a month costs a fraction of the table. Exported clips are never removed by retention.

Remote cameras: bandwidth and latency

Sending cameras to a data centre moves the cost to the site's upload. Frigate writes new segments to a cache first and only then decides whether retention keeps them, so the record stream has to flow all the time, motion or not. Each camera therefore sends its main stream plus the detect substream, continuously. Frigate's Add Camera wizard shows an estimated bandwidth figure per stream: add them up and compare the sum with the upload your line actually sustains, with room left for everything else on it.

• Latency: every alert arrives one network round trip and some buffering later than it would on site. Ping the test IP in each city and pick the nearest; the looking glass runs traceroutes from our edge.
• Outages: if the site's line drops, nothing reaches the server for that window. Keep the cameras' own card recording on if they have it.
• Connections: the camera wizard's restream option lets Frigate and the live view share one connection to each camera, instead of each pulling its own copy across the tunnel.
• Exposure: never forward RTSP ports to the internet. Put the site router and the server on a WireGuard tunnel and let Frigate reach the camera subnet through it.

Inbound traffic is not billed and the port is unmetered, so the data centre side costs nothing extra per camera. The upload at the site is the limit.

Setting it up

1. Configure an m710x with the Hailo-8 and Debian or Ubuntu. When it is handed over, check AVX2 and that the module is on the PCIe bus.
2. Install the Hailo driver with Frigate's own script, which pins its driver version, rather than a newer HailoRT release. For a Coral, build gasket-builder and check /dev/apex_0.
3. Install Docker Engine (the apt-repository method in Docker's docs), then bring the WireGuard tunnel to the camera site up.
4. Write the compose file, launch the stack (docker compose up -d) and read the generated admin password with docker logs frigate.
5. Add cameras with the wizard or in YAML: substream for detect, main stream for record.

grep -m1 -o avx2 /proc/cpuinfo        # Frigate needs AVX and AVX2
lspci -nn | grep -Ei '1e60|1ac1'       # Hailo-8 = 1e60:2864, Coral = 1ac1:089a

# Hailo-8: Frigate's own host driver script (read it before you run it)
wget https://raw.githubusercontent.com/blakeblackshear/frigate/dev/docker/hailo8l/user_installation.sh
sudo chmod +x user_installation.sh
./user_installation.sh
sudo reboot
ls -l /dev/hailo0 && cat /sys/module/hailo_pci/version

services:
  frigate:
    container_name: frigate
    restart: unless-stopped
    stop_grace_period: 30s
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "512mb"            # size it with Frigate's shm calculator
    devices:
      - /dev/hailo0:/dev/hailo0   # Coral M.2 instead: /dev/apex_0:/dev/apex_0
      - /dev/dri/renderD128:/dev/dri/renderD128   # m710x graphics, for decoding
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./config:/config
      - ./storage:/media/frigate
      - type: tmpfs
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "127.0.0.1:8971:8971"     # ssh -L 8971:127.0.0.1:8971 root@server

mqtt:
  enabled: False        # turn on if you run Home Assistant / an MQTT broker
detectors:
  hailo:
    type: hailo8l          # the same type serves the Hailo-8
    device: PCIe
model:
  width: 320
  height: 320
  input_tensor: nhwc
  input_pixel_format: rgb
  input_dtype: int
  model_type: yolo-generic
  labelmap_path: /labelmap/coco-80.txt
ffmpeg:
  hwaccel_args: preset-vaapi
record:
  enabled: True
  motion:
    days: 7
  alerts:
    retain:
      days: 30
cameras:
  gate:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.10.20:554/sub    # low-res substream
          roles: [detect]
        - path: rtsp://192.168.10.20:554/main   # full-res main stream
          roles: [record]

Running it

• Port 8971 is the authenticated UI. Port 5000 is unauthenticated and 8554 restreams without a password by default: do not publish either.
• Docker's -p and ports: publish on all interfaces and bypass ufw. The compose file publishes the UI on loopback only, which an SSH port forward reaches; to use it over the tunnel instead, publish it on the WireGuard interface's IP.
• Change the generated admin password under Settings > Users; Frigate requires at least 12 characters.
• Back up /config: it holds the config file and the SQLite database.
• Hold the kernel and the Hailo or Coral driver at versions that work together. Nothing on our side updates the machine for you.
• Watch System > Metrics: if detections are being skipped, tune motion masks before adding hardware.

When a box on site is the better answer

Frigate's own hardware page favours a small local PC with two network ports, so the cameras sit on a private network with no internet access. That is the right design when the site's upload cannot carry every stream, when footage must keep recording through an internet outage, or when alerts must fire within the shortest possible delay. A server here earns its place when several sites feed one recorder, when the recorder must not be stolen along with the cameras, or when the site already has fibre with upload to spare.