Self hosted Kafka in KRaft mode, on a machine of its own

What a Kafka broker asks of the hardware

The Hardware and OS page sets no minimum machine. Its own example is dual quad-core Xeons with 24 GB of memory and eight 7200 rpm SATA drives, and it names disk throughput as the usual bottleneck.

Our arithmetic: 50 MB/s of writes wants about 1.5 GB of page cache for that buffer, beside the 6 GB heap; at the default 168-hour retention, 5 MB/s is about 3.0 TB per replica. Linux and Solaris are the tested platforms.

Sizing the machine

Kafka leans on the kernel's page cache rather than its heap, so memory no other tenant can claim matters more than core count. A dedicated server is the whole machine with no hypervisor, root access, and a port marked Unmetered with no transfer allowance or overage fee, which suits a workload where every consumer group reads the full stream again. Past a gigabit, a 10 Gbps port is offered on the lines that take it.

• Single-socket rack servers for one broker in development, staging or a non-critical pipeline. Every racked single-, dual- and quad-socket server has a full out-of-band console (HPE iLO, Dell iDRAC or a Supermicro BMC).
• Dual-socket lines (Xeon E5-2630L v3, E5-2650L v2, Gold 6130) for more partitions, connections and memory channels.
• Several data drives as separate log directories: a custom build takes up to 24 bays in RAID 0/1/5/6/10 or JBOD, in 4 to 24 hours. Rack and blade lines carry a hardware RAID controller; no Moonshot cartridge does.
• Small machines for a separate controller quorum, such as the NVMe line. Only the m510 and m710x cartridges carry a console; the rest have power control only. Skip the X-Gene cartridge: it boots Ubuntu 16.04 and 18.04 only.

A racked machine is handed over in about 30 minutes once payment is approved. All of it sits on our own network, AS19624, in five locations: New York, Miami, San Francisco, Amsterdam and Bucharest.

Listeners, advertised.listeners and the three ports

listeners is where the broker binds; advertised.listeners is what it publishes, through the controller, to clients and other brokers. Per the protocol guide, a client uses its bootstrap address only to fetch cluster metadata, then sends produce and fetch requests to the host and port each partition leader advertises; the broker configs reference adds that advertising 0.0.0.0 is not valid. The shipped config/server.properties listens on :9092 and :9093 with no host named, all PLAINTEXT, and advertises localhost:9092. That suits a laptop. A remote client bootstraps, is told to dial localhost, and fails, and nothing on those ports asks for a login.

CLIENT advertises the DNS name you set as H, and the self-signed certificate carries that name in its SAN, because clients verify the broker's host name by default since Kafka 2.0. The property names come from the docs; three listeners, port 9094 and an ANONYMOUS super user on loopback are our choices.

Install Kafka 4.4.0 and make the certificate

Create the DNS A record before you run it. The block checks the tarball against the SHA-512 file on downloads.apache.org (the downloads page also offers signatures and KEYS for GPG), installs OpenJDK 21 (fully supported; the docs recommend 25), unpacks into /opt/kafka-4.4.0 behind a /opt/kafka link, and makes a PKCS12 keystore valid 825 days, our pick.

# As root on Ubuntu 24.04. Set H to the DNS name clients will dial; its A record must point here.
H=kafka.example.net; V=4.4.0; T=kafka_2.13-$V.tgz; S=/root/kafka-secrets; KS=/etc/kafka/broker.p12
if [ "$H" = kafka.example.net ] || ! getent ahostsv4 "$H" >/dev/null; then echo "STOP - set H first"
elif [ -e /opt/kafka ] || [ -e $S ]; then echo "STOP - already installed, see the upgrade steps"
else
apt-get update && apt-get install -y openjdk-21-jre-headless curl ufw && cd /root &&
curl -fsSLO https://dlcdn.apache.org/kafka/$V/$T && curl -fsSLO https://downloads.apache.org/kafka/$V/$T.sha512 &&
E=$(cut -d: -f2- $T.sha512 | tr -dc 0-9A-Fa-f | tr A-F a-f) && [ ${#E} -eq 128 ] &&
[ "$E" = "$(sha512sum $T | cut -d' ' -f1)" ] && { id kafka || useradd -r -M -s /usr/sbin/nologin kafka; } &&
mkdir /opt/kafka-$V && tar -xzf $T -C /opt/kafka-$V --strip-components=1 && ln -s /opt/kafka-$V /opt/kafka &&
install -d -m 750 -o kafka -g kafka /var/lib/kafka /var/log/kafka && install -d -m 750 -g kafka /etc/kafka &&
install -m 600 /dev/null $S && P=$(openssl rand -hex 24) && [ -n "$P" ] && printf 'H=%s\nP=%s\n' $H $P > $S &&
SP=$P keytool -genkeypair -alias broker -keyalg RSA -validity 825 -dname "CN=$H" -ext "SAN=DNS:$H" \
 -storetype pkcs12 -keystore $KS -storepass:env SP && chgrp kafka $KS && chmod 640 $KS &&
SP=$P keytool -exportcert -rfc -alias broker -keystore $KS -storepass:env SP -file /etc/kafka/broker-cert.pem &&
echo "installed" || echo "STOP - read the error above"
fi

Only if a first run ends in STOP - read the error above (never after already installed), remove what it made (rm -rf /opt/kafka /opt/kafka-4.4.0 /etc/kafka /root/kafka-secrets) and run it again.

Write the KRaft configuration

One node runs both roles, which the KRaft page calls combined mode: simpler, but not recommended in critical deployments. SCRAM-SHA-512 credentials live in the metadata log, and the SASL guide says SCRAM belongs with TLS, hence SASL_SSL. The StandardAuthorizer lets only super users reach a resource no ACL matches. One broker means replication factor 1 for the internal topics, as in the shipped file; automatic topic creation is off.

# As root, after "installed". Writes /etc/kafka/kraft.properties; refuses to overwrite it.
. /root/kafka-secrets && C=/etc/kafka/kraft.properties
if [ -e $C ]; then echo "STOP - $C exists already"
else
install -m 640 -g kafka /dev/null $C && cat > $C <<EOF &&
process.roles=broker,controller
node.id=1
controller.quorum.bootstrap.servers=127.0.0.1:9093
controller.listener.names=CONTROLLER
inter.broker.listener.name=INTERNAL
listeners=CLIENT://0.0.0.0:9092,CONTROLLER://127.0.0.1:9093,INTERNAL://127.0.0.1:9094
advertised.listeners=CLIENT://$H:9092,CONTROLLER://127.0.0.1:9093,INTERNAL://127.0.0.1:9094
listener.security.protocol.map=CLIENT:SASL_SSL,CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT
sasl.enabled.mechanisms=SCRAM-SHA-512
listener.name.client.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required;
ssl.keystore.type=PKCS12
ssl.keystore.location=/etc/kafka/broker.p12
ssl.keystore.password=$P
ssl.key.password=$P
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
super.users=User:ANONYMOUS
auto.create.topics.enable=false
log.dirs=/var/lib/kafka
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
share.coordinator.state.topic.replication.factor=1
share.coordinator.state.topic.min.isr=1
EOF
echo "config written" || echo "STOP - read the error above"
fi

Only if its first run ends in STOP - read the error above (not after already exists), rm /etc/kafka/kraft.properties and run it again.

Format, firewall, start

--standalone makes this node the only voter, the bootstrap method the KRaft page recommends. ufw comes on with SSH only before Kafka first listens. The unit sets the 100,000-descriptor floor and the 6 GB heap; kafka-server-start.sh falls back to 1 GB when KAFKA_HEAP_OPTS is unset. The last lines pass only if 9093 and 9094 listen on loopback alone.

# As root, after "config written". Formats storage once, turns on ufw, starts Kafka, checks the ports.
K=/opt/kafka/bin; L=127.0.0.1:9094; U=/etc/systemd/system/kafka.service
if [ -e $U ] || [ -e /var/lib/kafka/meta.properties ]; then echo "STOP - already formatted or started"
else
ID=$($K/kafka-storage.sh random-uuid) && [ -n "$ID" ] && $K/kafka-storage.sh format --standalone -t "$ID" \
 -c /etc/kafka/kraft.properties && chown -R kafka:kafka /var/lib/kafka &&
sed -i 's/^IPV6=.*/IPV6=yes/' /etc/default/ufw && ufw allow OpenSSH && ufw --force enable &&
printf '%s\n' '[Unit]' 'Description=Apache Kafka' 'After=network-online.target' 'Wants=network-online.target' \
 '[Service]' 'User=kafka' 'Environment="KAFKA_HEAP_OPTS=-Xmx6g -Xms6g" LOG_DIR=/var/log/kafka' \
 'LimitNOFILE=100000' 'ExecStart=/opt/kafka/bin/kafka-server-start.sh /etc/kafka/kraft.properties' \
 'Restart=on-failure' '[Install]' 'WantedBy=multi-user.target' > $U &&
systemctl daemon-reload && systemctl enable --now kafka &&
for i in $(seq 40); do $K/kafka-metadata-quorum.sh --bootstrap-server $L describe --status >/dev/null 2>&1 && break; sleep 3; done &&
$K/kafka-metadata-quorum.sh --bootstrap-server $L describe --status && ss -Hltn | grep -q '127.0.0.1]*:9093 ' &&
! ss -Hltn | grep -E ':909[34] ' | grep -qv '127\.0\.0\.1' && echo "9093 and 9094 are loopback only" ||
echo "STOP - run: journalctl -u kafka -n 50"
fi

Only if the first run ends in the journalctl STOP line (never after already formatted or started), nothing is stored yet: systemctl disable --now kafka; rm -f /etc/systemd/system/kafka.service; rm -rf /var/lib/kafka/* and run it again.

An application login with its own ACLs

1. In a root shell: K=/opt/kafka/bin L=127.0.0.1:9094; install -m 600 /dev/null /root/app1.pw && openssl rand -hex 24 > /root/app1.pw.
2. Create the SCRAM login (for its few seconds of runtime, ps can see the password): $K/kafka-configs.sh --bootstrap-server $L --entity-type users --entity-name app1 --add-config "SCRAM-SHA-512=[password=$(cat /root/app1.pw)]" --alter.
3. Grant topics starting orders.: $K/kafka-acls.sh --bootstrap-server $L --add --allow-principal User:app1 --producer --topic orders. --resource-pattern-type prefixed, then again with --consumer --topic orders. --group app1- in place of --producer --topic orders..
4. Create a topic: $K/kafka-topics.sh --bootstrap-server $L --create --topic orders.events --partitions 3 --replication-factor 1.
5. On the app host, which needs the same Kafka download and Java for the test, app1.properties (mode 600) holds sasl.mechanism=SCRAM-SHA-512, security.protocol=SASL_SSL, sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="app1" password="..."; with the generated password, then ssl.truststore.type=PEM and ssl.truststore.location= a copy of /etc/kafka/broker-cert.pem.
6. Admit that host with ufw allow from 203.0.113.10 to any port 9092 proto tcp, using its real address, and test: kafka-console-producer.sh --topic orders.events --bootstrap-server kafka.example.net:9092 --command-config app1.properties, with your H.

From one node to a cluster

The KRaft page asks for 3 or more controllers (N failures need 2N + 1) in processes separate from the brokers. It gives no way to convert this combined node: build the cluster fresh and move producers over. Generate one cluster ID and pass it with -t when formatting every node: the first controller with --standalone, later ones with --no-initial-controllers followed by kafka-metadata-quorum.sh add-controller, brokers with no extra flag. Every node lists the controllers in controller.quorum.bootstrap.servers. Once 9093 and replication cross machines, run them on SASL_SSL, or SSL with ssl.client.auth=required: SSL without client authentication leaves every peer as User:ANONYMOUS, so replace super.users with the brokers' and controllers' own principals. Open 9093 to cluster members only, replication to brokers only, by IP. Raise internal topics to replication factor 3 and set min.insync.replicas, the in-sync replicas an acks=all write needs.

Retention, upgrades, backups, licence

Retention defaults to 168 hours with no size cap (log.retention.bytes -1); topics can override both. Watch df -h /var/lib/kafka and /opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server 127.0.0.1:9094 describe --replication. To upgrade, unpack the new release beside the old one with the same checksum test, ln -sfn /opt/kafka-NEW /opt/kafka, systemctl restart kafka, and once it behaves finalize: K=/opt/kafka/bin L=127.0.0.1:9094; $K/kafka-features.sh --bootstrap-server $L upgrade --release-version and the target release. Finalizing raises the metadata version, which the upgrade notes say cannot be downgraded across versions with metadata changes: keep the old directory until then. Since 4.4 a broker cannot be older than the kafka-storage tool that formatted its directory. The certificate lapses after 825 days: openssl x509 -enddate -noout -in /etc/kafka/broker-cert.pem prints the date; replace it and redistribute broker-cert.pem before then.

Durability in Kafka comes from replicas, and one node has none. Keep the identity clients depend on: umask 077; f=/root/kafka-conf-$(date +%F-%H%M%S).tgz; tar -czf $f -C / etc/kafka root/kafka-secrets && tar -tzf $f >/dev/null && echo "$f ok", then copy it off the machine yourself and keep it private: it holds the keystore and its password, plus config, but not topic data, users or ACLs. To rebuild with the same H: run the install block, tar -xzf the archive over it with -C /, skip the configuration block, run the start block, then recreate topics, logins and ACLs.

Kafka is under the Apache License 2.0 (the repository's LICENSE); LICENSE-binary lists bundled components under EPL 2.0, CDDL 1.1, MIT and other terms. Our general reading, not legal advice: it is permissive, so running it for yourself or behind a paid service owes nobody source code. Passing copies on, changed or not, means including the licence, keeping the NOTICE attributions and copyright notices, marking changed files and honouring the bundled licences. Its patent licence ends for anyone who sues claiming the work infringes a patent; it grants no trademark rights.