Kafka has changed a lot in the last few years. Zookeeper has been removed, one Kafka docker container maker has stopped making containers, and the configurations have changed.

Without much futher ado, the most recent version of a hosted Kafka container can be used with the following docker compose:

services:
  kafka:
    image: 'bitnami/kafka:latest'
    ports:
      - '9092:9092'
      - '9094:9094'
    environment:
      - KAFKA_CFG_NODE_ID=0
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,EXTERNAL://localhost:9094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka:9093
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
      - ALLOW_PLAINTEXT_LISTENER=yes
  init-kafka:
    image: 'bitnami/kafka:latest'
    depends_on:
      - kafka
    entrypoint: [ '/bin/sh', '-c' ]
    command: |
      "
      # blocks until kafka is reachable
      kafka-topics.sh --bootstrap-server kafka:9092 --list

      echo -e 'Creating kafka topics'
      kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists --topic incomingdata --replication-factor 1 --partitions 1
      kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists --topic outgoingdata --replication-factor 1 --partitions 1

      echo -e 'Successfully created the following topics:'
      kafka-topics.sh --bootstrap-server kafka:9092 --list
      "

Some things to note about this:

  1. Kafka works by binding on specific hosts. Internally this docker compose will connect with the hostname of “kakfa” and port 9092.

  2. For ‘kafka-init’ the same container is reused in order to execute setup commands on start. This docker-compose is intended for a local development machine usage.

  3. KAFKA_CFG_LISTENERS, KAFKA_CFG_ADVERTISED_LISTENERS, and KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP are configured to add external host support for accessing the Kafka broker outside of docker. It does so on port 9094.

  4. The configuration: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR allows for you to use the created topics without a replication value being set.

  5. There’s no ZooKeeper. As of version 4.0, there is no more requirement to use Zookeeper.

  6. The wurstmeister docker image is no longer being produced. Most of the Kafka docker examples online tend to use the wurstmeister version. They stopped releasing images as of 2023.

  7. This uses the Bitnami version of Kafka which is based on the open source Kafka project. The Confluent version is built on top of the commercial Confluence Platform and has hooks for their licensed version.

Also, something to note:

  1. The kafkacat utility has become kcat. On Arch AUR it can be installed via kcat-cli.
  2. Also consider looking at my awesome-kafka repo and contributing content.