[bitnami/kafka] latest update crash

[benitogf]

Name and Version

bitnami/kafka

What architecture are you using?

None

What steps will reproduce the bug?

Using the latest build and this compose file:

version: '3.3'
services:
  kafka:
    image: bitnami/kafka:latest
    hostname: 'xtable'
    ports:
      - "9092:9092"
    environment:
      - TZ=Asia/Hong_Kong
      - ALLOW_PLAINTEXT_LISTENER=yes
    volumes:
      - /var/lib/bitnami/kafka:/bitnami/kafka

getting:

kafka haven't been configured to work in either raft or zookeper mode

and crash

rollback to 3.5.1-debian-11-r3 works

What is the expected behavior?

work with the same configuration? what changes are needed?

What do you see instead?

application crash

Additional information

No response

[migruiz4]

Hi @benitogf,

The latest update of the bitnami/kafka image requires additional env variables to be deployed.

In order to continue support for both Zookeeper and KRaft modes, the image no longer selects a default mode but has to be configured with some minimum settings depending on the desired mode.

To continue using KRaft mode, please use the following docker-compose:

containers/bitnami/kafka/docker-compose.yml

Lines 1 to 26 in 0039bc1

	# Copyright VMware, Inc.
	# SPDX-License-Identifier: APACHE-2.0
	version: "2"
	services:
	kafka:
	image: docker.io/bitnami/kafka:3.5
	ports:
	- "9092:9092"
	volumes:
	- "kafka_data:/bitnami"
	environment:
	# KRaft settings
	- KAFKA_CFG_NODE_ID=0
	- KAFKA_CFG_PROCESS_ROLES=controller,broker
	- KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka:9093
	# Listeners
	- KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093
	- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://:9092
	- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
	- KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
	- KAFKA_CFG_INTER_BROKER_LISTENER_NAME=PLAINTEXT
	volumes:
	kafka_data:
	driver: local

I would recommend taking a look at the 'Notable changes' section of the README for more details: https://github.com/bitnami/containers/blob/main/bitnami/kafka/README.md#351-debian-11-r4-341-debian-11-r50-332-debian-11-r176-and-323-debian-11-r161

[migruiz4]

I'm sorry for the inconvenience, but due to our release format being based on the upstream version + image revision we don't have a method to control major releases.

For production environments, we recommend not using rolling tags such as latest or 3.5 and instead using immutable tags such as bitnami/kafka:3.5.1-debian-11-r4. This will give you better control when upgrading.

[benitogf]

Thanks @migruiz4 I liked better the auto Kraft 😄 wonder if it might come back when zookeper support is dropped? or the plan is to keep zookeeper ?

[migruiz4]

Yes, It will default to KRaft again when Zookeeper support is dropped.

When Kafka 4.0 is released, the major version that will drop Zookeeper support definitely, we will reconsider the default values for the bitnami/kafka image.

Although, Kafka gives each version a 3-years support lifecycle, meaning that 3.5 will be supported until at least August 2026, it is also possible that we may need to keep some of those explicit values for a bit longer.

It will all depends on what are the Kafka default values for that new 4.0 major version.

[dspaeth-breuni]

I've recently encountered an issue which I believe could be a result of changes made to the bitnami/kafka:3 Docker image. Our project uses a pinned version in a Docker-compose file, which ideally should ensure a consistent environment. However, it seems that the image behaves differently now, possibly due to some changes introduced in the image itself.

The situation becomes particularly confusing because the behavior isn't consistent across environments. In our local development setup, the application works perfectly fine due to Docker's image caching. However, when we try to deploy the same setup within our pipeline, it fails.

One potential cause for this could be the introduction of new required environment variables. If this is indeed the case, it would be immensely helpful if an error message could be output stating the need for these new variables.

Introducing breaking changes to an existing Docker image version can be quite disruptive, especially if there's no clear communication or error messages guiding us on how to adjust our configurations.

Could you please provide some insights into this issue? We'd appreciate any assistance on whether any changes have been made that might explain this behavior and how we can update our setup to work with the current state of the image.

Thank you.

[migruiz4]

Hi @dspaeth-breuni,

As previously mentioned, for production environments we recommend using immutable tags, such as bitnami/kafka:3.5.1-debian-11-r7 instead of rolling tags, e.g. bitnami/kafka:3.5.
That would ensure the same behavior between environments and better control of updates.

It is difficult to determine the cause of the issue as you haven't shared the error logs, but considering your pipelines would be pulling the latest images, it is possible that the error is caused by the environment variables change.

We have updated the bitnami/kafka README.md as well as the example docker-compose.yaml.

In the README.md you can find all the documentation required to deploy your Bitnami Kafka containers, as well as the Notable changes section that describe all the changes introduced in the latest release.

Note: we are currently facing an issue where the docker-compose-cluster.yaml has not yet being updated, please do not use that file as reference. For a cluster example please use this instead as reference for each node type.

[dspaeth-breuni]

Hi @migruiz4 ,

Thank you for your detailed advice on using immutable tags in production environments. I appreciate your recommendation and have taken it into consideration for future deployments.

In regard to our issue, it seems that the problem originated from the absence of specific variables including KAFKA_CFG_NODE_ID and KAFKA_CFG_CONTROLLER_QUORUM_VOTERS, among others. The error message "Kafka hasn't been configured to work in either Raft or Zookeper mode" was followed by a warning "Kafka has been configured with a PLAINTEXT listener". This sequence initially led us to believe the problem was an out-of-memory error, rather than the real issue at hand.

Another element that compounded our issue was image caching on our local machines. This concealed the problems occurring remotely where fresh images were being pulled, thus adding another layer of complexity to the problem.

As for semantic versioning, I would like to clarify if this is not the intended goal? We are currently exploring different versioning strategies and are committed to adopting the best practices in this domain.

Thank you once again for updating the bitnami/kafka README.md and the example docker-compose.yaml. We look forward to your response and further assistance.

[erich-ef]

I'm sorry for the inconvenience, but due to our release format being based on the upstream version + image revision we don't have a method to control major releases.

For production environments, we recommend not using rolling tags such as latest or 3.5 and instead using immutable tags such as bitnami/kafka:3.5.1-debian-11-r4. This will give you better control when upgrading.

I was using 3.5.1 but I was still affected by this behavior change.

[migruiz4]

Hi @erich-ef,

bitnami/kafka:3.5.1 is also a rolling tag, immutable tags have the format bitnami/kafka:3.5.1-debian-11-rX.

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[chasebenedict1]

@migruiz4 I am getting the same issue as mentioned by other people above:

$ docker run bitnami/kafka:3.5.1-debian-11-r7
kafka 13:59:08.99 
kafka 13:59:08.99 Welcome to the Bitnami kafka container
kafka 13:59:08.99 Subscribe to project updates by watching https://github.com/bitnami/containers
kafka 13:59:09.00 Submit issues and feature requests at https://github.com/bitnami/containers/issues
kafka 13:59:09.00
kafka 13:59:09.01 INFO  ==> ** Starting Kafka setup **
kafka 13:59:09.11 ERROR ==> Kafka haven't been configured to work in either Raft or Zookeper mode. Please make sure at least one of the modes is configured.
kafka 13:59:09.12 WARN  ==> Kafka has been configured with a PLAINTEXT listener, this setting is not recommended for production environments.

I pulled an immutable tag like suggested and everything. Not sure what else I could do to make this work.

[benitogf]

@chasebenedict1 hey this is the entry I'm running on compose now:

  kafka:
    image: bitnami/kafka:3.5.1-debian-11-r3
    hostname: 'bitnami'
    ports:
      - "9092:9092"
    environment:
      - ALLOW_PLAINTEXT_LISTENER=yes
    volumes:
      - /var/lib/bitnami/kafka:/bitnami/kafka

[migruiz4]

Hi @benitogf @chasebenedict1,

As I mentioned in my previous comment #43120 (comment).

In order to improve compatibility with both Zookeeper mode and KRaft more, the bitnami/kafka image now requires additional minimum configuration to run.

In case you are looking for docker run examples, this section of the README.md should help you:

docker run -d --name kafka-server --hostname kafka-server \
    -e KAFKA_CFG_NODE_ID=0 \
    -e KAFKA_CFG_PROCESS_ROLES=controller,broker \
    -e KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093 \
    -e KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT \
    -e KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka-server:9093 \
    -e KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER \
    bitnami/kafka:latest

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[niemyjski]

What is the minimal changes to get this to run, upgrading from 3.4.0 to 3.6.1 broke.

Looks like I was only missing the line

KAFKA_CFG_PROCESS_ROLES=controller,broker

[migruiz4]

Hi @niemyjski,

I think Kafka 3.6 introduced some breaking changes upstream that affect upgrades, maybe the following guide helps you: https://kafka.apache.org/36/documentation/streams/upgrade-guide

If you are affected by both the image refactor and 3.6 upgrade, I would recommend upgrading to 3.5 first, then perform upgrade to 3.6 according to their guide.