[bitnami/kafka] Kafka is crashing specifically in docker swarm mode

[mvenkatsriram]

Name and Version

bitnami/kafka:3.4.1-debian-11-r5

What architecture are you using?

None

What steps will reproduce the bug?

Following docker compose file works fine in docker compose up -d. But it fails in swarm mode

version: '3.7'
services:
  kafka:     
    image: bitnami/kafka:3.4.1-debian-11-r5
    # image: bitnami/kafka
    hostname: kafka
    ports:
      - '20091:9091'      
      - '20092:9092'      
      - '20093:9093' 
    environment:      
      - KAFKA_ENABLE_KRAFT=yes      
      - TAKE_FILE_OWNERSHIP=yes
      - KAFKA_CFG_DELETE_TOPIC_ENABLE=true
      - KAFKA_CFG_MIN_INSYNC_REPLICAS=2
      - KAFKA_CFG_NUM_NETWORK_THREADS=3
      - KAFKA_CFG_NUM_IO_THREADS=8
      - KAFKA_CFG_SOCKET_SEND_BUFFER_BYTES=102400
      - KAFKA_CFG_SOCKET_RECEIVE_BUFFER_BYTES=102400
      - KAFKA_CFG_SOCKET_REQUEST_MAX_BYTES=369295618      
      - KAFKA_CFG_NUM_RECOVERY_THREADS_PER_DATA_DIR=1      
      - KAFKA_CFG_LOG_SEGMENT_BYTES=1073741824
      - KAFKA_CFG_LOG_RETENTION_CHECK_INTERVAL_MS=300000
      - KAFKA_CFG_INITIAL_BROKER_REGISTRATION_TIMEOUT_MS=240000
      - KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS=3000
      - ALLOW_PLAINTEXT_LISTENER=yes     
      - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
      - KAFKA_HEAP_OPTS=-Xms4G -Xmx4G
      - KAFKA_CFG_LISTENERS=SSL://:9092,CONTROLLER://:9093
      - KAFKA_CFG_SSL_KEYSTORE_PASSWORD=serversecret
      - KAFKA_CFG_SSL_KEY_PASSWORD=serversecret
      - KAFKA_CFG_SSL_KEYSTORE_TYPE=JKS
      - KAFKA_CFG_SSL_KEYSTORE_LOCATION=/opt/bitnami/kafka/config/certs/kafka.keystore.jks
      - KAFKA_CFG_SSL_TRUSTSTORE_LOCATION=/opt/bitnami/kafka/config/certs/kafka.truststore.jks
      - KAFKA_CFG_SSL_TRUSTSTORE_PASSWORD=serversecret
      - KAFKA_CFG_SSL_TRUSTSTORE_TYPE=JKS
      #- KAFKA_OPTS=-Djavax.net.debug=all      
      - KAFKA_CFG_ADVERTISED_LISTENERS=SSL://10.169.38.3:20092
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=SSL:SSL,CONTROLLER:PLAINTEXT
      - KAFKA_CFG_SECURITY_INTER_BROKER_PROTOCOL=PLAINTEXT
      - KAFKA_KRAFT_CLUSTER_ID=ZTI3MmVmYTkwMDRmNDEzOT
      #- BITNAMI_DEBUG=yes
      - KAFKA_CFG_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM=
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=SSL
      - javax.net.ssl.trustStore=/opt/bitnami/kafka/config/certs/kafka.truststore.jks
      - javax.net.ssl.trustStorePassword=serversecret
      - enable.ssl.certificate.verification=false
      - ssl.endpoint.identification.algorithm=
      # - KAFKA_CFG_LOG_RETENTION_HOURS = 1
      - log.retention.ms = 300000
      - log.roll.ms = 300000
    secrets:       
      - ssl_password
    volumes:
      - "kafka_data:/bitnami"
      - "/abc_rpt/rptapp/keystore/kafka.keystore.jks:/opt/bitnami/kafka/config/certs/kafka.keystore.jks:ro"
      - "/abc_rpt/rptapp/truststore/kafka.truststore.jks:/opt/bitnami/kafka/config/certs/kafka.truststore.jks:ro"
      - /etc/localtime:/etc/localtime:ro

volumes:  
  kafka_data:
    driver: local

secrets:
  ssl_password:
    file: ./ssl_password.txt

networks:
  default:
    ipam:
      driver: default
      config:
        - subnet: "192.168.2.0/24"

What is the expected behavior?

When running with following command

sudo docker stack deploy --prune -c docker-compose.yml abcapp

It should work

What do you see instead?

Following error

[abc@newserver ~]$ sudo docker logs 0449f4a1900d
kafka 13:47:00.70
kafka 13:47:00.70 Welcome to the Bitnami kafka container
kafka 13:47:00.71 Subscribe to project updates by watching https://github.com/bitnami/containers
kafka 13:47:00.71 Submit issues and feature requests at https://github.com/bitnami/containers/issues
kafka 13:47:00.71
kafka 13:47:00.71 INFO  ==> ** Starting Kafka setup **
kafka 13:47:00.80 WARN  ==> You set the environment variable ALLOW_PLAINTEXT_LISTENER=yes. For safety reasons, do not use this flag in a production environment.
kafka 13:47:00.81 INFO  ==> Initializing Kafka...
kafka 13:47:00.82 INFO  ==> No injected configuration files found, creating default config files
kafka 13:47:01.27 INFO  ==> Initializing KRaft...
kafka 13:47:01.28 INFO  ==> Formatting storage directories to add metadata...
kafka 13:47:02.71 INFO  ==> ** Kafka setup finished! **

kafka 13:47:02.73 INFO  ==> ** Starting Kafka **
[2023-08-02 08:17:03,488] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2023-08-02 08:17:03,803] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2023-08-02 08:17:03,913] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
java.lang.IllegalArgumentException: No MetadataVersion with metadata version 11
        at org.apache.kafka.server.common.MetadataVersion.fromFeatureLevel(MetadataVersion.java:412)
        at org.apache.kafka.metadata.bootstrap.BootstrapMetadata.recordToMetadataVersion(BootstrapMetadata.java:70)
        at org.apache.kafka.metadata.bootstrap.BootstrapMetadata.fromRecords(BootstrapMetadata.java:54)
        at org.apache.kafka.metadata.bootstrap.BootstrapDirectory.readFromBinaryFile(BootstrapDirectory.java:106)
        at org.apache.kafka.metadata.bootstrap.BootstrapDirectory.read(BootstrapDirectory.java:78)
        at kafka.server.KafkaRaftServer$.initializeLogDirs(KafkaRaftServer.scala:171)
        at kafka.server.KafkaRaftServer.<init>(KafkaRaftServer.scala:56)
        at kafka.Kafka$.buildServer(Kafka.scala:85)
        at kafka.Kafka$.main(Kafka.scala:93)
        at kafka.Kafka.main(Kafka.scala)

Additional information

No response

[Mauraza]

Hi @mvenkatsriram,

Could you try the last version of Kafka?

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.