[bitnami/kafka] Getting the Keystore SubjectAltNames do not match error while updating the kafka certificate dynamically.

[irparajababu]

Name and Version

bitnami/kafka:3.3

What architecture are you using?

None

What steps will reproduce the bug?

/opt/kafka/bin/kafka-configs.sh --bootstrap-server $NODE_NAME:9093 --entity-name $NODE_ID --entity-type brokers --add-config listener.name.SSL.ssl.keystore.location=/bitnami/kafka/config/certs/kafka.keystore.jks --alter --command-config /opt/client.properties

What is the expected behavior?

Certificate should be update without any issues

What do you see instead?

Getting the below error while updating the kafka certificate dynamically.

Command:

/opt/kafka/bin/kafka-configs.sh --bootstrap-server $NODE_NAME:9093 --entity-name $NODE_ID --entity-type brokers --add-config listener.name.SSL.ssl.keystore.location=/bitnami/kafka/config/certs/kafka.keystore.jks --alter --command-config /opt/client.properties

Is there any way to disable to the certificate validation while updating the certificate dynamically using above command.

ERROR:

ERROR Encountered metadata application fault: Error updating broker with new configuration: listener.name.SSL.ssl.keystore.location -> /bitnami/kafka/config/certs/kafka.keystore.jks in MetadataDelta up to 20639134 (org.apache.kafka.server.fault.LoggingFaultHandler)
org.apache.kafka.common.config.ConfigException: Validation of dynamic config update of SSLFactory failed: org.apache.kafka.common.config.ConfigException: Keystore SubjectAltNames do not match:  existing={alias=kafka-broker, SAN=[[test, test2 test3]], new={alias=kafka-non-prod, SAN=[[ test1]]}
        at org.apache.kafka.common.security.ssl.SslFactory.createNewSslEngineFactory(SslFactory.java:184)
        at org.apache.kafka.common.security.ssl.SslFactory.validateReconfiguration(SslFactory.java:116)
        at org.apache.kafka.common.network.SslChannelBuilder.validateReconfiguration(SslChannelBuilder.java:88)
        at kafka.server.DynamicBrokerConfig.processReconfigurable(DynamicBrokerConfig.scala:609)
        at kafka.server.DynamicBrokerConfig.processListenerReconfigurable(DynamicBrokerConfig.scala:597)
        at kafka.server.DynamicBrokerConfig.$anonfun$reloadUpdatedFilesWithoutConfigChange$4(DynamicBrokerConfig.scala:339)
        at kafka.server.DynamicBrokerConfig.$anonfun$reloadUpdatedFilesWithoutConfigChange$4$adapted(DynamicBrokerConfig.scala:335)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
        at kafka.server.DynamicBrokerConfig.$anonfun$reloadUpdatedFilesWithoutConfigChange$1(DynamicBrokerConfig.scala:335)
        at kafka.server.DynamicBrokerConfig.reloadUpdatedFilesWithoutConfigChange(DynamicBrokerConfig.scala:335)
        at kafka.server.metadata.BrokerMetadataPublisher.reloadUpdatedFilesWithoutConfigChange(BrokerMetadataPublisher.scala:329)
        at kafka.server.metadata.BrokerMetadataPublisher.$anonfun$publish$15(BrokerMetadataPublisher.scala:255)
        at java.base/java.util.HashMap$KeySet.forEach(HashMap.java:929)
        at kafka.server.metadata.BrokerMetadataPublisher.$anonfun$publish$14(BrokerMetadataPublisher.scala:215)
        at kafka.server.metadata.BrokerMetadataPublisher.$anonfun$publish$14$adapted(BrokerMetadataPublisher.scala:214)
        at scala.Option.foreach(Option.scala:407)
        at kafka.server.metadata.BrokerMetadataPublisher.publish(BrokerMetadataPublisher.scala:214)
        at kafka.server.metadata.BrokerMetadataListener.kafka$server$metadata$BrokerMetadataListener$$publish(BrokerMetadataListener.scala:343)
        at kafka.server.metadata.BrokerMetadataListener$HandleCommitsEvent.$anonfun$run$2(BrokerMetadataListener.scala:148)
        at kafka.server.metadata.BrokerMetadataListener$HandleCommitsEvent.$anonfun$run$2$adapted(BrokerMetadataListener.scala:148)
        at scala.Option.foreach(Option.scala:407)
        at kafka.server.metadata.BrokerMetadataListener$HandleCommitsEvent.run(BrokerMetadataListener.scala:148)
        at org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:121)
        at org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:200)
        at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:173)
        at java.base/java.lang.Thread.run(Thread.java:829)

Additional information

No response

[Mauraza]

Hi @irparajababu,

Could you share the values are you using to deploy the chart of Kafka?

[irparajababu]

Hi @irparajababu,

Could you share the values are you using to deploy the chart of Kafka?

Hi @Mauraza Please find the below values.

- TAKE_FILE_OWNERSHIP="yes"
- KAFKA_CFG_DELETE_TOPIC_ENABLE="true"
- KAFKA_CFG_MIN_INSYNC_REPLICAS="2"
- KAFKA_CFG_NUM_NETWORK_THREADS="3"
- KAFKA_CFG_NUM_IO_THREADS="8"
- KAFKA_CFG_SOCKET_SEND_BUFFER_BYTES="102400"
- KAFKA_CFG_SOCKET_RECEIVE_BUFFER_BYTES="102400"
- KAFKA_CFG_SOCKET_REQUEST_MAX_BYTES="104857600"
- KAFKA_CFG_NUM_PARTITIONS="6"
- KAFKA_CFG_DEFAULT_REPLICATION_FACTOR="3"
- KAFKA_CFG_NUM_RECOVERY_THREADS_PER_DATA_DIR="1"
- KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR="3"
- KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTOR="3"
- KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISR="2"
- KAFKA_CFG_LOG_RETENTION_HOURS="168"
- KAFKA_CFG_LOG_SEGMENT_BYTES="1073741824"
- KAFKA_CFG_LOG_RETENTION_CHECK_INTERVAL_MS="300000"
- KAFKA_CFG_ZOOKEEPER_CONNECTION_TIMEOUT_MS="6000"
- KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS="3000"
- ALLOW_PLAINTEXT_LISTENER="no"
- KAFKA_INTER_BROKER_USER="admin"
- KAFKA_INTER_BROKER_PASSWORD="{{ kafka_user_password }}"
- KAFKA_CLIENT_USERS="user"
- KAFKA_CLIENT_PASSWORDS="{{ kafka_user_password }}"
- KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE="false"
- KAFKA_HEAP_OPTS="-Xms4G -Xmx4G"
- KAFKA_CFG_LISTENERS=SSL://:9093
- KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL=SSL
- KAFKA_INTER_BROKER_LISTENER_NAME=SSL
- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=SSL:SSL
- KAFKA_CFG_SECURITY_PROTOCOL=SSL
- KAFKA_CFG_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM=
- KAFKA_CFG_SSL_KEYSTORE_PASSWORD={{ kafka_ssl_password }}
- KAFKA_CFG_SSL_KEY_PASSWORD={{ kafka_ssl_password }}
- KAFKA_CFG_SSL_KEYSTORE_TYPE=JKS
- KAFKA_CFG_SSL_KEYSTORE_LOCATION=/bitnami/kafka/config/certs/kafka.keystore.jks
- KAFKA_CFG_SSL_TRUSTSTORE_LOCATION=/bitnami/kafka/config/certs/kafka.truststore.jks
- KAFKA_ZOOKEEPER_TLS_TRUSTSTORE_FILE=/bitnami/kafka/config/certs/zookeeper.truststore.jks
- KAFKA_CFG_SSL_TRUSTSTORE_PASSWORD={{ kafka_ssl_password }}
- KAFKA_ZOOKEEPER_PROTOCOL=SSL
- KAFKA_ZOOKEEPER_USER="user"
- KAFKA_ZOOKEEPER_PASSWORD={{ kafka_user_password }}
- KAFKA_ZOOKEEPER_TLS_KEYSTORE_PASSWORD={{ kafka_ssl_password }}
- KAFKA_ZOOKEEPER_TLS_TRUSTSTORE_PASSWORD={{ kafka_ssl_password }}
- KAFKA_ZOOKEEPER_TLS_TYPE=JKS
- KAFKA_CFG_SSL_TRUSTSTORE_TYPE=JKS
- KAFKA_CFG_AUTHORIZER_CLASS_NAME=kafka.security.authorizer.AclAuthorizer
- KAFKA_CFG_ALLOW_EVERYONE_IF_NO_ACL_FOUND=true
- KAFKA_CFG_SUPER_USERS=User:CN=localhost,OU=home,O=home,L=bellevue,ST=wa,C=us
- KAFKA_OPTS=-Djavax.net.debug=all

[Mauraza]

Hi @irparajababu,

What images are you using? because in the container bitnami/kafka the command is in /opt/bitnami/kafka/bin/kafka-configs.sh

[irparajababu]

Hi @irparajababu,

What images are you using? because in the container bitnami/kafka the command is in /opt/bitnami/kafka/bin/kafka-configs.sh

@Mauraza We are using the kafka client CLI for connecting the kafka broker and update the certificate and kafka brokers are running with bitnami/kafka:3.3 version

[Mauraza]

Hi @irparajababu,

could you add the environmental variable BITNAMI_DEBUG=true?
Could you check the client of Kafka with ⬇️?

To create a pod that you can use as a Kafka client run the following commands:

    kubectl run kafka-client --restart='Never' --image docker.io/bitnami/kafka:3.5.0-debian-11-r4 --namespace NAMESPACE --command -- sleep infinity
    kubectl exec --tty -i kafka-client --namespace default -- bash

[irparajababu]

let me try and update you

[Mauraza]

Hi @irparajababu,

Did you try it?

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.