Protobuf producer serialization slow

[bichselb]

Description

I am using the protobuf producer outlined here. Unfortunately, its serialization is so slow that it a bottleneck in my whole system. Specifically, it takes 1.6s to serialize 1 million messages, even though pure protobuf can serialize the same 1 million messages in 100ms.

How can I speed up serialization to avoid this 15x overhead? I would like to avoid parallelization.

How to reproduce

I put my whole experimental setup here: https://github.com/bichselb/confluent-kafka-go-performance/tree/serialization-slow

My serialization is a straight-forward adaptation of the protobuf producer example, except that I enabled CacheSchemas (without this, the code is another order of magnitude slower):

func serializeWithSchemaregistry(nMessages int) {
	registry, err := schemaregistry.NewClient(schemaregistry.NewConfig("http://schema-registry:8081"))
	if err != nil {
		fmt.Printf("Error creating schema registry client...\n")
		panic(err)
	}
	defer registry.Close()

	serializer, err := protobuf.NewSerializer(
		registry,
		serde.ValueSerde,
		&protobuf.SerializerConfig{
			SerializerConfig: *serde.NewSerializerConfig(),
			CacheSchemas: true,  // big difference for performance
		},
	)
	if err != nil {
		fmt.Printf("Error creating serializer...\n")
		panic(err)
	}
	defer serializer.Close()

	start := time.Now()
	topic := "mytopic"
	for i := 0; i < nMessages; i++ {
		message := pb.MyMessage{
			A: int32(i),
			B: int32(i),
		}

		payload, err := serializer.Serialize(topic, &message)
		if err != nil {
			fmt.Printf("Error serializing message...\n")
			panic(err)
		}
		// Pseudo-use of payload to prevent compiler optimizations
		_ = copy(payload, payload)
	}
	fmt.Printf("Protobuf+schemaregistry: Serialized %d messages in %v\n", nMessages, time.Since(start))
}

Checklist

Please provide the following information:

• confluent-kafka-go version: v2.5.4
• Apache Kafka broker version: confluentinc/cp-kafka:7.7.1
• Client configuration: Not needed to showcase this issue
• Operating system: Docker compose on an Ubuntu system
• Provide client logs: See https://github.com/bichselb/confluent-kafka-go-performance/blob/serialization-slow/results.log
• Provide broker log excerpts: See https://github.com/bichselb/confluent-kafka-go-performance/blob/serialization-slow/results.log