docs: add dynamic workers documentation

website/docs/guides/consumers/built-in-workers-algorithms/_category_.json

@@ -0,0 +1,8 @@
+{
+    "label": "Built-in Dynamic Workers Algorithms",
+    "position": 4,
+    "link": {
+      "type": "generated-index"
+    }
+  }
+

website/docs/guides/consumers/built-in-workers-algorithms/consumer-lag-based-worker-balancer.md

@@ -0,0 +1,52 @@
+---
+sidebar_position: 1
+---
+
+# Consumer Lag-Based Worker Balancer
+
+The `WithConsumerLagWorkerBalancer` method in KafkaFlow is a powerful feature that allows you to dynamically calculate the number of workers for each application instance based on the consumer's lag. This feature is designed to optimize message processing, especially in scenarios where some application instances have more partitions and naturally need to deal with higher message throughput. By adjusting the number of workers based on message lag, this feature helps ensure efficient message processing and load balancing across all application instances.
+
+In this documentation, you'll find an overview of how this feature works, a use case example, and step-by-step instructions on how to configure it.
+
+:::info
+This method replaces the call of `WithWorkersCount` in the consumer's setup.
+:::
+
+## Use Case Example
+
+### Balancing Message Lag Across Application Instances
+
+Consider a scenario where you have a Kafka consumer application deployed in a distributed environment. In this environment, some application instances may have more partitions assigned to them, resulting in a higher message throughput potential. However, variations in message lag across partitions can lead to uneven workloads.
+
+Here's how the `WithConsumerLagWorkerBalancer` feature can help:
+
+- **Total Workers**: You specify the `totalWorkers`, which represents the total number of workers to be distributed across all application instances.
+
+- **Minimum and Maximum Instance Workers**: You set the `minInstanceWorkers` and `maxInstanceWorkers` parameters, defining the minimum and maximum number of workers allowed for each application instance.
+
+- **Evaluation Interval**: You define an `evaluationInterval` that determines how often the number of workers should be recalculated based on the consumer's lag.
+
+With this configuration, the `WithConsumerLagWorkerBalancer` feature dynamically adjusts the number of worker threads for each application instance based on the lag in the Kafka topic. Application instances with partitions experiencing higher lag will have more workers allocated to help balance the message lag across all instances.
+
+### Benefits in Elastic Infrastructure
+
+This feature is particularly valuable in elastic infrastructure environments like Kubernetes, where you need to manage the total number of workers across all application instances to prevent overloading dependencies. By dynamically adjusting worker counts, the feature ensures that each instance scales its resources efficiently, improving overall application performance and resource utilization.
+
+## How to Configure
+
+Configuring Consumer Lag-Based Worker Balancer is straightforward with the fluent interface provided by KafkaFlow. Here's a simple example:
+
+```csharp
+.AddConsumer(
+    consumer => consumer
+        ...
+        .WithConsumerLagWorkerBalancer(
+            50, // The total number of workers to be distributed across all application instances.
+            3,  // The minimum number of workers for each application instance.
+            20) // The maximum number of workers for each application instance.
+        ...
+        )
+)
+```
+
+With this configuration, KafkaFlow will dynamically adjust the number of worker threads for each application instance, ensuring efficient message processing while considering the lag in the Kafka topic. This feature provides a powerful way to optimize resource allocation in your Kafka-based applications, making them more adaptive to varying message loads and distribution across partitions.

website/docs/guides/consumers/dynamic-workers-configuration.md

@@ -0,0 +1,46 @@
+---
+sidebar_position: 3
+---
+
+# Dynamic Worker Configuration
+
+In this section, we will learn how Dynamic Worker Configuration works and how to configure it. In version 3 of KafkaFlow, we have introduced a new feature that allows you to dynamically configure the number of workers for a specific consumer based on a custom algorithm. This feature enables greater flexibility in managing worker threads, as each application instance can have a different number of workers, depending on the algorithm you define.
+
+This documentation page explains how to use and configure this feature effectively.
+
+## Use Case Example
+
+Imagine a scenario where your application's message load varies throughout the day. During peak hours, you want to allocate more worker threads to process messages quickly, and during off-peak hours, you want to reduce the number of worker threads to save resources. This dynamic adjustment can be achieved using the custom dynamic worker configuration feature.
+
+## How to Configure
+
+Configuring Dynamic Worker Configuration is straightforward with the fluent interface provided by KafkaFlow. Here's a simple example:
+
+```csharp
+.AddConsumer(
+    consumer => consumer
+        ...
+        .WithWorkersCount(
+            (context, resolver) =>
+            {
+                // Implement a custom logic to calculate the number of workers
+                if (IsPeakHour(DateTime.UtcNow))
+                {
+                    return Task.FromResult(10); // High worker count during peak hours
+                }
+                else
+                {
+                    return Task.FromResult(2); // Lower worker count during off-peak hours
+                }
+            }, 
+            TimeSpan.FromMinutes(15)); // Evaluate the worker count every 15 minutes
+        ...
+        )
+)
+```
+
+In this example, the number of worker threads is adjusted dynamically based on whether it's a peak hour or off-peak hour. You can implement your custom logic in the `WithWorkersCount`` method to suit your application's specific requirements.
+
+That's it! Your KafkaFlow consumer will now dynamically adjust the number of worker threads based on your custom logic and the specified evaluation interval.
+
+This feature provides a powerful way to optimize resource utilization and throughput in your Kafka-based applications.