Add support for scaling based on Nakadi

[mikkeloscar]

This adds support for scaling based on Nakadi subscriptions by utilizing the nakadi stats api: https://nakadi.io/manual.html#/subscriptions/subscription_id/stats_get

It's possible to define an HPA like below:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
  annotations:
    metric-config.external.my-nakadi-consumer.nakadi/interval: "60s" # optional
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: custom-metrics-consumer
  minReplicas: 0
  maxReplicas: 8 # should match number of partitions for the event type
  metrics:
  - type: External
    external:
      metric:
          name: my-nakadi-consumer
          selector:
            matchLabels:
              type: nakadi
              subscription-id: "708095f6-cece-4d02-840e-ee488d710b29"
              metric-type: "consumer-lag-seconds|unconsumed-events"
      target:
        # value is compatible with the consumer-lag-seconds metric type.
        # It describes the amount of consumer lag in seconds before scaling
        # additionally up.
        # if an event-type has multiple partitions the value of
        # consumer-lag-seconds is the max of all the partitions.
        value: "600" # 10m
        # averageValue is compatible with unconsumed-events metric type.
        # This means for every 30 unconsumed events a pod is scaled up.
        # unconsumed-events is the sum of of unconsumed_events over all
        # partitions.
        averageValue: "30"
        type: AverageValue

To scale on either consumer-lag-seconds or unconsumed-events.

Fix #5

TODO

• Unit tests
• Better documentation in README

[AlexanderYastrebov]

I think subscription is uniquely identified by the owning_application, event_types and consumer_group https://nakadi.io/manual.html#creating-subscriptions so maybe (as a future improvement) this metric handler could also support these three fields and figure out subscription id by listing subscriptions https://nakadi.io/manual.html#getting-and-listing-subscriptions

This could be handy to define the same logical subscription (same owning_application, event_types and consumer_group) in staging and production environments (instead of templating subscription-id value).

[mikkeloscar]

jFYI: this could be a feature improvement but not prioritized for now. But it's simple to add later even if we don't do it in the first iteration.

[AlexanderYastrebov]

👍

[AlexanderYastrebov]

👍

[adyach]

maybe we can improve it in the future that autoscaler does that on its own. it is possible to know number of partitions for Subscription

[mikkeloscar]

Then we need some abstraction on top of HPA to hide this setting for users. Not saying we should not, but it's more complex than what we can implement with the native resource.

[adyach]

is that really per pod or per the whole the stack? so if I have 2 pods and average number goes beyond 30 then a pod is added. it means it is 30 per 2 pods.

[mikkeloscar]

I guess this is how you define it. The autoscaler will always scale higher rather than lower. So if the averageTarget is 30 and you have 31 unconsumed events, it will scale to 2 pods, but it will not scale higher until 61 unconsumed events where it then needs 3 pods.

[mikkeloscar]

👍

[AlexanderYastrebov]

👍