How Kedify Cut Kafka Scaler Connections by 90%

by Josef Karasek

October 27, 2026

KEDA’s Kafka scaler turns a consumer group’s message backlog, meaning messages that have arrived in Kafka but have not yet been processed, into a Kubernetes scaling signal. Kafka calls this backlog consumer lag. When it grows, KEDA adds consumers; when it clears, KEDA removes them. At higher ScaledObject counts, however, measuring the backlog can create its own broker workload.

Every standard kafka trigger independently opens and manages its own connections to Kafka. With 50 ScaledObjects targeting the same cluster, the operator creates 50 separate connection sets. Each one repeatedly asks Kafka for cluster and topic information, the latest message positions, and its consumer group’s progress. Any one poll is inexpensive; repeating much of the same work across every ScaledObject is not.

The new Kedify Kafka Scaler calculates lag the same way and accepts the same Kafka configuration, while allowing compatible ScaledObjects to share connections and avoid duplicate lookups. In our 50-ScaledObjects benchmark, it reduced Kafka connections by 89.9%, metadata requests by 97.3%, and scaler-related Kafka request traffic by 27.0%.

Why Kafka polling becomes expensive

On each polling cycle, a Kafka scaler needs three categories of information:

1. Topic details to discover the relevant partitions.
2. Consumer progress to learn how far each configured consumer group has read.
3. Latest message positions to calculate the backlog for each partition.

Within one scaler, requests for partitions served by the same Kafka server can be grouped together, which avoids some duplicate network calls. Separate scalers cannot share those groups, so with 50 ScaledObjects the operator still repeats the same metadata lookup 50 times and asks for the latest message positions 50 times, even when every trigger targets the same topic.

Connections amplify the problem. Each scaler maintains its own Kafka connections and local topic information. A temporary server or authentication error can also cause multiple scalers to reconnect at once, turning one interruption into a wave of new connections and repeated requests.

This is a density problem rather than a Kafka lag problem. The scaling calculation is still correct, but its fixed cost grows with the number of triggers.

An opt-in upgrade path

The Kedify Kafka Scaler is a separate trigger type, so installing or upgrading Kedify does not change the behavior of existing kafka triggers. Teams can decide where the efficiency gains matter, adopt the new scaler one workload at a time, and compare broker and operator metrics before expanding the rollout.

Both trigger types can run in the same installation. This makes it practical to start with a busy Kafka cluster or a representative group of ScaledObjects while leaving other workloads untouched. Rolling back is equally straightforward: switch the trigger type back to kafka.

The configuration contract remains familiar. Moving an eligible ScaledObject requires changing the trigger type, not redesigning its scaling policy:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: orders-consumer
spec:
  scaleTargetRef:
    name: orders-consumer
  pollingInterval: 5
  minReplicaCount: 0
  maxReplicaCount: 20
  triggers:
    - type: kedify-kafka
      metadata:
        bootstrapServers: kafka-0.kafka:9092,kafka-1.kafka:9092,kafka-2.kafka:9092
        consumerGroup: orders-workers
        topic: orders
        lagThreshold: '100'
        offsetResetPolicy: latest

Authentication through TriggerAuthentication and the existing Kafka metadata options continue to work as they do for the standard scaler. No pooling-specific metadata field is required, so teams get lower connection and request overhead without another set of tuning controls to operate.

Running Kafka autoscaling at scale?

Talk with the KEDA maintainers at Kedify about reducing broker and operator overhead.

Get Started

The benchmark

We tested four versions of the implementation on AKS:

1. The original kafka scaler behavior
2. Shared Sarama clients only
3. Shared clients plus coordinated topic metadata
4. The final Kedify Kafka Scaler design, including coordination of overlapping latest-offset requests

The workload used 50 Kafka ScaledObjects against one topic on a three-broker Kafka cluster with 96 partitions. Every ScaledObject polled every five seconds. We allowed a 120-second warmup, measured for 600 seconds, and sampled Prometheus every five seconds.

Values are means from the measured 600-second windows.

Where Kedify Kafka helps most

The new scaler is designed for dense installations: many ScaledObjects connecting to the same Kafka cluster with compatible settings. That includes platforms where teams use different consumer groups but share brokers, TLS material, authentication, and Kafka version.

The benefit is smaller when there is only one Kafka trigger or every trigger uses different credentials, broker sets, TLS material, or Kafka versions. Topic-scoped metadata for different topics also requires separate clients. Workloads dominated by per-consumer-group offset requests will retain more of their original request rate because those queries cannot be safely combined.

Choose based on your deployment:

• Keep kafka when you want the existing isolated-client behavior.
• Use the Kedify Kafka Scaler when many compatible triggers make duplicate connections and broker requests a measurable operational cost.
• Benchmark with your broker count, partition layout, polling interval, and authentication mode before turning one result into a capacity assumption.

For large-scale platform environments, this Kafka optimization complements Kedify’s multitenant KEDA and multi-cluster scaling capabilities: tenant-aware isolation within clusters and consistent autoscaling across a fleet.