Container Virtualization DevOps

Kubernetes Work Targets AI Scheduling, GPU Allocation and Large-Cluster Scale

Kubernetes is being adapted to handle the scale and coordination demands of artificial intelligence workloads, with work spanning cluster performance, accelerator scheduling and faster container startup. At KubeCon + CloudNativeCon North America 2026, the Cloud Native Computing Foundation (CNCF) and vendors outlined changes intended to help operators run AI workloads more efficiently.

The need is already substantial: the CNCF’s January 2026 Annual Cloud Native Survey found that 66% of organizations hosting generative AI models use Kubernetes for inference. Kubernetes 1.38, scheduled for December 16, is expected to advance several capabilities, although their status remains subject to the release process. Other efforts discussed at the conference are being developed separately through projects including Karpenter and containerd.

Addressing control-plane pressure

Mike Stefaniak, AWS’s head of containers, described challenges encountered as AWS pushed Kubernetes beyond its conventional design envelope of 5,000 nodes toward clusters with 100,000 nodes. In a benchmark involving one million Pods, a lock in the API server’s watch cache caused writers to wait more than half a second. Stefaniak said a lazy-snapshot implementation cut that wait to two milliseconds.

He also described changes to Horizontal Pod Autoscaler processing that removed serialization, along with scheduler improvements that replaced repeated volume-reference calculations with incremental updates. According to Stefaniak, scheduling rates exceeded 300 Pods per second under sustained backlog in AWS’s largest clusters. He said these changes have shipped in Kubernetes 1.37.

For Kubernetes 1.38, AWS co-authored a proposal to compress API objects before they are stored in etcd, the system’s backing data store. Stefaniak said the proposal had been accepted for the 1.38 alpha and that initial code had been merged. The goal is to ease storage pressure in very large clusters; the feature’s inclusion and eventual status will depend on the release process.

Coordinating distributed AI workloads

AI training and inference can depend on multiple workers that need to operate as a coordinated workload. If only some workers can be placed, a training job may remain unable to make progress while occupying GPU capacity. Gang scheduling addresses this by holding a workload until its required minimum group can be admitted.

The approach coordinates admission; it does not guarantee that every container starts at the same moment. Workload and PodGroup APIs provide scheduling primitives for groups of Pods. The roadmap also targets beta maturity for topology-aware placement and CompositePodGroups. These efforts are intended to support communication-sensitive placement and workloads composed of multiple coordinated groups, including distributed training and disaggregated inference.

Dynamic Resource Allocation (DRA) complements group scheduling by representing accelerator capabilities and available capacity in greater detail. The planned maturity advances cover partitionable devices, consumable and fractional capacity ranges, and device binding conditions. These mechanisms go beyond allocating accelerators by simply counting whole GPUs.

Provisioning accelerators and starting containers faster

Karpenter, an open-source Kubernetes node lifecycle manager, provisions and terminates compute resources in response to unscheduled Pod requests and other conditions. Stefaniak said Karpenter can now model devices on the instance types it may provision. That work is intended to connect DRA’s device inventory with just-in-time node creation, allowing GPU capacity to scale from zero and be placed according to workload needs. It is Karpenter development, not a Kubernetes 1.38 feature.

Large container images create another delay. AI images can reach tens of gigabytes, and parallel downloading can shift the bottleneck from transfer to unpacking. Stefaniak said a parallel-unpack implementation upstreamed in containerd 2.2 reduced the time for a 10GB image to move from registry to node from two minutes to consistently under 45 seconds in AWS testing. That result is an AWS-reported test outcome, not a general performance guarantee.

Efficiency includes agent workloads

Conference discussions also addressed how to use existing hardware more effectively when running AI agents. CNCF Executive Director Jonathan Bryce described the objective as getting more work from the same hardware footprint and power consumption. He highlighted OpenShell’s default-deny approach to agent permissions and Agent Substrate’s handling of idle agents. Bryce cited a demonstration that increased the number of agents supported per node from one to 65 by putting agents to sleep and waking them when they needed to work. The work remains in progress.

Portability is another open challenge. Dilpreet Bindra, VMware’s senior director of engineering, said the goal is to define and verify workload behavior across environments to reduce fragmentation. The initiatives presented at KubeCon point to a broader operational shift: AI workloads require coordination among scheduling, hardware allocation, node provisioning and workload controls. The planned changes may improve utilization, but consistent production behavior will depend on implementation and adoption across the Kubernetes ecosystem.