Hybrid Cloud Software Defined Data Center

Kueue and Coscheduling coordinate Kubernetes AI training workloads

Distributed AI training puts pressure on two assumptions built into Kubernetes scheduling: that pods can be placed independently and that any eligible node is suitable. Training workers often need to start together and communicate across GPUs with low latency. Kueue and the Kubernetes Coscheduling plugin address those requirements with topology-aware placement and gang scheduling, while Kubernetes develops related native capabilities.

The distinction matters because a job can be running without making efficient use of its accelerators. If its workers are spread across distant parts of a cluster, communication between them can slow each training step. If only some workers start, allocated GPUs may sit idle while the rest of the group waits for capacity.

Placement affects communication costs

Distributed training workers exchange data as they update a model. Data-parallel training commonly uses an AllReduce operation to aggregate and distribute gradients. Strategies that divide a model across workers, including tensor, pipeline and expert parallelism, can require frequent gather, reduce or all-to-all communication during a step.

The network path between workers affects how quickly those exchanges complete. GPUs in the same node may communicate through NVLink or NVSwitch when available, or through PCIe. Communication between nodes uses the cluster network, while traffic across racks can add further latency. In large jobs, each step may wait for its slowest worker, so delays on communication paths can accumulate. Smaller fine-tuning jobs may be less sensitive to the same penalty.

Topology-aware scheduling (TAS) helps keep a workload’s pods within a suitable physical domain. Kueue can group nodes into domains such as hosts, racks or blocks, using topology labels already applied to nodes. For example, a cluster may use the label topology.kubernetes.io/rack. Kueue does not discover the physical topology itself; administrators or a cloud provider’s topology labeler must supply the labels.

Kueue admission and gang scheduling

Kueue is a Kubernetes-native job queuing system that works above the kube-scheduler. It evaluates workloads against factors such as quota, priority and preemption, then decides when a job can proceed. Its Workload object records resource requirements and admission information, including topology assignments. At admission, Kueue can add scheduling constraints to job pod templates so the kube-scheduler places the pods within the selected domain.

That admission decision does not, by itself, ensure that all workers start together. The Coscheduling plugin uses a PodGroup to track a gang’s minimum worker requirement and scheduling state. Pods associated with the same group are held until enough members can be scheduled as a unit, avoiding a partial start that consumes resources without allowing the training job to run as intended.

The two mechanisms address different stages. Kueue determines whether a workload can enter scheduling and where its pods should fit; Coscheduling coordinates the group’s assembly. If pods do not become ready within a configured timeout, Kueue can evict and requeue an admitted workload. Such an event can reflect slow scheduling, image pulls, resource pressure or node failure, so it is a signal to investigate rather than a diagnosis on its own.

Monitoring the scheduling path

Operational visibility needs to cover admission, placement, gang assembly and application progress. Kueue’s OpenMetrics metrics include kueue_pending_workloads, which distinguishes active queueing from workloads deemed inadmissible, and kueue_admission_wait_time_seconds, which measures time to admission. A sustained inadmissible count can point to exhausted quota, an unavailable topology domain or an unsatisfied constraint. Eviction metrics include reason labels such as NodeFailures and PodsReadyTimeout.

Operators can inspect a Kueue Workload’s status.admission.podSetAssignments[].topologyAssignment to verify the assigned topology levels and domains. They can then compare that assignment with GPU interconnect and network telemetry. PCIe activity can help indicate GPU-to-host transfers during cross-node communication, but it does not measure the network hop itself. Low throughput despite a valid assignment may therefore require investigation of drivers, hardware or the network fabric.

For Coscheduling, the PodGroup’s specification and status help show whether a gang is waiting, being assembled or running. Its spec.minMember records the minimum group size; status.phase reports states such as Pending, Scheduling, Running, Unknown or Failed. Comparing PodGroup state with Kueue admission status helps separate an upstream queue or quota issue from a later scheduling problem.

Finally, infrastructure signals need to be tied to training progress. Framework metrics such as steps, samples or tokens per second can reveal whether GPU activity is producing useful work. For Ray, placement-group states and task counts offer additional clues when resources for a placement group are unavailable. A throughput decline alongside PodGroup delays and weak interconnect performance can help teams investigate scheduling or placement as contributors.

Native Kubernetes features are still developing

Kueue and Coscheduling remain separately installed components. Kubernetes is adding related native features, but they are not yet a drop-in replacement. Gang scheduling arrived as an alpha feature in v1.35. Kubernetes v1.36 added alpha capabilities including topology-aware scheduling, workload-aware preemption and initial Job controller integration behind a feature gate, as well as a native PodGroup API distinct from the Coscheduling plugin’s custom resource.

In v1.37, gang scheduling and workload-aware preemption reached beta under the GenericWorkload feature gate, which remains disabled by default; topology-aware workload scheduling is still alpha. The existing monitoring practices for Kueue, PodGroups, topology assignments and training throughput can give operators a baseline as they assess these evolving scheduler capabilities.