DevOps Software Configuration Management

Kyverno 1.19.2 cuts repeated object conversion in Kubernetes policy evaluation

Kyverno 1.19.2 changes how its CEL-based admission engines prepare Kubernetes requests for policy evaluation, eliminating repeated conversion of the admitted object for each matching policy. In benchmarks reported by Raunak Madan, a software engineer intern at Nirmata, the change substantially reduced engine allocations and processing time as policy counts increased.

The profiling result challenged an expected explanation for admission cost: CEL expression evaluation itself was not the main source of allocations. Before the fix, converting the incoming object into the unstructured map CEL can read accounted for 86.8% of allocations in the ValidatingPolicy engine. Afterward, that share fell to 2.9%, while the largest remaining profile frame was the code that evaluates policy expressions.

One conversion per admission request

The Kubernetes API server sends the admitted object as JSON bytes. Kyverno must convert those bytes into a map-like representation before CEL policies can inspect the request. Previously, that work took place inside the loop over matched policies, so the same object was parsed repeatedly. With a Pod Security Standards policy set, around 16 policies may match a typical Pod creation.

The repeated work made object preparation grow with the number of matching policies and with object size. The 1.19.2 change constructs the CEL request value once before the policy loop, then reuses it for each matched policy. Kyverno still uses the existing conversion code: it temporarily leaves the admitted object out of a request copy, converts the remaining fields, and then inserts the object. This approach aims to preserve the representation policies already receive rather than introducing a separate conversion implementation.

Preserving existing behavior mattered because Kyverno’s engines did not all expose request.object in the same way. Validating and generating policies received the object as sent by the API server, while mutating policies received a normalized version. The engines also differed in how an empty object appeared to CEL, including for DELETE requests. The implementation therefore retains each engine’s established contract while sharing the faster underlying method. Equivalence tests compare the new output with the previous implementation across request types.

Benchmark results and scope

The reported engine benchmark used a roughly 50 KB Pod and varied the number of matched policies. It measures request-value construction, matching and CEL evaluation inside the engine; it does not include the webhook round trip, API server or etcd. For the ValidatingPolicy engine, allocations fell from 2,327 to 854 with one policy, from 37,055 to 2,718 with 16 policies, and from 148,131 to 8,672 with 64. Corresponding engine times changed from 0.31 ms to 0.10 ms, 4.91 ms to 0.16 ms, and 20.01 ms to 0.36 ms.

The MutatingPolicy engine showed a related improvement. At 64 policies, allocations dropped from 258,943 to 52,689, while reported engine time went from 10.59 ms to 2.97 ms. These figures come from a single-machine benchmark and should not be read as end-to-end cluster admission latency. The post-fix profile completes more iterations in the same time, so profile shares—not raw totals—are the appropriate comparison.

The image verification engine received the same conversion change. Madan reports a memory benefit there, rather than a comparable speed improvement, because fetching signatures and attestations over the network remains its main time cost.

Comparison with native Kubernetes policies

The post describes a separate comparison between Kyverno and the API server’s ValidatingAdmissionPolicy, measured on Kyverno 1.19.0, before the new fix. For one policy, native admission took about 9.2 ms versus Kyverno’s 12.8 ms. Across a 16-policy set, Kyverno took about 20 ms versus native’s 29 ms. The Kyverno results in that comparison may improve with the fix, but the comparison has not been rerun.

Those measurements used a single host, loopback for the webhook and dry-run operation, so they exclude the real network cost of a webhook hop and etcd writes. The post also reports CPU use of about 3.2 cores for Kyverno’s engine against 8.2 for native admission under matched load. Its API server profile attributed 67% of CPU to policy dispatch machinery and about 5.5% to CEL bytecode execution. The results are specific to the test setup; a real multi-node cluster adds network latency to Kyverno requests.

Other changes and remaining limitations

Version 1.19.2 also addresses a regression in the older ClusterPolicy JMESPath engine. Tests across four releases found a 12% to 23% regression relative to 1.13.6, appearing between 1.16.4 and 1.17.0. The reported fix replaces an upfront full copy of evaluation context at each checkpoint with a shallow snapshot and copies values when they are written, retaining rule isolation.

The CEL change does not remove the one JSON conversion per request, which remains to preserve the existing raw representation of request.object. Nor does it redesign mutating policies’ handling of full objects between policies or address tail latency under sustained load on a single replica. The benchmark harness and engine tests are available in the Kyverno repository, and the reported measurements should be reproduced in environments where actual admission latency matters.