Build & Release Automation DevOps

Argo CD self-heal can skip migration hooks after sync windows close

A race in Argo CD can leave an application marked Synced and Healthy even though its database migration did not run. The behavior arises when an automated sync window closes and a self-heal operation reaches the controller before a regular reconciliation. In a five-region production example described by CNCF Ambassador Afzal Ansari, two of five Applications deployed new code without the migration, leaving them on an incompatible database schema for 36 to 44 minutes.

Ansari traced the issue to how Argo CD handles different refresh events. His investigation, published November 20, 2026, included reading controller code and reproducing the behavior in a lab. The findings matter for teams that rely on Sync hooks to coordinate schema changes with application releases: a hook runs as part of a particular sync operation, but not every operation is a full sync.

Why a successful sync can be misleading

The reported production setup used one Argo CD Application per region, with automated sync, pruning and self-heal enabled. A deny sync window prevented production changes outside business hours. The database migration was configured as a Sync hook Job. After the window closed, all five Applications rolled out the same commit, but only three ran the migration.

The two affected Applications appeared Synced and Healthy on the new commit. Their operation status reported that all tasks had run. Controller logs provided a less reassuring account: the operations listed 13 resources and showed a self-heal attempt count of one. Ansari’s analysis found that these were partial syncs to correct drift, rather than normal full syncs. Partial syncs do not run hooks, so the migration Job was skipped.

In his reproduction, the application included a ConfigMap and Deployment containing a release identifier, plus a migration Job configured as a Sync hook. A four-minute deny window allowed him to control when syncing became available. He reproduced the failure in five of six attempts after adjusting refresh timing. An initial attempt instead produced a full sync, illustrating that the event arriving first can determine the outcome.

The refresh race

Ansari distinguishes two refresh paths in the controller. A level-2 refresh, triggered by the reconciliation timer roughly every two to three minutes, compares the application with Git. A level-1 refresh follows a change to a managed object and reuses information already held by the controller.

When a deny window ends, Argo CD does not receive a special notification that forces a full sync. The next refresh discovers that syncing is permitted. During the window, timer-driven refreshes can record the new Git revision in status.sync.revision even though the last actual sync remains on the previous revision. If a level-1 refresh arrives first, the controller can compare that recorded revision with itself and conclude there are no changes. With self-heal enabled, it may then apply only out-of-sync resources. That partial operation omits hooks and does not create normal sync history.

A timer-driven refresh follows a different path: it compares the target revision with the last synced revision and can initiate a full sync, which runs the hook. Ansari summarizes the underlying issue as the level-1 path treating the last compared revision as if it were the last synced revision.

Make the migration a managed resource

Ansari’s preferred mitigation is to represent the migration as desired state rather than as a hook side effect. In his lab, he removed the hook annotations, assigned the Job a sync wave of -1 so it ran before the application resources, and changed the Job name for each release. Because the new Job is a new managed resource, a partial sync sees it as missing and applies it. In the test, the migration completed before the new ConfigMap and Deployment were applied.

This approach requires care. Job names should be unique per release because Kubernetes Job pod templates are immutable. Pruning should remain enabled to remove old Jobs. Ansari cautions against setting ttlSecondsAfterFinished: Kubernetes could delete a completed Job, after which self-heal might recreate it and run the migration again. He also recommends keeping migrations idempotent and testing failure handling in staging; he said he had not tested a deliberately failing migration.

A temporary alternative and its trade-off

Where changing charts is not immediately practical, Ansari tested disabling self-heal for applications that run migrations while leaving automated sync and pruning enabled. In his comparison, the application with self-heal enabled deployed the release without its migration. The application with self-heal disabled waited after the initial “no changes” decision. A later timer-driven refresh initiated a full sync, and the hook ran without a manual sync.

The trade-off is that manual changes to resources can remain OutOfSync until a later commit. The broader operational lesson is to avoid making a critical database migration depend solely on a hook executing in one specific kind of sync. Making the migration a versioned resource in Git can ensure it is included when Argo CD applies the release, regardless of whether the operation is full or partial.