Kubernetes admission gates and the cluster drift problem

Kubernetes admission gates decide what runs, but drift undoes them. Learn what to gate on and how to keep the live cluster matching git.

Examples in this article are illustrative, not customer case studies. Product availability and coverage are described on the linked capability pages.

Your manifests in git pass every admission check, yet the ingestion deployment in the data-platform namespace is running with a host-path mount nobody reviewed, added with kubectl edit during an incident and never written back. This post shows what Kubernetes admission gates should check, how drift quietly undermines them, and how to reconcile the two so every running workload has an evidenced reason to be there.

Kubernetes admission control is the API server step in which webhooks evaluate each create or update request against policy and allow or deny it before the object is stored. Admission gates decide what may run; drift detection confirms that what is running is still what the gate approved. A cluster needs both.

Admission is the last point where Kubernetes can refuse a workload

An admission webhook sees every create and update request it is registered for before the API server persists it. It can inspect the whole object, evaluate it against policy and return allow or deny synchronously. Nothing later in the lifecycle has that authority, because nothing later intercepts the request before it takes effect. Once the ingestion pods are scheduled, removing them means evicting a live workload, which is disruptive enough that it rarely happens promptly.

Everything upstream of admission, from code review to image scanning to pipeline checks, is preparation. Admission is where the decision is made.

A gate’s failure behaviour matters as much as its policy

Admission webhooks run in the request path under tight time limits. When the policy engine is slow or unreachable, the webhook’s failurePolicy decides what happens next. Set to Fail, the API server refuses requests it cannot evaluate; set to Ignore, it admits them unchecked. A gate that says yes whenever it cannot reach its policy is not a gate but a timing-dependent suggestion.

Failing closed has a cost, so it needs design to go with it: a highly available policy service, webhooks scoped to the namespaces and resources that need them, and a fast override for genuine emergencies that records who used it and why. The post on why fail-closed gates beat severity counts covers that override path in detail.

Kubernetes admission gates need a narrow floor: digest, privilege, registry

It is tempting to turn a compliance checklist into admission policy. Gates that hold up under operational pressure are narrower, and three make a sensible floor for the data-platform namespace and most others.

Image pinned by a known, scanned digest. The manifest should reference an image by digest rather than a mutable tag, and that digest should match something your pipeline actually analysed. Checking that an image exists tells you nothing; the useful question is whether these exact bytes were scanned and what the scan found. Binding images to workloads explains why a tag cannot carry that guarantee.

No privileged containers without a recorded reason. Privileged mode, host networking, host-path mounts and similar escapes from the container boundary should be exceptions with an explicit, recorded justification, not defaults inherited from a Helm chart.

No unknown registries. A workload pulling from an unapproved registry bypasses every scanning and provenance control upstream of admission. This gate is simple to write and closes off the “just pull it from the internet” shortcut entirely.

Beyond this floor, teams add rules that match their own risk tolerance: resource limits, required namespace labels and service account restrictions.

Drift means the gate approved a cluster that no longer exists

GitOps promises that live cluster state is a reconciled reflection of what is committed to a repository. In practice the promise breaks quietly. An engineer patches a deployment during an incident and forgets to backport the change. A controller mutates a field the reconciler does not own. An operator applies a custom resource update directly because the pipeline was slow and the fix was urgent.

Each change is reasonable on its own; together they are corrosive. Admission evaluated the ingestion deployment when it was first applied, and git still shows that approved version. The cluster runs something else, and a security review that starts from the repository describes a workload that no longer exists.

Untracked kubectl edits are the drift that matters most

The host-path mount in the data-platform namespace arrived through a direct edit: a change sent straight to the API server, bypassing the pipeline that would have triggered scanning, review and a recorded change request. Whether admission caught it depended on details nobody had checked. The webhook was registered only for CREATE, so it never saw the update. Had it been set to Ignore, any outage of the policy service would have waved the change through as well.

RBAC can restrict who is able to make direct edits, but restricting access is not the same as recording what was done with it. A cluster where several engineers can change production directly, with no requirement to reconcile those changes into source control, accumulates undocumented state for as long as nobody looks. Posture assessments built only from manifests inherit that blind spot, which is why KSPM starts with a live inventory rather than a repository read.

Reconcile drift and keep the evidence

Banning direct edits outright tends to push incident responders towards workarounds. A better control reconciles drift and leaves evidence behind:

  • Detect drift by comparing live cluster state with the manifests that should describe it, as often as your environment actually changes.
  • Resolve each difference deliberately: backport it to source control if it was legitimate, revert it if it was not, and never leave the gap open.
  • Register admission webhooks for updates as well as creates, so a hand edit meets the same policy as a pipeline deploy.
  • Record every admission decision, allow and deny, so a question about a workload’s origin gets a signed answer rather than a search through expiring logs.

Treated this way, admission gates and drift detection are two halves of one control. The gate decides what may run; drift detection confirms that what is running is still what the gate approved. For the ingestion deployment, the host-path mount either appears in git with a recorded reason or disappears from the cluster. The same discipline applies to cloud resources, as closing IaC drift on live evidence shows.

Key takeaways

  • Register admission webhooks for both create and update requests, and set failurePolicy deliberately rather than by default.
  • Start with a narrow floor of gates: scanned image digests, no unjustified privilege and approved registries only.
  • Compare live cluster state with git continuously and resolve every difference by backporting or reverting it.
  • Record every admission decision, including allows, so each running workload has an evidenced origin.

How SecurityVault approaches this

SecurityVault connects collected cluster configuration and image-digest context with policy evaluation. Admission decisions can be recorded alongside upstream scan evidence. Test webhook scope, update handling, failure behavior and emergency procedures in your own cluster before enforcement. Posture collection complements admission by checking observed configuration after deployment.

Use a non-production namespace to test a permitted create, a prohibited update and an evaluator outage. Confirm which requests reach the webhook and which are excluded by its registration. Then make an approved configuration change and compare the live collection with the source declaration. Agree how emergency changes are reconciled and who reviews overrides. These checks help the platform team understand the combined control: admission evaluates a request at one point in time, while later collection can reveal changes or gaps that still require investigation.

Explore Deployment gates or talk to our team about an evaluation with your engineering, security and compliance stakeholders.

Further reading

See it on your stack

Bring KSPM into one finding graph.

SecurityVault is the enterprise security operating system: one graph of findings, policy and evidence from code to cloud. Explore the KSPM workflow with your engineering, security and compliance teams. No repository access is needed for the initial demonstration.