Examples in this article are illustrative, not customer case studies. Product availability and coverage are described on the linked capability pages.
A release gate rejects a risky image in a demonstration but allows it when a required evidence field is missing. Policy testing needs to exercise incomplete inputs and evaluation failures as deliberately as the normal allow and deny decisions.
Consider a fictional shipment service whose release policy requires an assessment of the candidate image and an acceptable provenance result. The team is moving the rule from a written checklist into a controlled gate. The exercise is to define its input contract, test the decision boundary and preserve a reasoned result that an engineer can act on.
Policy-as-code testing begins with a decision contract
Write down the exact decision the gate is responsible for. Identify the candidate artefact, the evidence required to evaluate it and the expected output. A gate should not quietly switch from evaluating an image digest to evaluating a repository simply because the input payload changed shape.
For the shipment service, include a candidate digest and references to the relevant assessment and provenance evidence. Define how the gate establishes that those records concern the same candidate. Evidence about another digest must not become acceptable merely because it has a recent timestamp or a successful status.
Separate a policy denial from an evaluation error in the output contract. Both may block release under fail-closed behaviour, but their causes and owners differ. An engineer should be able to distinguish a disallowed candidate from missing input, unavailable evidence or a policy execution failure.
Build fixtures around the boundary, not the implementation
Create small, readable input fixtures that represent meaningful release situations. Start with a permitted candidate, then change one property at a time. This makes it clear which condition is responsible for a different outcome and keeps tests useful when the implementation is reorganised.
| Fixture | Expected outcome | Reason to inspect |
|---|---|---|
| Matching candidate with required evidence | Allow if all stated conditions pass | Baseline accepted input |
| Evidence for a different digest | Deny | Candidate/evidence mismatch |
| Required assessment absent | Block with an incomplete-input reason | Missing evidence |
| Evidence outside the accepted freshness policy | Deny or hold as explicitly specified | Stale observation |
| Malformed input | Block with a validation reason | Contract violation |
| Evaluation service unavailable | Block with an operational error | No decision was obtained |
Keep example values synthetic. A policy fixture should not require access to a live registry or a production credential simply to test a branch of decision logic. Integration tests can separately verify how the real evidence is collected and represented.
Policy-as-code testing must cover unknown values
Missing, null, empty and explicitly negative values can have different meanings. Define those meanings at the input boundary instead of allowing a language default or caller assumption to decide release behaviour. Tests should establish that an absent required fact cannot silently satisfy a rule intended for a verified fact.
In the shipment example, distinguish an assessment that completed with an accepted result from an assessment that has not completed. Also distinguish a record that was not supplied from one that could not be retrieved. These states may all block the candidate, but the reported reason should lead the owner towards the correct remedy.
Review the caller as well as the policy. A correct policy cannot protect the release if the pipeline interprets a timeout, empty response or unexpected output shape as permission. Add an integration fixture for those cases at the point where the gate controls the next action.
Use coverage to find gaps, not to certify correctness
Open Policy Agent provides policy tests and can report coverage for evaluated rules and expressions. That feedback helps identify code paths the current fixtures never exercise. It does not establish that the input model is complete or that the assertions encode the intended release policy.
Read the expected result of each test alongside the written decision contract. A suite can execute every branch while accepting the wrong artefact identity in every fixture. Reviewers should ask which security condition each test protects, rather than relying only on a coverage summary.
For the shipment policy, deliberately alter a critical comparison in a local test copy and confirm a relevant fixture fails. This small mutation exercise helps reveal tests that merely check a response exists. Restore the implementation afterwards and retain the useful regression, not the experimental mutation.
Verify reasons and exception handling
An explainable denial should identify the candidate and the relevant failed requirement without exposing sensitive input. Test that the reason changes when the underlying condition changes. A generic failure message can leave developers repeatedly rerunning a gate whose evidence contract was never satisfied.
If exceptions are supported, define their identity, scope and validity conditions explicitly. Test an exception for the wrong candidate, an expired exception and an exception lacking the required approval record. An exception must not become a reusable bypass merely because its identifier appears in the payload.
Preserve the original assessment and denial context when an authorised exception permits progress. The decision history should explain why the release proceeded despite the unmet condition. It should not rewrite the underlying finding into a passing technical result.
Promote the rule through a controlled rollout
Evaluate the candidate policy against retained representative inputs before making it enforce a release boundary. Compare its decisions with the approved expectations and review disagreements. Where an existing gate already protects the workflow, keep that control active while testing a proposed replacement.
Identify the policy version, rollout owner and recovery procedure before activation. A rollback should restore a known reviewed policy and its corresponding input contract, not replace the gate with an unconditional allow. If an operational incident requires an exception, record it through the same accountable process used for other release decisions.
Observe initial decisions for both expected denials and unexpected evaluation errors. A rise in missing-input failures may indicate an integration contract problem rather than a sudden change in software risk. Route those failures to the owner who can repair the evidence path without weakening the intended rule.
Retain enough information to reproduce a decision
Store the policy version, candidate identity, relevant evidence references, decision time and reason. Keep sensitive input in appropriate restricted storage instead of copying it into broadly visible pipeline output. A reviewer should be able to reconstruct why the gate reached its outcome without obtaining reusable credentials.
For the shipment service, rerun a retained synthetic input against the same policy version and compare the result. If external lookups affect the decision, record the relevant observations rather than assuming they will remain unchanged. Reproducibility turns a gate from a momentary pipeline colour into evidence that survives the release.
Key takeaways
- Define candidate identity, required evidence and output semantics before implementing the rule.
- Test incomplete input, mismatched evidence and evaluation errors alongside allow and deny cases.
- Verify the caller fails closed and that exceptions remain narrow and attributable.
- Retain policy versions and evidence references so decisions can be reviewed later.
How SecurityVault approaches this
SecurityVault’s deployment-gates workflow uses policy decisions at delivery boundaries with recorded outcomes. Open Policy Agent rules contribute to those decisions, while connected finding and artefact context provides relevant inputs. The operational value is a decision that names the candidate, evidence and reason, rather than an unexplained failure in the pipeline.
Policy authors still need to define the intended contract and prove it with representative tests. A fail-closed result prevents missing decisions from silently permitting progress, but it does not make an incorrectly written rule correct. Teams should evaluate both the decision logic and the caller’s handling of errors, exceptions and incomplete evidence.
Use a synthetic release candidate with matching, missing and mismatched evidence during an evaluation. Ask how the policy version and human decisions remain available after the pipeline finishes. The policy-authoring documentation is the next step for discussing that contract and its verification requirements.