During my last three cluster audits, I observed a recurring pattern: developers bypassing security controls by using the privileged: true flag in their Pod specs to "fix" permission issues. This isn't just a bad practice; it is a direct path to container escape. With vulnerabilities like CVE-2024-21626 (a runc breakout) and CVE-2022-0185 (heap overflow in the Linux kernel) documented in the NIST NVD targeting low-level container runtimes, Kubernetes clusters without strict admission control are ticking time bombs.
Open Policy Agent (OPA) solves this by treating policy as code. We no longer rely on manual audits or Pod Security Policies (which are deprecated). Instead, we inject a programmable gatekeeper into the Kubernetes API request lifecycle. This guide details the CLI-driven implementation of OPA Gatekeeper to harden your production workloads.
What is Open Policy Agent (OPA)?
OPA is a domain-agnostic, open-source policy engine. It allows us to offload policy decisions from our applications and infrastructure. In a Kubernetes context, OPA doesn't care about your business logic; it cares about the JSON representation of your AdmissionReview request.
We use a declarative language called Rego to write these policies. Rego is purpose-built for querying complex hierarchical data structures. When a user runs kubectl apply, the Kubernetes API server sends a webhook request to OPA. OPA evaluates the request against our Rego rules and returns an "allow" or "deny" decision.
The Role of Policy-as-Code in Kubernetes Security
Policy-as-Code (PaC) moves security "left" by integrating it into the CI/CD pipeline and the cluster's admission layer. Hardening CI/CD Pipelines ensures that only verified requests reach your orchestration layer. Without PaC, security teams are stuck in a reactive loop, hunting for misconfigurations via logs or periodic scans. By the time a scanner finds a privileged container, an attacker might have already pivoted to the host.
With OPA, we define the guardrails upfront. If a developer tries to deploy a container with a hostPath mount to /etc, the API server rejects the request immediately. The developer receives a clear error message explaining the violation, reducing the feedback loop from days to seconds.
Benefits of Decoupling Policy from Application Logic
Decoupling policy means your security rules evolve independently of your application code. We observed that this separation of concerns allows security researchers to update policies globally without requiring developers to rebuild their container images.
- Consistency: Apply the same security standards across AWS EKS, Google GKE, and on-premise clusters.
- Auditability: Policies are stored in Git, providing a clear audit trail of who changed which security rule and when.
- Granularity: Unlike standard RBAC, OPA can look inside the
specof a resource to make decisions based on image tags, resource limits, or specific labels.
Understanding Admission Controllers
Kubernetes admission controllers are plugins that govern how the cluster is used. They act as a gatekeeper between the API server and the object store (etcd). There are two types of admission controllers relevant to OPA:
- Mutating Admission Controllers: These can modify the object before it is stored. For example, automatically injecting a sidecar proxy or adding default resource limits.
- Validating Admission Controllers: These can only allow or deny a request. OPA Gatekeeper primarily functions as a validating controller to enforce security constraints.
Validating vs. Mutating Webhooks
We generally prefer validating webhooks for security hardening because they are deterministic and easier to debug. A validating webhook takes the incoming YAML, checks it against a set of rules, and returns a true/false result.
Mutating webhooks are powerful but can lead to "hidden" configurations. If a mutating webhook changes a container's image tag without the developer knowing, it can cause unexpected failures. In our hardening workflows, we use validation to block non-compliant resources and mutation only for non-critical defaults like imagePullPolicy.
The Rego Query Language Explained
Rego is the heartbeat of OPA. It is a Datalog-inspired language that allows for high-performance evaluation of JSON data. When writing Rego for Kubernetes, we focus on the input.review.object path. This path contains the full manifest of the resource being created or updated.
A typical Rego rule consists of a head and a body. If the body evaluates to true, the head (the violation) is triggered. We tested various Rego structures and found that using helper functions for iterating over containers significantly reduces code duplication.
Key Differences Between OPA and OPA Gatekeeper
Standard OPA is a general-purpose engine that requires you to manage your own sidecars and synchronization logic. OPA Gatekeeper is a Kubernetes-native wrapper for OPA. It simplifies the integration by using Custom Resource Definitions (CRDs) to manage policies.
- OPA: Requires manual management of Rego files and webhook configurations.
- Gatekeeper: Uses
ConstraintTemplatesto define the logic andConstraintsto apply that logic to specific resources. - State Awareness: Gatekeeper can "replicate" cluster state, allowing policies to check for uniqueness (e.g., ensuring no two Ingresses use the same hostname).
Why Gatekeeper is the Preferred Method for Kubernetes
Gatekeeper is the industry standard because it integrates with the Kubernetes API. You don't need to learn a new CLI for deployment; you use kubectl. It also provides an "Audit" functionality that periodically scans existing resources in the cluster for policy violations that occurred before the policy was implemented.
ConstraintTemplates and Constraints Architecture
The Gatekeeper architecture follows a "Template-Instance" pattern. A ConstraintTemplate defines the Rego logic and the parameters the policy accepts. A Constraint is an instantiation of that template, where you define the specific parameters (like which namespaces to exclude or which registries are allowed).
Prerequisites for Cluster Setup
Before installing Gatekeeper, ensure your cluster is running Kubernetes 1.16 or later. You also need helm installed on your local machine and administrative access to the cluster. For DevOps engineers managing multiple environments, using a secure SSH access for teams solution can streamline the initial configuration and troubleshooting of these security components.
Installing Gatekeeper via Helm or YAML
We prefer the Helm installation method because it simplifies the management of the admission webhook certificates. Run the following commands to set up the Gatekeeper operator:
# Add the Gatekeeper Helm repository
helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts
Update the local repo cache
helm repo update
Install Gatekeeper in the gatekeeper-system namespace
helm install gatekeeper gatekeeper/gatekeeper \ --namespace gatekeeper-system \ --create-namespace \ --set enableExternalData=true
Verify the installation by checking the status of the controller and audit pods:
$ kubectl get pods -n gatekeeper-system
NAME READY STATUS RESTARTS AGE gatekeeper-audit-5c6b9b6b7-v8x2p 1/1 Running 0 2m gatekeeper-controller-manager-7f9d8d9d8-9lqzx 1/1 Running 0 2m
Defining Your First ConstraintTemplate
We will create a ConstraintTemplate that prevents the use of privileged containers. This is a critical step in mitigating CVE-2024-21626, as privileged containers have easier access to the host's filesystem and process space.
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate metadata: name: k8spspprivilegedcontainer spec: crd: spec: names: kind: K8sPSPPrivilegedContainer targets: - target: admission.k8s.gatekeeper.sh rego: | package k8spspprivilegedcontainer
violation({"msg": msg}) { c := input_container[_] c.securityContext.privileged msg := sprintf("Privileged container is not allowed: %v", [c.name]) }
input_container[c] { c := input.review.object.spec.containers[_] } input_container[c] { c := input.review.object.spec.initContainers[_] } input_container[c] { c := input.review.object.spec.ephemeralContainers[_] }
Apply this template using kubectl apply -f template.yaml. This doesn't enforce anything yet; it only defines the "how" of the policy.
Deploying Constraints to Enforce Policies
Now we create the Constraint which tells Gatekeeper to apply the logic from our template to all Pods in the cluster.
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPPrivilegedContainer metadata: name: no-privileged-containers spec: match: kinds: - apiGroups: [""] kinds: ["Pod"] excludedNamespaces: - kube-system - gatekeeper-system
By excluding kube-system, we ensure that critical cluster components (which often require elevated privileges) can still function while restricting application workloads.
Enforcing Container Image Registry Whitelists
One of the most effective ways to prevent supply chain attacks, as highlighted in the OWASP Top 10, is to restrict which registries your cluster can pull images from. In the Indian enterprise context, we often see organizations mandating that all images come from a private ECR or Azure Container Registry (ACR) hosted in the ap-south-1 (Mumbai) region.
A whitelist policy prevents developers from pulling unvetted images from public Docker Hub repositories. We can implement this by checking the image string in the Pod spec and ensuring it starts with our approved domain.
Mandating Resource Limits and Requests
Unbounded containers can lead to "noisy neighbor" problems and cluster-wide denial of service. We use OPA to ensure every container has cpu and memory limits defined. This is particularly important for FinTech SMEs in India running on tight infrastructure budgets, where a single runaway process can spike cloud costs in INR (₹) significantly.
Preventing Privileged Containers and Root Access
Beyond the privileged flag, we must also enforce runAsNonRoot: true and allowPrivilegeEscalation: false. Attackers often use setuid binaries to gain root access inside a container. Hardening Docker Secrets and enforcing non-root execution significantly increases the difficulty of an exploit succeeding.
Ensuring Required Labels on Namespaces and Pods
For compliance with the Digital Personal Data Protection (DPDP) Act 2023, Indian organizations must track where sensitive data is processed. We can use OPA to mandate a data-classification label on every Namespace.
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels metadata: name: ns-must-have-classification spec: match: kinds: - apiGroups: [""] kinds: ["Namespace"] parameters: labels: ["data-classification"]
Using the OPA CLI for Local Rego Testing
Never deploy Rego directly to a production cluster without local testing. The opa CLI provides a test command that allows you to run unit tests against your Rego files. I always structure my policy repository with a _test.rego file for every policy.
# Run local tests for your Rego policies
opa test ./rego/policies_test.rego ./rego/policies.rego -v
The output will show you which rules passed and which failed, allowing you to debug logic errors without waiting for a Kubernetes deployment cycle.
Dry-run and Warn Modes in Gatekeeper
If you are introducing OPA to an existing cluster, do not start with enforcementAction: deny. This will break existing workloads and cause friction with dev teams. Instead, use dryrun or warn mode.
- dryrun: Violations are logged in the Constraint's status but the request is not blocked.
- warn: The request is allowed, but a warning message is sent back to the user's CLI.
spec:
enforcementAction: warn match: kinds: - apiGroups: [""] kinds: ["Pod"]
Auditing Existing Resources for Policy Violations
Gatekeeper's audit functionality is invaluable for identifying technical debt. It periodically crawls the cluster and updates the status field of each Constraint with a list of violating resources. You can query these violations using kubectl:
# Get all violations for a specific constraint
kubectl get k8spspprivilegedcontainer no-privileged-containers -o jsonpath='{.status.violations}'
Optimizing Rego Performance for Large Clusters
As your cluster grows to hundreds of nodes and thousands of pods, OPA evaluation latency can impact API server performance. To optimize:
- Avoid large scans: Use specific
matchcriteria in your Constraints to limit the number of objects Gatekeeper processes. - Use indexing: Rego's
[_]operator is fast, but avoid deep nested loops where possible. - External Data: Be cautious when using the
external datafeature (e.g., calling an external API during admission) as it introduces a point of failure and latency.
Version Control and CI/CD Integration for Policies
Treat your policies like production code. Store them in a Git repository and use a CI pipeline to run gator verify. Gator is a CLI tool provided by the Gatekeeper project that allows you to test ConstraintTemplates and Constraints against sample manifests without a live cluster.
# Use gator to verify a manifest against a policy library
gator verify ./policy-library/privileged-containers/template.yaml \ ./policy-library/privileged-containers/constraint.yaml \ ./manifests/bad-pod.yaml
Monitoring Policy Execution with Prometheus and Grafana
Gatekeeper exports Prometheus metrics by default. We monitor two key metrics to ensure the health of our policy engine:
gatekeeper_constraints_not_resolved: Indicates issues with the template or constraint logic.gatekeeper_request_duration_seconds: Tracks the latency of admission requests. We aim for sub-50ms evaluation times.gatekeeper_violations: A gauge of total policy violations in the cluster, which we visualize in Grafana to track security posture over time.
Data Residency and Indian Compliance (DPDP Act 2023)
Under the DPDP Act 2023, processing sensitive personal data (SPD) of Indian citizens may require strict residency controls. We use OPA to enforce "Node Affinity" rules. This ensures that any Pod with the label data-type: sensitive-kyc is only scheduled on nodes located in Indian data centers (e.g., AWS ap-south-1a/b/c).
# Example Rego snippet for node affinity enforcement
violation({"msg": msg}) { input.review.object.metadata.labels["data-type"] == "sensitive-kyc" not has_indian_node_affinity(input.review.object) msg := "Pods processing KYC data must have nodeAffinity set to ap-south-1" }
This programmatic enforcement prevents accidental cross-border data transfers that could lead to heavy regulatory fines. By pinning these workloads at the admission level, we provide a technical guarantee of compliance that manual spreadsheets cannot match.
Next Command: Auditing your cluster for hostPath mounts
Run the following command to identify which pods are currently using hostPath mounts, as these are the primary targets for your next OPA policy:
kubectl get pods --all-namespaces -o jsonpath='{range .items[]}{.metadata.namespace}{":"}{.metadata.name}{"\t"}{.spec.volumes[].hostPath.path}{"\n"}{end}' | grep -vE "\s$"