During our recent audit of a large-scale Kubernetes cluster for a Mumbai-based fintech, we discovered that standard Role-Based Access Control (RBAC) was insufficient to prevent container breakouts. While RBAC controls who can interact with the API, it does not inspect the payload of the requests. We observed multiple pods running with privileged: true, which allowed them to exploit CVE-2024-21626 (a high-severity runc breakout documented in the NIST NVD) and access the underlying node's filesystem.
To remediate this, we implemented Open Policy Agent (OPA) via Gatekeeper to enforce fine-grained constraints. This approach shifts security from reactive monitoring to proactive prevention, aligning with the implementing policy as code methodology and the "Security by Design" mandate of India's Digital Personal Data Protection (DPDP) Act 2023.
Introduction to Open Policy Agent (OPA) and Rego
What is Open Policy Agent?
Open Policy Agent (OPA) is an open-source, general-purpose policy engine that decouples policy decision-making from policy enforcement. In a typical Kubernetes environment, OPA acts as an Admission Controller. When a user sends a kubectl apply command, the API server sends a JSON representation of that request to OPA. OPA evaluates the request against a set of rules and returns an "allow" or "deny" decision.
Understanding Rego: The Policy Language for OPA
Rego is the declarative language used to write OPA policies. It is inspired by Datalog, a well-understood query language. Unlike imperative languages where you define how to check a condition, Rego allows you to define what conditions must be met. This is particularly useful for complex JSON structures found in Kubernetes manifests, especially when securing K8s manifests against misconfigurations.
Why Use Policy as Code in Modern Infrastructure?
The DPDP Act 2023 requires Indian organizations to implement technical measures to prevent unauthorized access to personal data. Beyond policy engines, providing secure SSH access for teams ensures that administrative entry points are hardened. Hardcoding these checks into your application or relying on manual reviews is prone to human error. By using OPA, we treat security policies like source code: they are versioned, tested, and integrated into CI/CD pipelines. This ensures that every deployment, whether on EKS, AKS, or on-premise clusters, adheres to the same security baseline.
Getting Started: Open Policy Agent Rego Tutorial
Setting Up Your OPA Environment
To begin developing policies, you need the OPA binary on your local workstation. I recommend using the OPA CLI for testing and debugging before deploying to a cluster.
Download the OPA binary for Linux (AMD64)
$ curl -L -o opa https://openpolicyagent.org/downloads/v0.61.0/opa_linux_amd64_static
Make it executable and move to path
$ chmod 755 opa $ sudo mv opa /usr/local/bin/
Verify installation
$ opa version
Basic Syntax: Packages, Rules, and Expressions
Every Rego file must start with a package declaration. Rules are the building blocks of policies. A rule is a set of expressions; if all expressions evaluate to true, the rule is true.
package system.authz
A simple rule that evaluates to true if the user is 'admin'
default allow = false
allow { input.user == "admin" }
How OPA Evaluates Policies: Input, Data, and Query
OPA evaluation involves three components:
- Input: The JSON data sent to OPA (e.g., a Kubernetes Pod spec).
- Data: External information OPA can use (e.g., a list of authorized user IDs in a JSON file).
- Query: The specific rule OPA is asked to evaluate (e.g.,
data.system.authz.allow).
Writing Your First Rego Policy
Defining the 'Allow' and 'Deny' Logic
In Kubernetes, we typically use a "Deny by Default" or "Explicit Deny" pattern. Instead of a simple boolean allow, we often return a set of error messages. This provides immediate feedback to the developer when a deployment fails.
Working with Variables and Constants
We use variables to iterate over arrays in the input. In Kubernetes, this is essential because a Pod can contain multiple containers.
package kubernetes.admission
deny[msg] { # Iterate over all containers in the pod container := input.request.object.spec.containers[_]
# Check for privileged mode container.securityContext.privileged == true
msg := sprintf("Container '%v' is running in privileged mode. This is prohibited.", [container.name]) }
Handling JSON Input Data
The input object is the root of the JSON tree. When writing policies for Kubernetes, you must understand the structure of the AdmissionReview object. We frequently target input.request.object, which contains the actual manifest being submitted.
Practical Open Policy Agent Examples
Example 1: Simple Role-Based Access Control (RBAC)
While Kubernetes has native RBAC, OPA can extend it. For instance, you might want to allow a specific user to delete pods only during business hours (9 AM to 6 PM IST).
package rbac.example
default allow = false
allow { input.user == "dev-ops-lead" input.method == "DELETE" input.resource == "pods"
# Logic for checking IST business hours (simplified) time.now_ns() > 0 # Replace with actual time logic }
Example 2: Restricting HTTP Methods for API Security
If you are using OPA as a sidecar for a microservice, you can restrict which HTTP methods are allowed for specific endpoints to mitigate risks identified in the OWASP Top 10.
Example input.json
{ "method": "POST", "path": ["v1", "payment"], "user": "guest" }
package http.authz
default allow = false
allow { input.method == "GET" input.path == ["v1", "health"] }
allow { input.user == "authenticated_user" input.method == "POST" input.path == ["v1", "payment"] }
Example 3: Validating User Attributes in Middleware
We can use OPA to validate that incoming requests contain specific headers or claims, such as a valid department ID required for DPDP compliance.
package middleware.validation
deny { not input.headers["X-Department-ID"] }
deny { input.headers["X-Department-ID"] != "Finance" input.path == ["v1", "payroll"] }
Advanced OPA Rego Policy Examples
Using Iteration and Comprehensions in Rego
Rego's [_] operator is an anonymous variable that acts as an iterator. If you need to check if any element in a list meets a condition, or if all elements meet a condition, you use comprehensions.
Check if any container uses a prohibited image tag
prohibited_tags := ["latest", "dev", "test"]
deny[msg] { container := input.request.object.spec.containers[_] tag := split(container.image, ":")[1]
# Check if the tag is in the prohibited list some i; prohibited_tags[i] == tag
msg := sprintf("Image '%v' uses a prohibited tag: %v", [container.image, tag]) }
Implementing Kubernetes Admission Control Policies
In a production environment, we often need to enforce that images only come from trusted registries, such as a private Tata Cloud registry. This prevents supply chain attacks where a developer might accidentally use a malicious public image, highlighting the need for automated container scanning.
package kubernetes.admission
deny[msg] { input.request.kind.kind == "Pod" container := input.request.object.spec.containers[_]
# Deny if image does not come from the approved internal Indian registry not startswith(container.image, "cr.tata.in/")
msg := sprintf("Image '%v' comes from an untrusted registry. Only cr.tata.in is allowed.", [container.image]) }
deny[msg] { input.request.kind.kind == "Pod" container := input.request.object.spec.containers[_]
# Mitigating CVE-2022-0185 by blocking CAP_SYS_ADMIN container.securityContext.capabilities.add[_] == "SYS_ADMIN"
msg := sprintf("Privileged capability SYS_ADMIN is prohibited for container '%v'.", [container.name]) }
Managing External Data Sources in Policy Evaluation
Sometimes a policy needs data from outside the request, like a list of blacklisted IP addresses maintained in a Redis cache or a JSON file. We can load this into OPA using the --data flag or by syncing it into Gatekeeper.
Syncing external data into Gatekeeper
$ helm install gatekeeper gatekeeper/gatekeeper \ --set enableExternalData=true \ --namespace gatekeeper-system \ --create-namespace
Testing and Debugging Rego Policies
Using the OPA Playground for Real-time Testing
The OPA Playground (play.openpolicyagent.org) is an invaluable tool for rapid prototyping. You can paste your Rego code, provide a sample JSON input, and see the results immediately. However, for sensitive internal policies, I recommend using the local CLI.
Writing Unit Tests for Rego Policies
OPA has a built-in test runner. We write tests in Rego, usually in a separate file or within the same package, with the prefix test_.
package kubernetes.admission
test_deny_untrusted_registry { # Mock input input := {"request": {"object": {"spec": {"containers": [{"image": "malicious-registry.com/app:v1"}]}}}}
# Assert that deny contains the expected message count(deny) == 1 deny["Image 'malicious-registry.com/app:v1' comes from an untrusted registry. Only cr.tata.in is allowed."] }
Run the tests using the following command:
$ opa test ./policies/ -v data.kubernetes.admission.test_deny_untrusted_registry: PASS (1.2ms) --------------------------------------------------------------------------- PASS: 1/1
Common Debugging Techniques with the OPA CLI
When a policy isn't behaving as expected, use the opa eval command to inspect specific parts of the data tree.
Evaluate a specific rule against an input file
$ opa eval --data policy.rego --input input.json "data.kubernetes.admission.deny"
Use the --explain flag to see the trace of evaluation
$ opa eval --explain full --data policy.rego --input input.json "data.kubernetes.admission.deny"
Best Practices for OPA Policy Development
Optimizing Rego Performance
OPA is highly efficient, but poorly written policies can introduce latency in the Kubernetes API server (Admission Webhook timeouts).
- Avoid redundant iterations: Don't iterate over the same array multiple times in different rules.
- Use indexing: OPA automatically indexes rules that use constant keys.
- Keep input small: Only send the necessary parts of the JSON object to OPA if possible.
Structuring Policy Repositories
I recommend a directory structure that separates policies by environment or resource type.
policies/ ├── common/ │ ├── authz.rego │ └── util.rego ├── k8s/ │ ├── ingress_rules.rego │ ├── pod_security.rego │ └── test/ │ └── pod_security_test.rego └── terraform/ └── rds_encryption.rego
Integrating OPA into CI/CD Pipelines
To prevent insecure configurations from ever reaching the cluster, run opa test and conftest (a utility for OPA) in your GitLab or GitHub Actions pipelines. If a developer submits a manifest that violates a policy, the build should fail.
Example CI step for policy validation
validate_policies: stage: test script: - opa test ./policies/ -v - conftest test manifests/deployment.yaml --policy ./policies/k8s/
Handling HostPath Volume Abuse
One of the most common attack vectors we see in Indian cloud environments is the abuse of hostPath volumes. Attackers use this to mount /var/run/docker.sock or the root filesystem of the node. We can block this globally with OPA.
package kubernetes.admission
deny[msg] { input.request.kind.kind == "Pod" volume := input.request.object.spec.volumes[_]
# Check if hostPath is used volume.hostPath
# Allow only specific, non-sensitive paths if absolutely necessary allowed_paths := ["/var/log/myapp"] not any_match(volume.hostPath.path, allowed_paths)
msg := sprintf("HostPath volume '%v' uses a prohibited path: %v", [volume.name, volume.hostPath.path]) }
any_match(path, list) { path == list[_] }
This policy ensures that even if a developer has create pod permissions, they cannot escalate their privileges to the node level. This is a critical control for maintaining isolation in multi-tenant clusters.
Monitoring Gatekeeper Constraints
Once policies are deployed as Constraints and ConstraintTemplates in Kubernetes, you must monitor their enforcement status.
List all active constraint templates
$ kubectl get constrainttemplates
Check for violations of a specific constraint
$ kubectl get k8sprivilegedcontainer -o yaml
The status field of the Constraint object will list existing resources in the cluster that violate the new policy, allowing you to remediate them without immediately breaking existing workloads (if using enforcementAction: dryrun).
To verify the health of the Gatekeeper webhook which manages these evaluations:
$ kubectl get mutatingwebhookconfigurations gatekeeper-mutating-webhook-configuration
By moving security logic into Rego, you ensure that your infrastructure remains compliant with the DPDP Act 2023 and resilient against the evolving landscape of container-based exploits.
$ opa eval --data policy.rego --input input.json "data.kubernetes.admission.deny"