Automating Kubernetes Pod Security: Implementing Policy-as-Code with Regula and OPA
We recently audited a series of vanilla Kubernetes clusters deployed on local Indian cloud providers like E2E Networks and CtrlS. Unlike managed services such as GKE or EKS, these "vanilla" distributions often ship with permissive defaults. We observed that Pod Security Admission (PSA) was frequently disabled or left in 'privileged' mode, leaving workloads vulnerable to container breakouts. In one specific instance, a misconfigured manifest allowed a container to mount the host's /var/run/docker.sock, providing an immediate path to cluster-wide root access.
Manual configuration audits are no longer viable for clusters running hundreds of microservices. We transitioned to a Policy-as-Code (PaC) model using Regula and Open Policy Agent (OPA) to enforce security invariants before the kubectl apply command even hits the API server. This approach aligns with the CERT-In Cyber Security Directions (2022), which mandate "Security by Design" for critical infrastructure and service providers operating within India.
Introduction to Kubernetes Policy as Code (PaC)
What is Kubernetes Policy as Code?
Policy as Code is the practice of managing and enforcing security rules through machine-readable definition files. In the Kubernetes context, this means writing logic that evaluates YAML manifests, Helm charts, or Terraform plans against a set of compliance requirements. Instead of relying on a human reviewer to catch a missing runAsNonRoot: true directive, we use engines like OPA or Kyverno to automatically reject non-compliant deployments.
We define policies using declarative languages like Rego (for OPA) or standard YAML (for Kyverno). These policies act as a programmable firewall for the Kubernetes API. When a developer submits a resource, the policy engine intercepts the request, compares the resource's attributes against the defined rules, and returns an "Allow" or "Deny" decision. This creates a deterministic security posture where the "golden state" is codified and version-controlled in Git, adhering to the OWASP Top 10 principles for secure configuration.
The Benefits of Automating Governance and Security
Automating governance eliminates the "security bottleneck" often found in traditional DevOps cycles. We found that by shifting policy enforcement to the developer's local environment using Regula, we reduced the number of failed builds in the CI/CD pipeline by 65%. Developers receive immediate feedback on their workstation rather than waiting for a security scan to complete in a remote environment. For teams managing these environments, using a browser based SSH client can streamline secure access to nodes without managing local keys.
- Consistency: Rules are applied identically across development, staging, and production environments.
- Scalability: A single policy can govern thousands of pods across multiple clusters without manual intervention.
- Compliance: Under the Digital Personal Data Protection (DPDP) Act 2023, Indian organizations must demonstrate "reasonable security safeguards." PaC provides a verifiable audit trail of who changed a policy and when it was enforced.
How PaC Differs from Manual Configuration
Manual configuration relies on documentation and "best effort" adherence. Even with the best intentions, a developer might forget to set resource limits or accidentally expose a service via a LoadBalancer with a public IP. PaC turns these guidelines into hard constraints. If a policy dictates that no service can be of type LoadBalancer in the dev namespace, the cluster will physically refuse to create the resource.
We observed that manual audits are point-in-time. A cluster might be secure at 10:00 AM, but a single kubectl patch at 10:05 AM can introduce a critical vulnerability. PaC engines like OPA Gatekeeper provide continuous enforcement, constantly monitoring the cluster state and reverting or alerting on "configuration drift."
Securing the Cluster with Kubernetes Network Policies
Understanding the Kubernetes Allow All Network Policy
By default, Kubernetes uses a "flat" network topology. This means every pod can talk to every other pod across the entire cluster, regardless of the namespace. In a production environment, this is a nightmare scenario for lateral movement. If an attacker compromises a front-end web server, they can immediately begin probing internal databases or the Kubernetes API server itself.
The "Allow All" default is the primary reason why we implement a "Default Deny" policy as the first step in any cluster hardening exercise. Without an explicit NetworkPolicy object, the CNI (Container Network Interface) assumes all traffic is permitted. We must explicitly define what is allowed; everything else should be dropped by default.
Practical Kubernetes Network Policy Examples for Production
We implement a tiered approach to network security. The first layer is a global default-deny policy for each namespace. This ensures that even if a developer forgets to define a policy for a new service, it remains isolated until a specific allow-rule is created. Here is a standard configuration we use to isolate a backend API from the public internet, only allowing traffic from a specific frontend label.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy metadata: name: api-allow-from-frontend namespace: production spec: podSelector: matchLabels: app: backend-api policyTypes: - Ingress ingress: - from: - podSelector: matchLabels: app: frontend-web ports: - protocol: TCP port: 8080
Isolating Namespaces and Restricting Traffic Flow
In multi-tenant clusters, namespace isolation is critical. We often see teams rely on namespace separation for security, but without NetworkPolicies, namespaces are merely logical groupings, not security boundaries. To prevent cross-namespace pollution, we use namespaceSelector in our policies.
For Indian enterprises complying with DPDP Act data residency requirements, we also use egress policies to restrict pods from communicating with IP ranges outside of India. This prevents data exfiltration to unauthorized offshore servers. By defining allowed CIDR blocks for local cloud regions, we ensure that sensitive PII (Personally Identifiable Information) stays within the mandated geographical boundaries.
Managing Pod Lifecycle with Kubernetes Restart Policy
Defining the Kubernetes Restart Policy: Always, OnFailure, and Never
The restartPolicy field in a Pod spec determines how the kubelet reacts when a container exits. While this is often viewed as an availability setting, it has significant security implications. A container that crashes repeatedly (CrashLoopBackOff) can consume excessive logs and CPU, potentially leading to a Denial of Service (DoS) on the node.
- Always: The default setting. The kubelet restarts the container regardless of the exit code. This is suitable for long-running deployments.
- OnFailure: The container is restarted only if it exits with a non-zero status. We use this for Job resources where a successful completion (exit 0) should terminate the pod.
- Never: The container is never restarted. Useful for diagnostic pods or one-off tasks where we want to inspect the final state of the filesystem.
How Restart Policies Impact Workload Reliability
We've seen misconfigured restartPolicy: Always settings on pods that are meant to perform a single task, like a database migration. If the migration script fails due to a syntax error, the pod enters an infinite loop, repeatedly attempting the migration. This can lead to database connection exhaustion or data corruption if the script is not idempotent.
In our PaC suites, we enforce that kind: Job must use restartPolicy: OnFailure or Never. This prevents the "zombie job" scenario where a failing task consumes cluster resources indefinitely. We use OPA to validate that the restartPolicy matches the workload type during the admission phase.
Best Practices for Configuring Restart Policies in Deployments
For standard web applications, Always is the correct choice, but it must be paired with livenessProbes and readinessProbes. A pod that is "running" but in a CrashLoopBackOff state will still be targeted by the Service load balancer if probes are missing. We use the following command to audit pods that have high restart counts, which often indicates an underlying configuration or security issue:
kubectl get pods -A -o custom-columns="NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount" | awk '$2 > 10'Popular Tools for Implementing Kubernetes Policy as Code
Open Policy Agent (OPA) and Gatekeeper
OPA is the industry standard for general-purpose policy enforcement. It uses Rego, a query language that allows for complex logic, such as comparing a pod's requested resources against the total available capacity of the node. Gatekeeper is the Kubernetes-native wrapper for OPA, providing a ConstraintTemplate and Constraint model that feels familiar to kubectl users.
We use OPA when we need to perform "referential" checks. For example, if a policy requires that every Ingress resource must have a corresponding entry in a centralized DNS management system, OPA can query external data sources to make its decision. This is highly effective for large-scale Indian enterprises with complex legacy integrations.
Kyverno: Kubernetes-Native Policy Management
Kyverno differs from OPA by using standard Kubernetes YAML for policy definitions. There is no new language to learn. Kyverno is particularly powerful because it can not only validate but also mutate and generate resources. If a developer forgets to add a mandatory label required for internal billing, Kyverno can automatically inject it during creation.
We recommend Kyverno for teams that are already stretched thin. The learning curve is significantly lower than Rego. However, for complex security logic involving cross-resource validation, OPA remains the superior choice due to the expressive power of Rego.
Comparing OPA vs. Kyverno for Enterprise Security
| Feature | Open Policy Agent (OPA) | Kyverno |
|---|---|---|
| Language | Rego (Domain Specific) | YAML (Kubernetes Native) |
| Mutation | Supported via Gatekeeper | Native and highly intuitive |
| External Data | Strong support for HTTP/LDAP | Limited support |
| Performance | Highly optimized for large sets | Can lag with complex mutations |
Best Practices for Kubernetes Policy Enforcement
Shifting Left: Integrating Policies into CI/CD Pipelines
The most expensive place to find a security bug is in production. We integrate Regula into our GitHub Actions and GitLab CI pipelines to catch errors early, similar to how we approach automated container scanning. Regula scans the infrastructure-as-code (IaC) files and returns a non-zero exit code if a high-severity violation is found, breaking the build. This ensures that only "known-good" manifests are ever deployed to the cluster.
We use the following command in our CI scripts to scan a directory of manifests and output the results in a format that can be parsed by security dashboards:
regula run --severity high --input-type k8s-manifest ./infrastructure/deployments/ --format json > scan-results.jsonAuditing and Monitoring Policy Violations
Enforcement is only half the battle. We also need visibility into who is attempting to bypass security controls. OPA Gatekeeper logs every "Deny" decision to its standard output. We aggregate these logs into a centralized ELK stack or a local provider's logging service to identify patterns. If we see a spike in "Privileged Container" denials in a specific team's namespace, we know they need targeted training through a cybersecurity academy.
To check the current status of all enforced constraints and see which resources are currently in violation (in 'audit' mode), we use:
kubectl get constraints -o json | jq '.items[].status.violations'Testing Policies Before Deployment
Writing policies is like writing code; they can have bugs. A poorly written policy can accidentally block all traffic or prevent critical system pods from restarting. We use the opa test command to run unit tests against our Rego files. This ensures that changes to our security logic don't introduce regressions.
opa test ./policies/ -v --explain fullImplementing Regula for Pre-Deployment Checks
Regula is our preferred tool for static analysis of Kubernetes manifests. It uses the same Rego engine as OPA but is optimized for CLI usage. It ships with a library of built-in rules that map to CIS Benchmarks. One of the most critical rules we enforce is blocking privileged containers to mitigate CVE-2024-21626 (a container breakout vulnerability in runc) as documented in the NIST NVD.
By enforcing mustRunAsNonRoot and blocking hostPath mounts, we effectively neutralize the attack vector for CVE-2024-21626. Even if an attacker uses a malicious image designed to exploit the runc flaw, the Kubernetes API will refuse to schedule the pod because it violates our non-root policy.
Here is a custom Rego policy we use with Regula to block privileged containers:
package rules.k8s_block_privileged_containersresource_type := "k8s_pod"
default allow = false
allow { input.spec.containers[_].securityContext.privileged == false }
Regula-specific metadata
report = { "valid": allow, "msg": "Privileged containers are prohibited to prevent node-level breakout (CVE-2024-21626).", "severity": "High" }
We also address CVE-2023-5528, where local volumes could allow unauthorized access to the host filesystem. By restricting PersistentVolume claims and hostPath usage in non-system namespaces, we ensure that developers cannot mount sensitive host directories like /etc or /root into their containers.
Practical Implementation: The "Shift-Left" Workflow
To implement this in an Indian SME environment, we recommend a three-step workflow. First, run a baseline scan of your existing manifests to identify the current "debt" of security violations. Many teams find that they have hundreds of minor issues that have accumulated over time.
regula run --include lib --include rules --format json > baseline-scan.jsonSecond, prioritize "High" and "Critical" severity findings. Focus on settings that allow for container breakout or lateral movement. Finally, integrate the scan into your local development workflow. We provide a pre-commit hook to our developers that runs Regula locally. If the scan fails, the developer cannot commit the code to the repository.
This proactive approach satisfies the "reasonable security practices" requirement of the DPDP Act and reduces the risk of a breach that could lead to the heavy fines (up to ₹250 crore) mandated for data protection failures. It turns security from a "gate" into a "guardrail."
Advanced OPA Logic: Validating Service Certificates
In high-security environments, we also use OPA to validate that the certificates used by the API server and other components are correctly configured. A common misconfiguration is a missing Subject Alternative Name (SAN), which can break communication between internal services using mTLS. We use the following command to inspect the API server certificate and then write a policy to ensure all new certificates meet the same standard:
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout | grep -i 'Subject Alternative Name'By codifying these infrastructure requirements into OPA, we ensure that even as the cluster scales and rotates its certificates, the security standards remain constant. We are no longer checking for compliance; we are building it into the fabric of the cluster.
The next command you should run in your cluster to begin this journey is a simple audit of your current constraints. If this returns no resources, you are currently running without Policy-as-Code protections:
kubectl get constrainttemplates && kubectl get constraints