The Visibility Gap in Default Kubernetes Deployments
During a recent security audit for a fintech firm based in Mumbai, I discovered that their production Kubernetes clusters, deployed via standard kubeadm scripts on local bare-metal infrastructure, were effectively blind. While they had Prometheus scraping metrics for performance tuning, the kube-apiserver was not configured to generate audit logs. In the event of a credential leak, an attacker could enumerate secrets or modify Ingress objects without leaving a trace in the persistent logs.
Kubernetes security monitoring is not merely about tracking CPU spikes; it is about capturing the intent and action of every identity within the cluster. Without centralized log analysis, your SIEM (Security Information and Event Management) is missing the most critical context: who did what, when, and from where. We start our hardening process by verifying if the API server is even recording these interactions.
$ grep -E "audit-policy-file|audit-log-path" /etc/kubernetes/manifests/kube-apiserver.yamlIf this command returns empty, your cluster is operating in a "black box" state. For organizations governed by the Digital Personal Data Protection (DPDP) Act 2023 or CERT-In mandates, this lack of logging is a direct compliance failure. CERT-In's April 2022 directions specifically require the maintenance of logs for 180 days, a requirement that cannot be met if the logs aren't being generated and shipped to a centralized repository like Wazuh or an ELK stack.
Implementing Robust Audit Policies
Enabling audit logging is only the first step; the second is defining what to log. Logging everything at the RequestResponse level will overwhelm your SIEM and inflate storage costs on providers like Netmagic or E2E Networks. We need a tiered approach. I recommend logging secrets and configmaps at the highest level while keeping pods at the Metadata level to reduce noise.
The following configuration represents a production-grade audit policy. It ensures that sensitive resource access is fully documented while ignoring the high-frequency system-level noise generated by node-to-master communication.
apiVersion: audit.k8s.io/v1
kind: Policy rules: - level: RequestResponse resources: - group: "" resources: ["secrets", "configmaps"] - level: Metadata resources: - group: "" resources: ["pods", "services"] omitStages: - "RequestReceived" - level: None userGroups: ["system:nodes"]
Applying the Policy to the Control Plane
To activate this, we modify the kube-apiserver manifest. I observed that many administrators forget to mount the host path into the pod, causing the API server to fail on restart. Ensure the --audit-policy-file flag points to the internal container path, and the volumeMounts are correctly mapped to the host's /etc/kubernetes/audit-policy.yaml.
Once applied, we verify the stream by tailing the log file directly on the master node using a web SSH terminal. This provides immediate feedback on whether the policy is too verbose or missing critical events.
$ tail -f /var/log/kubernetes/audit.log | jq .Role-Based Access Control (RBAC) and Identity Monitoring
RBAC is the primary defense mechanism in Kubernetes, yet it is frequently misconfigured. I often see "cluster-admin" privileges granted to CI/CD service accounts or developers for "troubleshooting convenience." This creates a massive attack surface. Monitoring RBAC involves not just looking at the RoleBinding objects, but actively testing the effective permissions of service accounts.
We use the can-i command to audit what a specific service account can actually do. This is essential for identifying "Shadow Admins" within the cluster.
$ kubectl auth can-i --list --as=system:serviceaccount:default:defaultDetecting Privilege Escalation via SIEM
When monitoring RBAC via a SIEM, I look for patch or update requests on ClusterRoleBindings. An attacker who has gained a foothold will often attempt to elevate their privileges by binding their compromised service account to the cluster-admin role. Your SIEM should trigger a high-severity alert the moment any identity other than the legitimate provisioning tool (like Terraform or Ansible) attempts such a modification.
In the Indian context, where many SMEs utilize managed services that might have default permissive roles, this manual verification is critical. We often find that default service accounts in the kube-system namespace have excessive permissions that can be exploited for lateral movement.
Network Policies: Hardening the Data Plane
By default, Kubernetes allows all-to-all communication between pods. This is a nightmare for incident response. If a public-facing Nginx pod is compromised, the attacker can probe every other service in the cluster, including internal databases or key-value stores. We must implement NetworkPolicies to enforce a zero-trust architecture.
I recommend a "Default Deny" ingress and egress policy for every namespace. This forces developers to explicitly declare which traffic is allowed. From a monitoring perspective, we need to track "dropped packets" or "denied connections." While standard Kubernetes logs don't show this, CNI plugins like Calico or Cilium provide detailed flow logs that can be ingested into your SIEM.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy metadata: name: default-deny-ingress spec: podSelector: {} policyTypes: - Ingress
Monitoring Traffic Anomalies
Once policies are in place, we look for anomalies. For instance, if a pod in the payments namespace attempts to connect to an external IP in a high-risk jurisdiction, or if there is a sudden surge in internal traffic between non-related services, it indicates a potential breach. Integrating these flow logs with SIEM correlation rules allows us to visualize the blast radius of a compromised container.
Runtime Security with Falco and SIEM Integration
Audit logs tell us what happened at the API level, but they don't tell us what happened inside the container. If an attacker executes apt-get update or opens a reverse shell, the API server won't see it. This is where runtime security tools like Falco become indispensable. Falco uses eBPF to monitor system calls and alerts on suspicious behavior based on a rule engine.
I've found that the most effective way to manage these alerts is to pipe them into a centralized SIEM. Instead of managing Falco alerts in isolation, we forward them via the falcosidekick component to a centralized dashboard. This allows us to correlate a "Shell spawned in container" Falco alert with an "Unauthorized Secret Access" API audit log event.
$ helm install falco falcosecurity/falco \
--set driver.kind=ebpf \ --set falcosidekick.enabled=true \ --set falcosidekick.config.webhook.address="http://siem-collector:8080"
Detecting CVE Exploitation in Real-Time
Consider CVE-2024-7646, an Ingress-nginx vulnerability that allows for the bypass of annotation validation. A SIEM monitoring the API logs would see unusual modifications to Ingress objects, while Falco would detect any subsequent unauthorized process execution if the attacker managed to gain shell access through the bypass. Monitoring for these specific patterns is much more effective than generic "threat detection."
Analyzing Kubernetes Events for Forensics
Kubernetes Events are often overlooked because they are ephemeral (persisted for only about an hour by default). However, they contain high-fidelity data about pod restarts, image pull failures, and node pressures. For forensic analysis, we must export these events to a persistent store before they are purged.
I use the following command to extract a JSON snapshot of all events across the cluster. In a production environment, we use an "event-exporter" pod to stream these directly to the SIEM.
$ kubectl get events --all-namespaces --sort-by='.lastTimestamp' -o jsonAnalyzing these events helps in identifying "Killed" pods, which might indicate a successful Buffer Overflow attack leading to a crash, or "ImagePullBackOff" errors, which could signal an attempt to pull a malicious image from a private registry that the attacker has partially compromised.
The Indian Regulatory Landscape and Log Retention
For Indian cybersecurity professionals, log retention is not just a technical preference but a legal requirement. The CERT-In directions of April 2022 mandate that service providers, intermediaries, and corporate entities maintain logs for 180 days within the Indian jurisdiction. If you are running Kubernetes on AWS (Mumbai region) or GCP (Delhi region), ensuring that your SIEM also resides within these boundaries or complies with cross-border data flow regulations under the DPDP Act is vital.
Failure to provide logs during a CERT-In investigation can result in significant penalties. I recommend a multi-tier storage strategy:
- Hot Storage (7-15 days): For immediate incident response and SIEM correlation.
- Warm Storage (30-60 days): For trend analysis and hunting.
- Cold Storage (180+ days): In compressed, encrypted buckets (like S3 or local MinIO) to satisfy regulatory requirements.
Monitoring Container Runtime Activity and Resource Anomalies
Resource usage spikes are often the first indicator of cryptojacking. In several clusters I've investigated in the Bengaluru tech corridor, I found that unauthorized xmrig miners were deployed via misconfigured Jenkins instances. While these miners were hidden from kubectl get pods (by using naming conventions that mimicked system pods), they were easily spotted by monitoring CPU usage anomalies at the node level.
We use Prometheus to track the container_cpu_usage_seconds_total metric. When this deviates significantly from the baseline established over the previous 30 days, an alert is triggered. However, we must correlate this with the pod_owner to ensure we aren't alerting on legitimate batch processing jobs.
$ curl -sSk -H "Authorization: Bearer $TOKEN" https://<master-ip>:6443/logs/kube-apiserver.logIf you can access the logs via the API server directly as shown above, it means your RBAC is likely too permissive. This endpoint should be strictly restricted to the logging infrastructure itself.
Identifying Misconfigurations in Real-Time
Misconfigurations are the leading cause of Kubernetes breaches. Using Admission Controllers like Kyverno or OPA Gatekeeper allows us to block non-compliant resources before they ever reach the etcd database. However, we still need to monitor "Allowed" requests that might be borderline or part of a larger attack chain.
For example, CVE-2023-5528 highlighted issues with input sanitization in storage plugins. Monitoring for unusual PersistentVolumeClaim (PVC) requests, especially those targeting Windows nodes or specific storage classes, can provide an early warning of an exploitation attempt.
Tracking Certificate Expiration
Expired certificates are a common cause of cluster downtime and can be exploited during "Man-in-the-Middle" attacks if validation is disabled as a "temporary fix." We must monitor the validity of the API server and Kubelet certificates.
$ openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout | grep -A 2 "Subject Alternative Name"Your monitoring stack should alert you 30 days before these certificates expire. Automating this via cert-manager is the industry standard, but the monitoring of the Certificate resource status remains a security priority.
Integrating Security Monitoring into CI/CD Pipelines
Security cannot be an afterthought; hardening the developer workspace must be integrated into the deployment pipeline. We use "Static Analysis" on Helm charts and YAML manifests before they are deployed. Tools like checkov or terrascan can be integrated into GitHub Actions or GitLab CI to catch issues like "Privileged: true" or "RunAsRoot: true" before they hit the cluster.
When a deployment is blocked by a security gate, this event should also be logged in the SIEM. This allows the security team to identify patterns—for instance, if a specific development team consistently attempts to deploy insecure configurations, it indicates a need for targeted training or updated base images.
Forensic Analysis and Incident Response
When an incident is detected, the speed of response is governed by the accessibility of your logs. If your logs are scattered across 50 nodes, you've already lost. Centralizing logs into a SIEM allows for "Pivot Analysis." You can start with a suspicious IP address found in the Nginx logs, pivot to the Pod name, then to the Service Account, and finally to the specific kubectl exec command that was used to compromise the system.
In a recent forensic exercise, we used centralized audit logs to prove that an attacker had exploited a leaked kubeconfig file. By filtering for userAgent strings that didn't match our internal CLI versions, we isolated every action taken by the adversary within seconds.
# Example: Searching SIEM logs for unauthorized exec commands
{ "verb": "create", "objectRef": { "resource": "pods", "subresource": "exec" }, "user": { "username": "compromised-service-account" } }
Building a Resilient Security Posture
The transition from "Observability" to "Security Monitoring" requires a shift in mindset. We are no longer looking for why a service is slow; we are looking for why a service is behaving differently. As Kubernetes environments grow in complexity, especially with multi-cloud deployments across providers like AWS and local Indian data centers, centralization becomes the only way to maintain a coherent security posture.
The future of Kubernetes security lies in eBPF-powered deep packet inspection and automated response. However, these advanced techniques are useless if the fundamentals—audit logging, RBAC monitoring, and network policy enforcement—are not in place. By aligning your monitoring strategy with frameworks like the MITRE ATT&CK for Containers and local regulations like the DPDP Act, you create a resilient environment capable of withstanding modern threats.
Next Command: Verify the integrity of your kube-apiserver by checking for unauthorized modifications to its static manifest file, which could indicate host-level compromise.
$ stat /etc/kubernetes/manifests/kube-apiserver.yaml