Technical Observation: The Default Ingress Insecurity
During a recent audit of a production cluster hosted on a local Indian cloud provider, I discovered that the default NGINX Ingress Controller configuration allowed TLS 1.0 and 1.1 by default. This configuration fails to meet the compliance standards set by the RBI’s Master Direction on IT Outsourcing and the DPDP Act 2023, which mandate strong encryption for data in transit. Following a secure configuration guide for NGINX is critical to prevent such vulnerabilities. Many teams assume that adding a tls block to their Ingress manifest is sufficient, but without explicit hardening of cipher suites and protocol versions, the cluster remains vulnerable to downgrade attacks like POODLE or BEAST.
I observed that while the Ingress resource simplifies traffic routing, it abstracts away the underlying complexity of the TLS handshake. For administrators managing these clusters, using a browser based SSH client simplifies secure access to the underlying nodes without exposing traditional ports. When a client initiates a connection, the Ingress controller—acting as a reverse proxy—must present a valid certificate, negotiate a cipher suite, and optionally verify the client's identity. In Kubernetes, this process is decentralized, relying on Secret objects that must be scoped correctly within namespaces to prevent unauthorized access to private keys.
What is SSL/TLS Termination in Kubernetes?
SSL/TLS termination occurs at the edge of the cluster, typically at the Ingress controller level. The controller decrypts the incoming HTTPS traffic and forwards it as plain HTTP to the backend services. This offloads the computational overhead of decryption from the application pods to the controller. However, in high-security environments, such as Indian fintech startups integrating with the Account Aggregator (AA) ecosystem, termination at the edge is often insufficient; we frequently implement "SSL Passthrough" or re-encryption to ensure data remains encrypted until it reaches the application container.
The Ingress controller uses a specific certificate for each host defined in the rules. If no certificate is found, most controllers fall back to a "Default SSL Certificate," which is often a self-signed dummy cert. I have seen this lead to significant downtime when automated scanners or health checks reject the self-signed certificate, causing the LoadBalancer to mark the entire ingress as unhealthy.
The Importance of Securing Ingress Traffic
Securing Ingress traffic is not just about encryption; it is about identity and integrity. Without proper SSL/TLS configuration, an attacker performing a Man-in-the-Middle (MITM) attack can intercept sensitive PII (Personally Identifiable Information). Under the DPDP Act 2023, failing to implement "reasonable security safeguards" to prevent data breaches can result in penalties up to ₹250 crore. For Indian businesses, implementing robust threat detection and log monitoring is a financial and legal necessity.
We must also consider the risk of internal lateral movement. If the Ingress controller is compromised, or if traffic between the Ingress and the backend service is unencrypted, an attacker who has breached a single pod can sniff traffic across the internal network. This is why mTLS (Mutual TLS) is becoming the standard for service-to-service communication within the cluster, as outlined in the Kubernetes security documentation.
How Ingress Controllers Handle Encrypted Connections
The Ingress controller monitors the Kubernetes API for changes to Ingress resources and TLS Secrets. When a new Ingress is created, the controller updates its internal configuration (e.g., nginx.conf for NGINX or traefik.toml for Traefik). I tested the reload latency on a cluster with 500+ Ingress rules and found that NGINX can experience slight traffic blips during configuration reloads, whereas Traefik handles dynamic updates more gracefully through its provider architecture.
Understanding Kubernetes TLS Secrets
Kubernetes stores SSL/TLS certificates in a specific type of Secret called kubernetes.io/tls. This secret must contain two keys: tls.crt (the certificate chain) and tls.key (the private key). One common mistake I see is developers using a generic Opaque secret type, which can lead to mounting issues or the Ingress controller failing to recognize the credentials. This abstraction can sometimes lead to URL validation bypass issues if the backend logic relies on headers that the Ingress fails to sanitize.
The Secret must reside in the same namespace as the Ingress resource. This is a critical security boundary. If you have a multi-tenant cluster where different teams occupy different namespaces, Team A cannot use a TLS Secret located in Team B's namespace. To share certificates across namespaces, you either need to replicate the secret or use a tool like cert-manager with a ClusterIssuer.
The Role of the Ingress Resource Spec
The spec.tls section of the Ingress manifest defines which hosts are covered by which secrets. It is an array, allowing you to specify different certificates for different subdomains. I have found that explicitly listing every subdomain, rather than relying on wildcards, provides better granularity for monitoring and auditing. Here is how the basic structure looks:
spec:
tls: - hosts: - api.warnhack.in secretName: warnhack-tls-secret rules: - host: api.warnhack.in http: paths: - path: / pathType: Prefix backend: service: name: api-service port: number: 80
Supported Ingress Controllers: NGINX, Traefik, and HAProxy
While NGINX is the "de facto" standard, Traefik and HAProxy offer different advantages for SSL/TLS management. Traefik has native support for Let's Encrypt and can handle ACME challenges without external tools. HAProxy is often preferred in high-throughput environments because of its superior SSL termination performance. I observed that NGINX Ingress tends to have the most extensive documentation for complex annotations, making it easier to implement specific hardening measures like custom cipher suites.
Generating a Private Key and Certificate Signing Request (CSR)
For internal services or testing environments, we often generate our own CA and sign certificates manually. This process helps in understanding the trust chain. I use the following commands to generate a robust 2048-bit RSA key and a CSR. It is important to adhere to OpenSSH security standards when managing remote keys and certificates. Using 4096-bit keys is more secure but can significantly increase the TLS handshake time, which may impact latency-sensitive applications.
$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout ingress-ca.key \ -out ingress-ca.crt \ -subj "/CN=WarnHack-CA/O=Security-Dept"
Once the CA is created, we generate the server key and the CSR for our specific domain. It is vital to include the Subject Alternative Name (SAN) extension, as modern browsers and Go-based tools (like many K8s components) no longer rely on the Common Name (CN) for validation.
Creating a Kubernetes TLS Secret via kubectl
After obtaining the signed tls.crt and the tls.key, we inject them into the cluster. I prefer using the kubectl create secret tls command over manually writing YAML to avoid accidentally committing base64-encoded private keys to Git repositories. This is a common source of credential leakage in the Indian startup ecosystem.
$ kubectl create secret tls ingress-tls-prod \
--cert=tls.crt \ --key=tls.key \ -n networking
Verify the secret creation and inspect the metadata to ensure the type is correctly set to kubernetes.io/tls. If the secret is not visible to the Ingress controller, check the RBAC permissions of the controller's ServiceAccount; it must have get and list permissions for secrets in that namespace.
Configuring the Ingress Manifest to Use TLS Secrets
The final step in manual setup is mapping the secret to the host. I often see teams forget to update the hosts list in the tls block, which leads to the controller serving the default certificate instead of the one provided in the secret. We can verify which certificate is being served using openssl s_client.
$ echo | openssl s_client -showcerts -servername api.warnhack.in -connect :443 2>/dev/null | openssl x509 -inform pem -noout -textIntroduction to Cert-Manager and Its Architecture
Manual certificate management is unsustainable for large-scale clusters. cert-manager is the industry standard for automating the issuance and renewal of certificates. It introduces several Custom Resource Definitions (CRDs): Issuers, ClusterIssuers, and Certificates. In my experience, using ClusterIssuers is more efficient for organization-wide policies, while Issuers are better for namespace-isolated environments.
The architecture relies on a control loop that watches for Certificate resources. When a certificate is nearing expiry, cert-manager initiates a challenge (HTTP-01 or DNS-01) with the Certificate Authority (CA), such as Let's Encrypt or ZeroSSL. For Indian companies using local DNS providers that might not have a native plugin, the HTTP-01 challenge is usually the most straightforward path, provided the Ingress is reachable from the internet.
Installing Cert-Manager in Your Cluster
I recommend using Helm for the installation to manage the lifecycle of the CRDs easily. Ensure that you enable the installCRDs flag. Without it, the installation will fail silently when it tries to create the Issuer resources.
$ helm repo add jetstack https://charts.jetstack.io
$ helm repo update $ helm install cert-manager jetstack/cert-manager \ --namespace cert-manager \ --create-namespace \ --set installCRDs=true
Configuring Issuers and ClusterIssuers
A ClusterIssuer is a cluster-scoped resource. For production environments, I always configure both a staging and a production issuer for Let's Encrypt. Let's Encrypt production has strict rate limits, and testing your configuration against the staging API prevents your IP from being temporarily banned due to repeated failed challenges.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer metadata: name: letsencrypt-prod spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: [email protected] privateKeySecretRef: name: letsencrypt-prod-account-key solvers: - http01: ingress: class: nginx
Automating Let's Encrypt SSL Certificates
Once the ClusterIssuer is active, you can automate certificate issuance by adding an annotation to your Ingress resource. cert-manager will detect the annotation, create a Certificate resource, and handle the challenge. This is the most effective way to ensure that certificates are always valid and renewed 30 days before expiration.
metadata:
annotations: cert-manager.io/cluster-issuer: "letsencrypt-prod" spec: tls: - hosts: - app.warnhack.in secretName: app-tls-cert
Enabling HTTP to HTTPS Redirection
Allowing unencrypted HTTP traffic is a security risk. Most Ingress controllers allow you to enforce HTTPS redirection via annotations. In NGINX, this is enabled by default in many installations, but I prefer to explicitly set it to avoid ambiguity. This ensures that any request hitting port 80 is immediately redirected with a 308 Permanent Redirect to port 443.
metadata:
annotations: nginx.ingress.kubernetes.io/ssl-redirect: "true" nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
Implementing HSTS (HTTP Strict Transport Security)
HSTS tells the browser that it should only ever interact with the site using HTTPS. This prevents SSL stripping attacks. When implementing HSTS, I recommend starting with a short max-age and gradually increasing it. For Indian banking integrations, a max-age of one year (31536000 seconds) is typically required during security audits.
metadata:
annotations: nginx.ingress.kubernetes.io/configuration-snippet: | more_set_headers "Strict-Transport-Security: max-age=31536000; includeSubDomains; preload";
Configuring Mutual TLS (mTLS) for Service-to-Service Security
mTLS requires the client to present a certificate that the server (Ingress) validates against a trusted CA. This is critical for B2B APIs where you only want to allow specific partners to access your endpoints. In the Indian Account Aggregator framework, mTLS is often used to secure the data transfer between the Financial Information Provider (FIP) and the Financial Information User (FIU).
To implement this, we first create a secret containing the CA certificate of our trusted clients. Then, we configure the Ingress to require and verify the client certificate.
apiVersion: networking.k8s.io/v1
kind: Ingress metadata: name: mtls-secured-ingress annotations: nginx.ingress.kubernetes.io/auth-tls-verify-client: "on" nginx.ingress.kubernetes.io/auth-tls-secret: "networking/ca-secret" nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1" nginx.ingress.kubernetes.io/ssl-protocols: "TLSv1.2 TLSv1.3" nginx.ingress.kubernetes.io/ssl-ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256" spec: tls: - hosts: - api.internal.warnhack.in secretName: ingress-tls-prod rules: - host: api.internal.warnhack.in http: paths: - path: / pathType: Prefix backend: service: name: backend-service port: number: 443
Using Wildcard Certificates across Multiple Namespaces
Managing individual certificates for 50+ microservices is a nightmare. Wildcard certificates (e.g., *.warnhack.in) simplify this. However, since TLS secrets are namespaced, you face a challenge. I use the Reflector controller or kubed to automatically sync the wildcard secret from a "gold" namespace (like cert-manager) to all other namespaces. This ensures that as soon as the wildcard cert is renewed by cert-manager, it is updated cluster-wide.
Debugging 'Certificate Not Valid' Browser Errors
When a browser shows a "Certificate Not Valid" error, the first thing I check is the certificate chain. Often, the intermediate certificate is missing from the tls.crt file. The tls.crt should contain the server certificate followed by the intermediate CA certificate. If you only provide the server certificate, the browser cannot verify the trust chain back to a root CA.
Use kubectl describe ingress <name> to see if the controller has successfully picked up the secret. If you see an "Event" stating "Error getting SSL certificate," it usually means the secret name is misspelled or it exists in a different namespace.
Fixing Secret Mismatch and Namespace Scoping Issues
If you have multiple Ingress controllers in a single cluster (e.g., one for internal traffic and one for external), ensure that your Ingress resource has the correct ingressClassName. I've spent hours debugging SSL issues only to find that the internal NGINX controller was trying to process an Ingress meant for the external HAProxy controller, leading to certificate mismatches.
$ kubectl get ingress -A
$ kubectl get ingressclass
Checking Cert-Manager Logs for Challenge Failures
If cert-manager fails to issue a certificate, the logs of the cert-manager pod are your best friend. Common failures include 503 errors during the HTTP-01 challenge because the .well-known/acme-challenge path was not correctly routed. I use the following command to tail the logs and watch the challenge process in real-time.
$ kubectl logs -n cert-manager -l app.kubernetes.io/instance=cert-manager | grep "error"Automating Certificate Renewal and Monitoring
Renewal should never be manual. Beyond cert-manager, you need monitoring. I use Prometheus with the blackbox_exporter to probe the Ingress endpoints and alert us when a certificate has less than 15 days of validity remaining. This provides a safety net in case the cert-manager renewal loop fails due to an API change or DNS issue.
# Prometheus Alert Rule
- alert: SSLCertExpiringSoon
expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 15 for: 10m labels: severity: critical annotations: summary: "SSL certificate for {{ $labels.instance }} expires in less than 15 days"
Restricting Cipher Suites and TLS Versions
To pass a security audit in the Indian financial sector, you must disable weak ciphers. I observed that many NGINX Ingress installations still allow CBC-mode ciphers, which are vulnerable to various attacks. You should restrict your configuration to use only GCM-based ciphers and TLS 1.2 or 1.3. This can be done globally via the NGINX ConfigMap or per-ingress via annotations.
$ kubectl exec -it -n ingress-nginx -- grep -A 10 'ssl_ciphers' /etc/nginx/nginx.confIntegrating with External Secrets Management
For organizations with mature security postures, storing private keys as Kubernetes Secrets (which are only base64 encoded and stored in etcd) is insufficient. We integrate with HashiCorp Vault or AWS Secrets Manager using the External Secrets Operator (ESO). This allows the cluster to fetch the TLS certificate directly from a hardened HSM-backed vault, ensuring that the private key is never stored in the application's Git repository or exposed through kubectl get secrets to unauthorized users.
When using ESO, the workflow involves defining an ExternalSecret resource that maps a Vault path to a Kubernetes Secret. The operator handles the synchronization and rotation, providing a seamless bridge between your centralized secret management and the Kubernetes Ingress controller.
Next technical step: Audit your current Ingress controller's SSL configuration by running nmap --script ssl-enum-ciphers -p 443 <your-ingress-ip> to identify any deprecated protocols (TLS 1.0/1.1) or weak ciphers (3DES, RC4) that must be disabled to comply with DPDP Act standards.