Implementing Zero-Trust Terminal Access: Securing SSH with Short-Lived Certificates and TLS
We recently observed a resurgence in SSH-targeted attacks following the disclosure of CVE-2024-6387, also known as regreSSHion. This signal handler race condition in OpenSSH's server (sshd) allows for unauthenticated remote code execution as root on glibc-based Linux systems. While patching is the immediate remediation, the vulnerability highlights a fundamental flaw in traditional SSH management: the reliance on long-lived credentials and persistent daemon exposure. We found that even patched systems remain vulnerable to credential theft and lateral movement if they rely on static .pem files or password authentication.
In our testing, we moved away from the "static key" model toward a Zero-Trust architecture using short-lived SSH certificates. This approach eliminates the need for authorized_keys management across thousands of nodes and provides secure SSH access for teams by binding SSH access to a centralized Identity Provider (IdP) via OIDC or SAML. This guide outlines the technical implementation of this architecture, focusing on Ubuntu and Amazon Linux environments common in Indian GCCs (Global Capability Centers).
Introduction to Secure SSH Access
What is Secure Shell Access?
Secure Shell (SSH) is the standard protocol for operating network services securely over an unsecured network. While the protocol encrypts the transport layer, the security of the access itself depends entirely on the authentication mechanism. In many legacy environments, we still see "Jump Servers" or "Bastion Hosts" that utilize static RSA keys. These keys often lack passphrases and are stored in insecure locations like local ~/.ssh directories or shared S3 buckets, creating a massive blast radius if a single developer machine is compromised.
Modern secure shell access requires a shift from "knowledge-based" or "possession-based" static factors to "ephemeral-identity-based" factors. By using TLS-wrapped identity flows to issue SSH certificates, we ensure that an attacker gaining access to a private key only has a window of minutes—rather than years—to exploit it.
The Importance of Secure SSH Access for Linux and Ubuntu Servers
Ubuntu servers form the backbone of most cloud-native infrastructures in India, from fintech startups in Bengaluru to large-scale public sector deployments. The default SSH configuration on many Ubuntu LTS releases is often too permissive for high-stakes environments. We observed that default configurations often allow for Keyboard-Interactive authentication, which is susceptible to brute-force attacks and bypasses certain hardware-backed security keys.
Implementing secure SSH access is not just a technical necessity but a regulatory one. Under the DPDP Act 2023, organizations are required to implement reasonable security safeguards to prevent personal data breaches. Since SSH access often provides a direct path to databases containing PII (Personally Identifiable Information), securing these entry points is critical to avoiding the ₹250 crore penalties stipulated by the Act.
Core Principles of SSH Secure Login
Implementing Key-Based Authentication
The first step in hardening any Linux node is the complete removal of password-based authentication. We recommend using Ed25519 keys over RSA. Ed25519 provides better security with shorter keys and faster performance. We generated our keys using the following command to ensure a high iteration count for the KDF (Key Derivation Function):
$ ssh-keygen -t ed25519 -a 100 -C "[email protected]"The -a 100 flag increases the number of KDF rounds, making the private key file significantly more resistant to offline brute-force attacks if the file is stolen. Once generated, the public key is appended to the ~/.ssh/authorized_keys file on the target server, ensuring the file permissions are strictly set to 0600.
Disabling Password-Based Logins for Enhanced Security
After verifying key-based access, we explicitly disable all other forms of authentication in /etc/ssh/sshd_config. We found that many administrators forget to disable ChallengeResponseAuthentication, which can sometimes fall back to passwords depending on the PAM (Pluggable Authentication Module) configuration.
# /etc/ssh/sshd_config
PasswordAuthentication no PubkeyAuthentication yes ChallengeResponseAuthentication no UsePAM yes AuthenticationMethods publickey
Setting AuthenticationMethods publickey forces the daemon to only accept keys, ignoring other methods even if they are enabled in PAM. After modifying the configuration, we always validate the syntax before restarting the service to avoid locking ourselves out:
$ sudo sshd -t
$ sudo systemctl restart ssh
Best Practices for Ubuntu Secure SSH Access
On Ubuntu systems, we recommend leveraging the ssh-import-id utility if you are working within a team that uses GitHub or Launchpad. However, for production environments, this should be avoided in favor of configuration management tools like Ansible or Terraform to ensure a single source of truth. We also recommend disabling X11 forwarding and agent forwarding unless specifically required, as these can be exploited by an attacker who has already compromised the server to attack the client machine.
X11Forwarding no
AllowAgentForwarding no AllowTcpForwarding yes PermitTunnel no
Managing Secure Root SSH Access
The Risks of Direct Root Login
Direct root login is a high-value target for automated scanners. If an attacker successfully authenticates as root, they bypass all local privilege escalation hurdles. Furthermore, direct root login obscures the audit trail. When multiple administrators log in as root, the auth.log fails to show which specific individual performed an action, violating the "identifiable user access" requirement in the CERT-In 2022 Cyber Security Directions.
Configuring Secure Root SSH Access: A Step-by-Step Guide
We advocate for PermitRootLogin no in all scenarios. Instead, administrators should log in as a standard user (e.g., ubuntu or ec2-user) and use sudo for privileged tasks. If your workflow absolutely demands root access via SSH, it must be restricted to specific commands or forced to use a specific key that is different from the standard user key.
To restrict root to a specific backup script, for example, we modify the authorized_keys file on the server:
# /root/.ssh/authorized_keys
command="/usr/local/bin/backup.sh",no-port-forwarding,no-x11-forwarding,no-agent-forwarding ssh-ed25519 AAAAC3Nza...
This "forced command" configuration ensures that even if the key is compromised, the attacker can only execute the backup.sh script and cannot spawn an interactive shell.
Secure Root SSH Access Best Practices (KodeKloud Methodology)
Following the KodeKloud methodology for infrastructure security, we implement a "Just-In-Time" (JIT) elevation model. Instead of persistent root keys, we use sudo with mandatory logging to a remote syslog server. This ensures that even if local logs are wiped by an attacker, the record of sudo commands persists in a secure, centralized location like an ELK stack or an AWS CloudWatch log group.
Securing SSH Access for Cloud Environments
Configuring Secure SSH Access to an EC2 Instance
In AWS environments, we observed that many teams still use the default .pem key pair generated at instance launch. This is a significant security risk because these keys are often shared among team members. We recommend using AWS Systems Manager (SSM) Session Manager instead of traditional SSH. SSM allows for terminal access over HTTPS, eliminating the need to open port 22 in Security Groups.
If traditional SSH is required, we use EC2 Instance Connect. This allows us to push a one-time-use public key to the instance metadata, which is valid for only 60 seconds. The command to connect looks like this:
$ aws ec2-instance-connect send-ssh-public-key \
--instance-id i-0123456789abcdef0 \ --instance-os-user ubuntu \ --ssh-public-key file://~/.ssh/id_ed25519.pub
Best Practices for Secure Shell Access to AWS EC2 Instances
We implement a multi-layered defense for EC2 access:
- IAM Policies: Restrict who can call
ec2-instance-connectorssm:StartSessionbased on tags. - VPC Endpoints: Use Interface VPC Endpoints for SSM so that traffic never leaves the AWS private network.
- Instance Metadata Service (IMDS): Force the use of IMDSv2 to prevent SSRF-based credential theft.
Managing Security Groups and IAM Roles for SSH
Security Groups should never be open to 0.0.0.0/0 for port 22. We use automation scripts to update Security Group rules with the current public IP of the developer's office or VPN gateway. Furthermore, the IAM role attached to the EC2 instance should have the minimum permissions necessary (Least Privilege). It should not have AdministratorAccess; rather, it should only have permissions to write logs and communicate with the required AWS services.
Advanced Hardening for Linux SSH Access
Changing Default SSH Ports to Prevent Brute Force
While changing the SSH port from 22 to something like 2222 or 49152 is "security by obscurity," we found it effectively reduces the noise in log files. Automated bots primarily scan port 22. By moving the port, we reduce the auth.log size by over 90%, making it easier to spot genuine anomalies. To change the port, edit /etc/ssh/sshd_config and update your firewall:
$ sudo sed -i 's/#Port 22/Port 2222/' /etc/ssh/sshd_config
$ sudo ufw allow 2222/tcp $ sudo ufw delete allow 22/tcp $ sudo systemctl restart ssh
Implementing Two-Factor Authentication (2FA) for SSH
For high-security environments, we integrate Google Authenticator or a hardware token like YubiKey via PAM. This ensures that even a stolen SSH private key is insufficient for access. On Ubuntu, we install the PAM module:
$ sudo apt install libpam-google-authenticator
$ google-authenticator
Then, we update /etc/pam.d/sshd to include auth required pam_google_authenticator.so and set KbdInteractiveAuthentication yes in sshd_config. Note that AuthenticationMethods must then be updated to publickey,keyboard-interactive to require both the key and the TOTP code.
Using Fail2Ban to Protect Secure Shell Access
Fail2Ban is essential for mitigating brute-force attempts. It monitors log files and bans IP addresses that show malicious signs, such as too many password failures. We use the following configuration for SSH:
# /etc/fail2ban/jail.local
[sshd] enabled = true port = 2222 filter = sshd logpath = /var/log/auth.log maxretry = 3 bantime = 1h
Implementing Zero-Trust with SSH Certificates
The pinnacle of secure SSH access is the transition from static keys to a Certificate Authority (CA) model. In this setup, the server does not hold a list of authorized keys. Instead, it trusts a CA key. Users must obtain a short-lived certificate signed by this CA to log in.
Setting Up the SSH Certificate Authority
First, we generate the CA key pair on an offline or highly secured machine (like a Vault server):
$ ssh-keygen -t ed25519 -f ssh_user_ca -C "WarnHack-Internal-CA"We then distribute the ssh_user_ca.pub to all target servers and configure sshd_config to trust it:
# /etc/ssh/sshd_config
TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem
Signing Short-Lived Certificates
When a developer needs access, they authenticate via OIDC (e.g., Google or Okta) and present their public key to the CA. The CA issues a certificate valid for, say, 1 hour. We use the following command to sign a key manually for testing:
$ ssh-keygen -s ssh_user_ca -I "session_id_8821" -n ubuntu -V +1h -z $(date +%s) id_ed25519.pubThe resulting id_ed25519-cert.pub allows the user to log in as the ubuntu principal. After 1 hour, the certificate expires, and the user must re-authenticate. This completely removes the need to revoke keys when an employee leaves the company; their access simply expires.
Automating with HashiCorp Vault
For enterprise-scale deployments, we use HashiCorp Vault's SSH secrets engine. Vault acts as the CA and integrates with corporate identity providers. A developer logs in to Vault and requests a signed key:
$ vault write ssh-client-signer/sign/developer-role \
public_key=@$HOME/.ssh/id_ed25519.pub \ valid_principals="ubuntu"
This workflow ensures that every SSH session is tied to a specific identity, providing the "identifiable user access" required by CERT-In and the DPDP Act. It also provides a centralized audit log of who requested access to which server and when.
Addressing Modern Vulnerabilities: Terrapin and regreSSHion
Mitigating CVE-2023-48795 (Terrapin Attack)
The Terrapin attack is a prefix truncation attack that targets the SSH handshake. It allows an attacker to downgrade the security of the connection by manipulating sequence numbers. We found that the most effective mitigation, short of upgrading all clients and servers, is to disable the vulnerable [email protected] cipher and any CBC-mode ciphers.
We updated our sshd_config to use only strict, modern MACs and Ciphers:
# Hardening against Terrapin
Ciphers [email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr KexAlgorithms curve25519-sha256,[email protected],diffie-hellman-group-exchange-sha256 MACs [email protected],[email protected]
Handling CVE-2024-6387 (regreSSHion)
For the regreSSHion vulnerability, the primary fix is updating OpenSSH. On Ubuntu, we ran:
$ sudo apt update && sudo apt install --only-upgrade openssh-serverIf patching is not immediately possible, we set LoginGraceTime 0 in sshd_config. While this makes the server vulnerable to a Denial of Service (DoS) by exhausting connections, it prevents the RCE exploit by removing the signal handler race condition window. In a production environment, a DoS is often preferable to a total system compromise.
Compliance and Audit: The Indian Context
CERT-In Directions and Log Retention
The Indian Computer Emergency Response Team (CERT-In) requires all service providers and corporate entities to maintain logs of their ICT systems for a period of 180 days. For SSH, this means capturing not just the login event, but the identity of the person logging in. Static keys make this impossible if keys are shared. By using the certificate-based approach, the KeyID field in the SSH certificate (which we set to the user's email) is logged by sshd, fulfilling the requirement.
DPDP Act 2023 and Administrative Access
The Digital Personal Data Protection (DPDP) Act 2023 emphasizes "Data Minimization" and "Storage Limitation." While these usually apply to customer data, they also apply to administrative logs that contain employee data. We ensure that our SSH logs are encrypted at rest and that access to the logs themselves is restricted to a few authorized security auditors, using the same Zero-Trust principles we applied to the servers.
Ongoing Monitoring and Auditing of Secure Shell Access
Security is not a static state. We utilize auditd to monitor sensitive files like /etc/ssh/sshd_config and /etc/ssh/trusted-user-ca-keys.pem. Any unauthorized modification to these files triggers an immediate alert in our SOC (Security Operations Center) for rapid threat detection.
We also implement a weekly audit of authorized_keys files across the fleet using a simple script to ensure no "rogue" keys have been added by developers trying to bypass the CA system:
import osdef check_rogue_keys(): for root, dirs, files in os.walk("/home"): if ".ssh" in dirs: auth_key_path = os.path.join(root, ".ssh/authorized_keys") if os.path.exists(auth_key_path): with open(auth_key_path, 'r') as f: keys = f.readlines() if len(keys) > 0: print(f"[!] Rogue keys found in {auth_key_path}")
if __name__ == "__main__": check_rogue_keys()
This script is integrated into our CI/CD pipeline and runs as a cron job on all production nodes, reporting results to a central dashboard.
Next Command: Verify your CA trust chain
To verify that your server correctly identifies a signed certificate and its associated principals, use the ssh-keygen -Lf command on the client side before attempting a connection. This will show the validity period and the critical options, ensuring you don't get locked out due to an expired certificate during a critical incident response.
$ ssh-keygen -Lf ~/.ssh/id_ed25519-cert.pub