The Failure of Static Key Management
I recently audited a production environment in the AWS Mumbai region where over 400 unique authorized_keys entries were scattered across 50 instances. This is a common architectural debt in Indian SMEs. When an engineer leaves the organization, the manual overhead of rotating these keys across a fleet of servers is rarely performed correctly. Using a team SSH access management platform can mitigate these risks. We often find "ghost keys" belonging to former employees still granting root access months after their offboarding.
The traditional SSH model relies on "Trust on First Use" (TOFU). When you first connect to a server, you are prompted to accept a fingerprint. Most users blindly type "yes," creating a massive opening for Man-in-the-Middle (MITM) attacks. By moving to an SSH Certificate Authority (CA), we eliminate both the "ghost key" problem and the TOFU prompt by implementing Zero Trust SSH with a cryptographically verified source of truth.
What is an SSH Certificate Authority?
An SSH CA is not the same as an X.509 CA used for Web SSL/TLS. It is a simpler, specialized implementation within OpenSSH. The CA consists of a standard SSH key pair. The private key is used to sign user and host public keys, generating a ".pub-cert" file. This certificate contains the user's public key, their identity, valid principals (usernames), and a strict expiration timestamp.
The target server does not need to know about the user's individual public key. It only needs to trust the CA's public key. When a user presents a signed certificate, the server verifies the CA's signature and checks if the certificate is still valid. If it is, access is granted. This shifts the burden of identity management from the target hosts to a centralized signing process.
SSH Keys vs. SSH Certificates: Why Certificates are Superior
Static SSH keys are permanent credentials. Once placed in an authorized_keys file, they remain valid until manually removed. SSH certificates, however, are ephemeral. We typically issue certificates with a Time-to-Live (TTL) of 8 to 12 hours. Once the certificate expires, it becomes a useless string of bits, mitigating the risk of credential exfiltration.
| Feature | Static SSH Keys | SSH Certificates |
|---|---|---|
| Trust Model | TOFU (Trust on First Use) | CA-based cryptographic trust |
| Expiration | None (Manual rotation) | Native TTL (e.g., 8 hours) |
| Revocation | Manual removal from every host | Centralized Key Revocation List (KRL) |
| User Management | O(N) - Scalability issues | O(1) - Constant complexity |
Benefits of Implementing SSH Certificate Authentication
In our testing, the most immediate benefit was the elimination of lateral movement risks. In many breaches, attackers exfiltrate ~/.ssh/id_rsa files from developer workstations. If those keys are static, the attacker gains indefinite access. With a CA setup, even if a certificate is stolen, its short lifespan (often matching a single work shift) significantly narrows the window of opportunity.
Eliminating TOFU Security Risks
When we implement Host Certificates, the client's SSH agent verifies the server's identity against the CA before the user even enters a password or provides a key. This prevents the "Warning: Remote Host Identification Has Changed" messages that users have been conditioned to ignore. If the server's certificate doesn't match the CA, the connection is dropped immediately.
Simplified User Lifecycle Management
For organizations governed by the DPDP Act 2023, maintaining strict access control is a legal necessity. With an SSH CA, we don't need to run Ansible playbooks to add or remove users from authorized_keys across 1,000 servers. We simply stop signing certificates for that user at the central CA. Their access across the entire infrastructure vanishes the moment their current certificate expires.
Enforcing Short-Lived Access and Automatic Expiration
We configure the CA to sign keys with a -V (validity) flag. For example, -V +8h ensures the key is only valid for the next 8 hours. This aligns with standard shift patterns. If an engineer needs to perform an emergency change at 2:00 AM, they must authenticate with the CA (ideally via an SSO provider like Okta or Azure AD) to get a fresh certificate.
Prerequisites for SSH Key Server Setup
We recommend a dedicated, hardened Linux instance for the CA. This host should have minimal network exposure. In a high-security environment, the CA private key should never even reside on the filesystem; it should be stored in a Hardware Security Module (HSM) or a cloud-native KMS.
Hardware and OS Requirements
- OS: Ubuntu 22.04 LTS or RHEL 9 (any modern Linux with OpenSSH 8.0+).
- Resources: Minimal (1 vCPU, 2GB RAM is sufficient).
- Network: Isolated VPC subnet with restricted ingress (Port 22 only from admin jumpbox).
Required OpenSSH Versions
Ensure all target hosts and clients are running OpenSSH 7.2 or higher. Earlier versions have known vulnerabilities in certificate parsing, as noted in the NIST NVD database. Check your version with:
ssh -VIf you are running legacy systems (e.g., CentOS 6), they may not support the ed25519 algorithm for certificates. In those cases, you may be forced to use rsa-sha2-512, though we strongly advise upgrading the underlying OS.
Step-by-Step SSH Certificate Authority Setup
The first step is generating the CA root signing keys. We use Ed25519 for its performance and security characteristics. I recommend using a strong passphrase for the CA key, even if you plan to automate the signing process later.
Generating the CA Root Signing Keys
Run the following on your designated CA host:
# Create a directory for CA keyssudo mkdir -p /etc/ssh/ca sudo chmod 700 /etc/ssh/ca
Generate the CA key pair
-t: algorithm, -f: filename, -C: comment, -N: passphrase
sudo ssh-keygen -t ed25519 -f /etc/ssh/ca/ssh_ca -C "WarnHack_Production_CA"
You will now have ssh_ca (private key) and ssh_ca.pub (public key). The private key must be protected with the highest level of scrutiny. If this key is compromised, an attacker can sign certificates for any user on any server in your fleet.
Configuring the SSH Server to Trust the CA
Every target server in your infrastructure must be told to trust certificates signed by this CA. We do this by distributing the ssh_ca.pub file to every server and modifying sshd_config.
# On the target serverCopy the CA public key to a standard location
sudo cp /path/to/ssh_ca.pub /etc/ssh/trusted-user-ca-keys.pem sudo chmod 644 /etc/ssh/trusted-user-ca-keys.pem
Next, edit /etc/ssh/sshd_config to include the following directives:
# /etc/ssh/sshd_config
TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem
Optional: Disable standard public key auth to force CA usage
AuthorizedKeysFile /dev/null
PubkeyAuthentication yes
Restart the SSH service to apply changes:
sudo systemctl restart sshHow to Setup SSH Certificate Authentication for Users
The user workflow changes slightly. Instead of just having a key pair, the user must now get their public key signed by the CA.
Generating User Key Pairs
The user generates a standard key pair on their local machine:
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -C "[email protected]"The user then sends their id_ed25519.pub to the CA administrator (or an automated portal).
Signing User Public Keys with the CA
The CA administrator signs the user's public key using the CA private key. This is where we define the constraints of the access.
# Signing command-s: CA private key
-I: Identity string (for logs)
-n: Principals (allowed usernames on target)
-V: Validity period (+8 hours)
-z: Serial number
ssh-keygen -s /etc/ssh/ca/ssh_ca \ -I "user_access_token_01" \ -n "ubuntu,ec2-user,admin" \ -V +8h \ -z 1 \ id_ed25519.pub
This generates id_ed25519-cert.pub. The user must place this file in the same directory as their private key (~/.ssh/).
Verifying the Signed Certificate
The user can inspect the contents of their certificate to verify the principals and expiration:
ssh-keygen -L -f ~/.ssh/id_ed25519-cert.pubThe output will show:
- Type: [email protected] user certificate
- Public key: ED25519-CERT SHA256:...
- Signing CA: ED25519 SHA256:... (matches your CA key)
- Key ID: "user_access_token_01"
- Valid: from 2023-10-27T10:00:00 to 2023-10-27T18:00:00
- Principals: ubuntu, ec2-user, admin
Advanced Configuration: Host Certificates
Most SSH implementations only focus on user-to-server authentication. However, server-to-user authentication is equally vital to prevent MITM attacks. Host certificates allow the server to prove its identity to the user.
Generating and Signing Host Certificates
On the target server, take the existing host public key (usually in /etc/ssh/ssh_host_ed25519_key.pub) and sign it with the CA. Note the -h flag, which designates this as a host certificate.
ssh-keygen -s /etc/ssh/ca/ssh_ca \
-h \ -I "db-prod-mumbai-01" \ -n "db-prod-mumbai.internal,10.0.1.50" \ -V +52w \ /etc/ssh/ssh_host_ed25519_key.pub
Move the resulting ssh_host_ed25519_key-cert.pub back to the server's /etc/ssh/ directory and add this line to /etc/ssh/sshd_config:
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pubUpdating known_hosts with CA Information
On the client side (the engineer's machine), we can now replace the cluttered known_hosts file with a single line that trusts any host signed by our CA. Add this to ~/.ssh/known_hosts:
@cert-authority *.internal ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... (CA Public Key)Now, when connecting to any *.internal host, the client will verify the host's certificate. No more TOFU prompts.
Managing the SSH CA Lifecycle
A CA is only as good as its revocation mechanism. If a developer's laptop is stolen, you must be able to invalidate their certificate even if it hasn't expired yet.
Implementing Key Revocation Lists (KRL)
OpenSSH uses a binary KRL format. To revoke a key, you create a KRL file and distribute it to all servers.
# Revoke a specific certificate by serial numberssh-keygen -k -f /etc/ssh/revoked_keys -z 1
Or revoke by the public key itself
ssh-keygen -k -f /etc/ssh/revoked_keys id_ed25519.pub
In sshd_config, point the server to this list:
RevokedKeys /etc/ssh/revoked_keysAutomating Certificate Issuance with HashiCorp Vault
Manual signing is not scalable. We recommend using HashiCorp Vault's SSH Secrets Engine. Vault acts as the CA, providing an API for users to exchange their OIDC tokens (from Google, Okta, etc.) for a signed SSH certificate.
# Example Vault CLI command for a user
vault write -field=signed_key ssh-client-signer/sign/developer-role \ public_key=@$HOME/.ssh/id_ed25519.pub > $HOME/.ssh/id_ed25519-cert.pub
This workflow integrates with the ssh-agent, allowing for a seamless "login once a day" experience that satisfies both developer productivity and security compliance.
Security Best Practices for SSH CAs
The CA private key is the crown jewel of your infrastructure, a critical component in preventing broken access control as defined by the OWASP Top 10. We have observed that many teams fail because they treat the CA key like just another SSH key.
Protecting the CA Private Key with HSMs
If you are using AWS, utilize CloudHSM or KMS to store the CA key. OpenSSH supports PKCS#11, which allows ssh-keygen to interact with hardware tokens. This ensures the private key never leaves the hardware.
# Signing using a PKCS#11 provider (example)
ssh-keygen -s /usr/lib/pkcs11/libsofthsm2.so -I "identity" id_ed25519.pub
Defining Strict Certificate Identity Constraints
Use the "critical options" and "extensions" features of SSH certificates. You can restrict a certificate so it can only be used from a specific source IP address or force the execution of a specific command.
# Sign a key that is only valid from a specific CIDR
ssh-keygen -s /etc/ssh/ca/ssh_ca -I "restricted_user" -O source-address=10.50.0.0/16 id_ed25519.pub
Monitoring and Auditing SSH Access
CERT-In's 2022 directives require organizations to maintain logs of ICT systems for 180 days. When using certificates, the server logs contain the Key ID (-I flag) and the CA signature details. This provides a clear audit trail of exactly which identity was used, which is essential for SSH session logging and SIEM integration.
Addressing CVE-2023-38408
This vulnerability involved remote code execution in ssh-agent when forwarding is enabled. While SSH CAs don't fix the underlying agent flaw, they mitigate the risk. By using short-lived certificates (e.g., 1 hour), you reduce the time an attacker has to hijack an agent. Furthermore, you can use the -O no-agent-forwarding extension when signing certificates for high-risk users to disable the feature entirely at the cryptographic level.
Next Command: Auditing Your Fleet
Before implementing a CA, you need to know the current state of your sprawl. Run this snippet across your fleet to identify how many unique static keys are currently authorized:
cat ~/.ssh/authorized_keys | awk '{print $NF}' | sort | uniq -c | sort -nrThis initial data point usually provides the necessary justification for management to invest in a centralized SSH Certificate Authority. Once the CA is live, your next step is to integrate it with an OIDC provider to automate the signing process, ensuring that SSH access is tied directly to your primary identity provider.