The Failure of Static Key Management
We recently audited a mid-sized Indian FinTech firm where the primary method of server access was a shared production.pem file. This file was found in the downloads folder of three different developers and had been shared over a Slack channel six months prior. This is a textbook example of "SSH Key Sprawl." In the Indian IT landscape, especially among SMEs and local MSPs, static keys are often treated as permanent credentials. Implementing SSH security hardening best practices is essential to mitigate these risks, as keys are rarely rotated, and when an employee leaves, the process of removing their key from hundreds of authorized_keys files is frequently overlooked.
The risk is not theoretical. With the 2022 CERT-In Cyber Security Directions mandating strict access logs and incident reporting within six hours, the administrative overhead of managing static keys becomes a compliance liability. If a developer's workstation is compromised, a static key provides indefinite lateral movement capabilities. This highlights the need for a shared SSH key alternative that provides visibility and control. We observed that moving to an SSH Certificate Authority (CA) eliminates this persistence by requiring users to obtain short-lived, signed certificates for every session.
Introduction to SSH Certificate Authority (CA)
What is an SSH Key Certificate Authority?
An SSH Certificate Authority is a trusted entity—typically a dedicated server or a hardware security module (HSM)—that signs public keys to create certificates. Unlike standard SSH keys, which rely on the "Trust On First Use" (TOFU) model or manual synchronization of authorized_keys, an SSH CA allows the server to verify a user's identity based on a cryptographic signature.
When we implement an SSH CA, the target server no longer needs to know about individual users. It only needs to trust the CA's public key. If a user presents a certificate signed by that CA, and the certificate is currently valid according to its internal timestamp, the server grants access. This shift from "who are you" to "who says you are who you say you are" is the foundation of identity-based access.
SSH Keys vs. SSH Certificates: Why Your Infrastructure Needs a CA
Standard SSH keys are simple pairs: a private key and a public key. The server stores the public key in ~/.ssh/authorized_keys. This model fails at scale for several reasons:
- No Expiration: Static keys are valid until manually deleted.
- No Metadata: You cannot easily restrict a key to specific commands or source IPs within the key itself without complex
authorized_keyssyntax. - Trust Issues: Users must manually verify host fingerprints, leading to "fingerprint fatigue" where users blindly accept new keys, making Man-in-the-Middle (MitM) attacks trivial.
- Revocation Complexity: Revoking a single compromised key requires updating every single server that key had access to.
SSH certificates solve these issues by embedding identity, permissions, and expiration dates directly into the signed object. We found that by using certificates, we could reduce the size of our authorized_keys files to zero, significantly lowering the attack surface for lateral movement, a key concern in the OWASP Top 10.
The Benefits of Using an OpenSSH Certificate Authority
The primary benefit we observed during deployment is the centralization of policy. Instead of decentralized files across 500 instances in an AWS Mumbai (ap-south-1) VPC, we manage access at the CA level.
- Automated Expiration: We typically set a Time-To-Live (TTL) of 4 to 8 hours. This aligns with the standard Indian workday and ensures that even if a certificate is stolen, it is useless by the next morning.
- Role-Based Access Control (RBAC): The CA can embed "principals" (usernames) into the certificate. A developer might get a certificate valid for
ubuntuandec2-user, while a database admin gets one forpostgres. - Auditability: Every certificate issued by the CA is logged. This provides a clear trail of who requested access and when, satisfying DPDP Act 2023 requirements for data access monitoring through integrated SIEM and log monitoring.
- Elimination of TOFU: By signing host keys as well, we eliminate the "Unknown Host" warning for end-users, ensuring they only connect to verified infrastructure.
How an SSH Certificate Server Works
The Architecture of an SSH Key Authority
The architecture consists of three main components: the CA, the Client, and the Host. The CA holds the sensitive root private key, which must be protected with extreme rigor. In our testing, we use a dedicated, hardened Linux instance or a specialized tool like HashiCorp Vault to act as the CA.
The workflow follows these steps:
- The Client generates a standard SSH key pair (e.g., Ed25519).
- The Client sends their public key to the CA (often via a web portal or CLI tool after SSO authentication).
- The CA verifies the user's identity (via OIDC, LDAP, or SAML).
- The CA signs the public key and returns a
-cert.pubfile to the Client. - The Client connects to the Host using this certificate.
- The Host verifies the CA's signature and allows entry.
The Role of the SSH Certificate Authority CA Root Key
The CA root key is the "God key" of your infrastructure. If this key is compromised, an attacker can sign their own certificates and gain access to any server trusting that CA. We recommend never storing this key on a standard filesystem in plaintext.
For high-security environments, we use a YubiKey or a cloud-native KMS to perform the signing. If you are starting with a software-based CA, ensure the private key is encrypted with a strong passphrase and stored on a volume with restricted access.
The Signing Process: How an SSH Certificate Server Issues Credentials
To generate the CA root key, we use the ssh-keygen command. We prefer Ed25519 for its performance and security characteristics over RSA.
# Generate the CA root key pair
ssh-keygen -t ed25519 -f ssh_ca -C "Internal-Infrastructure-CA" -N 'YOUR_STRONG_PASSPHRASE'
Once the CA key exists, we can sign a user's public key. In this example, we sign a developer's key, giving them access to the ubuntu and ec2-user principals for exactly 4 hours.
# Sign the user public key
ssh-keygen -s ssh_ca -I "dev-access-id-123" -n ubuntu,ec2-user -V +4h -z 1 id_ed25519.pub
The flags used here are critical:
-s: Specifies the CA private key.-I: An identity string used for logging and identification.-n: A comma-separated list of principals (usernames) the certificate is valid for.-V: The validity period (e.g., +4h for four hours).-z: A serial number to track the certificate.
Technical Configuration: Implementing ssh @cert-authority
Understanding the ssh @cert-authority Directive in known_hosts
To solve the "Host Key Verification" problem, we also sign the host keys of our servers. This prevents users from seeing the "Host key verification failed" error and ensures they are not connecting to a rogue server. On the client side, we add the CA's public host key to the ~/.ssh/known_hosts file using the @cert-authority prefix.
# Add this to ~/.ssh/known_hosts
@cert-authority *.internal.example.in ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... (CA_PUBLIC_KEY)
This directive tells the SSH client that any host with a name matching *.internal.example.in that presents a certificate signed by this specific CA should be trusted automatically.
Configuring the SSH Daemon to Trust the CA
The server-side configuration is remarkably simple. We only need to tell sshd which CA public key to trust for user authentication. First, copy the ssh_ca.pub (the public part of your CA key) to /etc/ssh/ssh_ca.pub on all target servers.
Then, modify /etc/ssh/sshd_config:
# /etc/ssh/sshd_config implementation
TrustedUserCAKeys /etc/ssh/ssh_ca.pub
Optional: Define a revocation list for emergency use
RevocationList /etc/ssh/revoked_keys
Restrict to specific principals and force short-lived sessions
Match User limited-user AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u MaxSessions 2 ClientAliveInterval 300 ClientAliveCountMax 0
After updating the config, restart the service: systemctl restart ssh. The server will now accept any valid certificate signed by the ssh_ca.pub.
Automating Host Verification with SSH Certificate Authorities
To fully automate this, we sign the host's own public key (usually /etc/ssh/ssh_host_ed25519_key.pub) using the CA. The resulting -cert.pub file is then referenced in the server's sshd_config.
# On the CA server, sign the host key
ssh-keygen -s ssh_ca -I "prod-web-01" -h -n prod-web-01.internal.example.in -V +52w ssh_host_ed25519_key.pub
On the host server, add this to sshd_config:
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub
This creates a bidirectional trust loop where both the user and the server are cryptographically verified before the session begins.
SSH Certificate Authorities for Version Control Systems
Setting up SSH Certificate Authorities for GitHub Enterprise
GitHub Enterprise and GitHub.com (for organizations) support SSH CAs to streamline repository access. Instead of developers adding their individual public keys to their GitHub profiles—which are permanent and difficult to audit—the organization adds its CA public key to the GitHub settings.
When a developer wants to git clone or push, they use a short-lived certificate signed by the company CA. GitHub validates the signature and the username (principal) against the developer's GitHub account.
Managing SSH Certificate Authority on GitHub for Secure Repository Access
To implement this on GitHub:
- Navigate to Organization Settings > Authentication Security > SSH Certificate Authorities.
- Click New CA and paste the contents of
ssh_ca.pub. - Require members to use SSH certificates by checking the "Require SSH Certificates" box.
We found that this significantly reduces the risk of "shadow IT," where developers use personal SSH keys to access corporate codebases. If a developer leaves the company, their access is revoked the moment their last certificate expires (usually within hours), regardless of whether their GitHub account is still active.
Integrating a GitLab SSH Certificate Authority
GitLab supports a similar flow, particularly when using the GitLab Shell. By configuring the AuthorizedKeysCommand in the GitLab server's sshd_config, you can point to a script that validates certificates.
However, many GitLab users prefer the authorized_keys proxy approach. We recommend using a tool like step-ca or Vault to bridge GitLab's internal identity provider with the SSH CA signing process. This ensures that a user must have an active GitLab session to receive a signed SSH certificate.
Best Practices for Managing an SSH Key Certificate Authority
Securing the SSH CA Private Key
The security of your entire infrastructure rests on the CA private key. We have seen organizations lose their entire fleet to a single compromised Jenkins server that held an unencrypted CA key.
Protection Strategies:
- Hardware Security Modules (HSM): Use AWS CloudHSM or an on-premise HSM to store the key. The key never leaves the hardware; the hardware simply signs the payload.
- Air-Gapping: For host-key signing (which happens infrequently), keep the CA key on an air-gapped machine.
- Multi-Signature: Require two administrators to authorize the signing of long-lived certificates.
Setting Short-Lived Expiration for SSH Certificates
We recommend a "Just-In-Time" (JIT) approach to certificate issuance. A certificate should never last longer than a single work shift.
# Check the details of a generated certificate
ssh-keygen -L -f id_ed25519-cert.pub
The output will show the validity window:
# Expected output snippet
Type: [email protected] user certificate Public key: ED25519-CERT SHA256:... Signing CA: ED25519 SHA256:... (ID: WarnHack-Internal-CA) Key ID: "user_access_token" Serial: 1 Valid: from 2023-10-27T10:00:00 to 2023-10-27T14:00:00 Principals: ubuntu ec2-user Critical Options: (none) Extensions: permit-pty permit-port-forwarding
If a certificate is issued for 4 hours, and an attacker steals it at hour 3, they only have 60 minutes of access. This is a massive improvement over static keys which might be valid for years.
Using SSH Certificate Authority GitHub Integration for Team Scaling
As Indian startups scale from 10 to 500 engineers, manual key management becomes impossible. By integrating the CA with an Identity Provider (IdP) like Okta, Azure AD, or Google Workspace, you can automate the entire lifecycle.
A developer runs a command like ssh-login, which opens a browser for SSO. Upon successful login, the IdP sends a signal to the CA to sign the developer's local public key for the next 8 hours. The developer never sees the CA key, and the admin never sees the developer's private key.
Conclusion: Transitioning to a Certificate-Based SSH Workflow
Summary of SSH Certificate Authority Benefits
Transitioning to an OpenSSH CA workflow effectively mitigates the risks highlighted by recent vulnerabilities like CVE-2023-38408 (ssh-agent RCE) and CVE-2024-6387 (regreSSHion) as cataloged in the NIST NVD. By enforcing short-lived sessions, we limit the window of opportunity for an attacker to exploit agent forwarding or signal handler race conditions.
In the context of Indian compliance, using an SSH CA provides:
- Granular Logging: Every
ssh-keygen -sevent is a log of intent. - Reduced Remediation Time: No need to "scrub"
authorized_keysafter a breach; just stop issuing certificates. - Cost Savings: Reduces the engineering hours spent on manual key rotations and access requests.
Next Steps for Deploying an OpenSSH Certificate Authority
We recommend a phased rollout. Start by signing host keys to eliminate TOFU warnings for your team. This builds confidence in the CA infrastructure without changing how users log in. Next, move a single non-critical environment (like a staging VPC) to TrustedUserCAKeys and disable authorized_keys entirely.
To verify your current setup and check for active identities in your agent, use:
$ ssh-add -l -e
This command will show if you are currently using certificates or standard keys. If you see (SHA256:...) without a certificate reference, you are still operating on legacy static keys and are vulnerable to key sprawl.
Technical Insight: The Power of AuthorizedPrincipalsFile
One often overlooked feature is the AuthorizedPrincipalsFile. It allows you to map a single system user (like deploy) to multiple logical roles.
# /etc/ssh/auth_principals/deploy
dev-lead senior-sre ci-cd-service
By placing this in your sshd_config, anyone with a certificate containing the principal dev-lead can log in as deploy, but a junior developer with only the dev-junior principal cannot. This allows for extremely fine-grained RBAC without ever touching the /etc/passwd file on the target host.