
The Orphaned Key Trap: Diagnosing Stale authorized_keys Sprawl and Incomplete Offboarding Across Cloud Fleets
secure-access
This series is part of WarnHack's Topical Authority research.
13 Articles
You pull an inventory of three hundred cloud instances, run an ad-hoc command to inspect /root/.ssh/authorized_keys, and find keys dating back four infrastructure iterations. An engineer departed six months ago. Their corporate email was suspended, single sign-on access cut, and cloud IAM roles deleted within an hour of their exit. Yet their personal Ed25519 public key is still sitting in local accounts across half your staging fleet and three production utility nodes.
Static public keys on disk do not expire. They do not query an identity provider (IdP) on login, and they do not care if the person holding the matching private key still works for you. Until organizations fully replace shared SSH keys with ephemeral identity tokens, running a systematic SSH authorized_keys sprawl audit across a sprawling, multi-account fleet is the only way to find out who actually has shell access right now.
The False Comfort of Key Comments (And Why Our First Script Failed)
The fastest way to fail an access audit is trusting the text at the end of an SSH public key string.
When we first tried to automate offboarding verification across several persistent compute nodes, the initial attempt looked like this standard one-liner:
# Naive check: extracting the trailing comment field
awk '{print $NF}' ~/.ssh/authorized_keys
The script compared those trailing strings against active employee email addresses pulled from our directory API. If the string did not match an active user, the script flagged the key for removal.
This approach blew up immediately. In the OpenSSH specification, the trailing comment is completely arbitrary metadata. It gets generated by default based on the local hostname of the developer's laptop (e.g., [email protected]) or can be suppressed entirely during generation:
$ ssh-keygen -t ed25519 -C "" -f ./id_deploy
# Generates a key with no trailing comment at all
When the comment is missing or formatted as a generic phrase like deploy-key-prod or ansible-worker, the script returned blank fields or false negatives. Worse, an offboarded contractor had provisioned a key where the comment was explicitly set to match our lead architect's email address. The key passed the naive audit cleanly even though the private key lived on a personal machine outside our control.
Grepping comments is security theater. The only reliable identifier for an SSH public key is its cryptographic hash.
Auditing Keys Across Distributed Filesystems
A proper audit must inspect every local account, uncover where authorized keys live, and generate raw fingerprints. Public keys do not hide just in /home/. They sit in /root, /var/lib, service daemon homes, and temporary mount points.
# Find all authorized_keys files and output their cryptographic fingerprints
find /home /root /var/lib -path '*/.ssh/authorized_keys' -type f -exec ssh-keygen -lf {} + 2>/dev/null
On a test target running Debian, the raw output looks like this:
256 SHA256:xK7vB...8dE root@infra-01 (ED25519)
2048 SHA256:m9LmP...1aQ no comment (RSA) 256 SHA256:9qRtZ...7uW dev-test (ED25519)
This output gives you the exact SHA256 fingerprint, the bit length, and the key cipher type. Weak 1024-bit RSA keys or deprecated DSA keys immediately jump out here alongside modern Ed25519 entries.
Cloud Fleet Blind Spots
A few structural quirks across cloud providers consistently subvert key hygiene:
- Cloud-init persistence: Cloud-init appends bootstrap keys once at initial instance boot and never handles reconciliation. If you rotate keys inside your infrastructure-as-code user-data scripts, running instances do not prune the original bootstrap keys. They stay in
~/.ssh/authorized_keysforever unless explicitly wiped. - Default distro accounts: Shared distro accounts—such as
ubuntu,ec2-user, orcentos—often retain launch-time public keys configured with fullNOPASSWD: ALLsudo access. If an engineer or external vendor places their key into/home/ubuntu/.ssh/authorized_keysto debug an outage, removing their named user account (e.g.,/home/alex/) later does nothing to stop them from entering through the distro user. - Permission drift: OpenSSH will reject keys if the file or parent directory permissions are too loose (e.g., group-writable), but administrators often "fix" this by changing permissions indiscriminately, sometimes masking rogue keys in directories that should never have had public access enabled in the first place.
Cross-Referencing Usage Logs with Local Storage
Finding a key on disk tells you it is authorized; it does not tell you if anyone is actually using it. To understand whether an old key is abandoned or actively driving an undocumented cron job, check the authentication history in the system logs:
# Pull the last 30 days of public key logins from journald
journalctl -u ssh -u sshd --since '30 days ago' -g 'Accepted publickey' -o json | jq -r '._SOURCE_REALTIME_TIMESTAMP, .MESSAGE'
You will get timestamped records showing the user account, incoming IP address, and the public key fingerprint:
Accepted publickey for ec2-user from 198.51.100.24 port 54210 ssh2: RSA SHA256:m9LmP...1aQ
Compare the SHA256 output from ssh-keygen -lf against the fingerprints appearing in journalctl. If an Ed25519 key exists in /root/.ssh/authorized_keys but has zero authentication records over the past month, it is an prime candidate for deprecation.
The Regulatory Log Trail
This disconnect is not just an operational mess; it creates immediate compliance failures. Under the CERT-In Directions (April 2022), organizations must maintain validated ICT system logs—specifically system access and authentication events—for a rolling duration of 180 days within Indian jurisdiction. Static orphaned keys on distributed cloud infrastructure subvert centralized SSO/IdP authentication auditing, undermining efforts to centralize access logs for threat detection and leading to untracked access that directly breaches CERT-In log traceability requirements.
When an engineer accesses an instance via an unmanaged static key, the authentication log records only a raw hash and a generic Unix user. It fails to map the access to a verified corporate identity, creating an unresolvable audit gap.
Safe Remediation: Removing Rogue Keys Without Outages
Deleting a line out of authorized_keys seems simple, but doing it blindly will knock out automated backups, broken deployments, or third-party monitoring agents running under unmonitored keys.
The ControlMaster Problem
Removing a public key from authorized_keys does not sever existing connections. OpenSSH multiplexing (ControlMaster) keeps connections alive through local sockets:
# Check for active SSH multiplexing master processes running under the target user
ps aux | grep '[s]sh.*ControlMaster'
If an offboarded user has an active session or a multiplexed background socket open, simply deleting their key from disk leaves their tunnel active. You have to locate and terminate the specific sshd child process associated with that user session to actually disconnect them.
Auditing Local Key Modification in Real Time
While you identify and phase out stale keys, prevent silent re-infection by placing an audit rule on critical key files. The Linux Audit framework (auditd) can monitor write operations to these files directly:
# Watch the root authorized_keys file for writes or attribute changes
auditctl -w /root/.ssh/authorized_keys -p wa -k ssh_key_tamper
If a legacy automation script or a forgotten configuration management run attempts to re-inject a removed key, the event is immediately captured in /var/log/audit/audit.log with the parent PID and executable name that executed the write.
Eliminating the Flat File: Centralized Fetching
Maintaining flat text files across hundreds of persistent systems is structurally broken. Modern infrastructure should decouple authentication from local storage entirely.
One direct step away from flat-file sprawl is offloading key lookup to an external authority using AuthorizedKeysCommand in your daemon configuration. Instead of searching local disk, OpenSSH queries a centralized binary or API on each connection attempt.
# /etc/ssh/sshd_config
# Eliminate static local key sprawl by decoupling authentication from local disk files AuthorizedKeysFile none AuthorizedKeysCommand /usr/local/bin/fetch_idp_keys %u %k %t AuthorizedKeysCommandUser nobody PubkeyAuthentication yes StrictModes yes
In this pattern, the local authorized_keys file is effectively disabled (AuthorizedKeysFile none). When a connection arrives, sshd passes the username (%u), the incoming raw key (%k), and key type (%t) to a hardened local script running as nobody. That script queries your internal directory service or metadata store.
If an employee leaves the company, their status changes in the central directory, and their key fails authentication fleet-wide within seconds—no remote disk cleanup required.
Short-Lived SSH Certificates
The definitive fix for key sprawl is abandoning static public keys altogether in favor of an OpenSSH Certificate Authority (CA). Instead of distributing public keys to servers, you distribute a single CA public key to the fleet via TrustedUserCAKeys /etc/ssh/ca.pub.
Engineers authenticate to your identity provider via their browser or CLI to obtain an SSH certificate signed by your internal CA. The certificate embeds their username, allowed principals, and an expiration time of four to eight hours. Servers simply validate the signature of the CA. Once the certificate expires, access ceases automatically. There are no static keys stored on the target machines, which renders local key sprawl physically impossible.
Current Open Question
Moving to AuthorizedKeysCommand solves the orphan problem, but it introduces an availability tradeoff during network partitions. If a transit gateway drops or an internal identity endpoint rate-limits lookups during a major fleet recovery event, sshd can hang or reject connections across the board. We are currently testing local read-through caching using an SSSD-backed cache layer versus pushing an SSH CA configuration out through baseline immutable images.
Explore Topics
Replace Shared SSH Keys with Browser Access
WarnHack Terminal: browser SSH, role-based access, CloudRelay session & command audit — Teleport/StrongDM alternative for lean teams.
Stay Ahead of Threats
Get the latest cybersecurity insights, tutorials, and threat intelligence delivered to your inbox.
Discussion
Enjoyed this article?