The authorized_keys Sprawl Problem: Detecting Stale Keys and Identity Drift Across Cloud Fleets

The authorized_keys Sprawl Problem: Detecting Stale Keys and Identity Drift Across Cloud Fleets

Every long-lived instance in our fleet eventually turns into a cryptographic sediment layer. You pull an Ubuntu 20.04 node that has survived two infra re-architectures, open up /root/.ssh/authorized_keys, and find fourteen keys. Four have comments like deploy-key-2021, six have personal email addresses from contractors who left eighteen months ago, and two have no comments at all—just raw base64 ending in ==.

Nobody wants to delete them because nobody knows what breaks if they do. That is how identity drift becomes a permanent architectural fixture.

Running a systematic ssh authorized_keys audit across a few hundred instances turns up messy truths quickly. If you rely on team discipline to clean up keys when people leave, you do not have key management. You have wishful thinking.

---

Phase 1: Discovery and the Permission Trap

Before analyzing key quality or cryptographic strength, you have to actually find the files. Most engineers assume keys only live in ~/.ssh/authorized_keys. In practice, service users, custom daemon paths, and cloud provisioning scripts put them in non-standard locations.

Start with local and service accounts by scanning home directories and system paths:

# Find all standard authorized_keys files across the filesystem
find /home /root -path '*/.ssh/authorized_keys' 2>/dev/null

That misses service users whose home directories live in /var/lib, /var/run, or custom paths specified in /etc/passwd. A more thorough check parses /etc/passwd directly to discover home directories for users that actually possess a shell or run daemons:

awk -F: '($7 !~ /nologin|false/) { print $1, $6 }' /etc/passwd | while read -r user homedir; do
if [ -f "$homedir/.ssh/authorized_keys" ]; then         echo "Found keyfile for $user: $homedir/.ssh/authorized_keys"     fi done

Next, check what sshd actually respects. If someone modified /etc/ssh/sshd_config or dropped a snippet into /etc/ssh/sshd_config.d/, keys might live in centralized directories:

sshd -T | grep -i 'authorizedkeysfile'

The default output is usually:

authorizedkeysfile .ssh/authorized_keys .ssh/authorized_keys2

If you see a system path like /etc/ssh/authorized_keys/%u, standard home directory scans will show you nothing while logins proceed uninhibited.

The StrictModes Gotcha

I ran into an issue where an audit script flagged valid keys that were completely unreachable in practice. Under standard OpenSSH security defaults, StrictModes is set to yes. If a user runs chmod 775 ~ or leaves group-write permissions on ~/.ssh, OpenSSH silently rejects all public keys in that directory without printing anything to the client. The connection simply falls back to the next available auth method (usually password) or exits with code 255.

# Verify permissions on .ssh and authorized_keys
find /home /root -name ".ssh" -type d -exec ls -ld {} \;
# Expected: drwx------ (0700)
find /home /root -path '*/.ssh/authorized_keys' -type f -exec ls -l {} \;
# Expected: -rw------- (0600)

If group-write permissions exist on the parent directory or the file itself, that key is technically dead. Do not waste time chasing owners of keys that OpenSSH refuses to load in the first place.

---

The Failed Approach: Auditing Last Used via atime

We wanted an easy, zero-dependency way to identify dead keys: look at the access timestamps on the key files to see which ones hadn't been touched in six months.

$ stat -c '%x %n' /home/*/.ssh/authorized_keys
2024-04-12 09:14:22.118934502 +0000 /home/deploy/.ssh/authorized_keys 2024-04-12 09:14:22.118934502 +0000 /home/ubuntu/.ssh/authorized_keys

This fails completely. It does not work for two distinct reasons.

First, almost every cloud machine is mounted with noatime or relatime in /etc/fstab. File read operations do not reliably update inode access times unless the file was modified since the last access or a full 24-hour window has passed.

Second, and more fundamentally: sshd reads authorized_keys sequentially from line 1 to EOF until it finds a cryptographic match. If user "Alice" is on line 1, and user "Bob" is on line 40, every time Alice connects, sshd stops reading at line 1. Bob's key line never gets evaluated. Conversely, if an unauthorized attacker sprays attempts against an account, sshd reads the entire file trying to match the candidate key. A single login attempt updates the atime for the entire file.

You cannot get per-key utilization granularity from filesystem metadata. The file is a monolithic blob to the kernel.

---

Phase 2: Technical Inspection of Keys and Options

To inspect what is actually inside those files, avoid naive text processing. A common mistake is writing awk '{print $2}' authorized_keys. That assumes every line begins with the key type (like ssh-ed25519 or ssh-rsa).

Keys with restrictive options violate this immediately. A line might begin with:

from="10.0.0.0/8",no-port-forwarding,no-pty ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... bot@ci

A simple awk splits on the options block and grabs the base64 payload as field 2, breaking downstream fingerprinting.

Safe Fingerprinting with ssh-keygen

OpenSSH provides its own parser. Use ssh-keygen -lf directly against the target file. It understands comments, options, and multiple keys on consecutive lines:

find /home /root -path '*/.ssh/authorized_keys' -exec ssh-keygen -lf {} \;

The output provides bit length, fingerprint hash, comment, and key algorithm:

3072 SHA256:7mN40R8J... deploy-ci (RSA)
256 SHA256:sK91Pq0L... dev-box-alice (ED25519) 1024 SHA256:x0P82kLs... old-bastion (DSA)

Flagging criteria should be straightforward:

• DSA: Instant revokation. DSA is hard-disabled in modern OpenSSH versions anyway; if it is in an authorized_keys file, it is pure dead weight.
• RSA < 2048 bits: Cryptographically broken. Flag for immediate removal.
• RSA 2048 bits: Mark for scheduled deprecation. We aim for Ed25519 or RSA 4096 across all systems.
• Missing Comments: Unidentifiable keys should be quarantined. If nobody knows who owns the fingerprint, it fails any identity review.

---

Correlating Keys with Actual SSH Sessions

Since filesystem timestamps cannot tell us which specific key within a file was used to authenticate, we have to look at the system journals.

By default, OpenSSH logs at LogLevel INFO. When a public key authentication succeeds, it writes a line like this to the auth log:

Accepted publickey for ubuntu from 192.168.1.50 port 41234 ssh2: RSA SHA256:7mN40R8J...

You can pull these historical sessions from journalctl:

journalctl -u sshd -u ssh --no-pager -g 'Accepted publickey' -o json | jq -r '._SOURCE_REALTIME_TIMESTAMP, .MESSAGE'

This gives you the timestamp and the fingerprint. But there is a blind spot: LogLevel INFO logs the fingerprint, but it does not log the full public key, the key comment, or the certificate serial number if you are using signed certificates.

If you have an authorized_keys file containing two keys that resolve to the same comment or need exact public-key-to-user mappings for external audits, INFO leaves gaps. Switching to LogLevel VERBOSE in your SSH daemon configuration fixes this by forcing sshd to print the full key fingerprint alongside the matching certificate ID or detailed algorithm metadata on every login.

---

Regulatory Context: The CERT-In Logging Mandate

This audit gap is not just an internal security cleanliness issue. Under the CERT-In Cyber Security Directions (April 2022), organizations are required to maintain ICT system logs across their infrastructure for a rolling period of 180 days within Indian jurisdiction.

If you operate infrastructure handling traffic in that region, having static authorized_keys files with arbitrary names like macbook.local or relying on shared SSH keys completely breaks this mandate. A log entry showing an accepted public key with fingerprint SHA256:abc... satisfies nothing during an incident review if you cannot deterministically map that fingerprint to a verified human identity, their employee ID, or corporate credentials.

---

The Cloud-Init Gotcha During Remediation

When you start pruning unauthorized keys, you run into cloud-init.

I cleaned out an old base image on AWS, deleted three unmapped keys from /home/ubuntu/.ssh/authorized_keys, verified logins, and called it a day. After an instance restart triggered by an underlying host migration, the deleted vendor keys were right back in the file.

Cloud-init reads metadata services (like AWS IMDS or OpenStack metadata) on every boot unless told otherwise. If your cloud-init config includes modules like ssh-authkey-fingerprints or keys-to-console, it re-reads instance launch metadata and re-injects the SSH public key specified when the VM was created three years ago.

To stop cloud-init from restoring deleted static keys, update /etc/cloud/cloud.cfg or drop an override into /etc/cloud/cloud.cfg.d/:

# /etc/cloud/cloud.cfg.d/99-disable-ssh-regeneration.conf
ssh_deletekeys: false cloud_final_modules:   - [ssh-authkey-fingerprints, false]

If you do not disable this, your automated pruning job will fight cloud-init on every instance reboot, and cloud-init will win.

---

Centralizing Key Verification via sshd_config

Maintaining flat text files on disk across hundreds of instances will always lead to drift. A better intermediate step—short of a full overhaul—is delegating key checks to a local execution wrapper using AuthorizedKeysCommand.

This allows sshd to query an authoritative local binary or secure cache instead of reading arbitrary files edited by whoever had sudo access last month.

# /etc/ssh/sshd_config.d/99-audit-and-centralize.conf
LogLevel VERBOSE AuthorizedKeysFile none AuthorizedKeysCommand /usr/local/bin/fetch-authorized-keys %u %k %t AuthorizedKeysCommandUser nobody StrictModes yes

Here, AuthorizedKeysFile none turns off ~/.ssh/authorized_keys entirely. When a user connects, sshd runs /usr/local/bin/fetch-authorized-keys with the target username (%u), the base64 key sent by the client (%k), and the key type (%t).

The script can check an internal read-only key store, verify against an inventory database, log the exact attempt to an off-box collector, and return the authorized public key to stdout. If the exit code is non-zero or stdout is empty, authentication fails.

No keys stored in user home directories. No stale keys surviving off-boarding.

---

Next Experiment

We are testing short-lived OpenSSH certificates (valid for 8 to 12 hours) signed by an internal Vault CA, paired with TrustedUserCAKeys in sshd_config. That would eliminate static public keys from the nodes completely.

The open issue is automation for cron jobs and headless data pipelines that currently rely on static deploy keys. I need to figure out whether renewing certificates via instance profile roles introduces more complexity than just locking down raw keys with strict from="ip" directives and custom audit hooks.