Detecting Persistent SSH Backdoors: Auditing authorized_keys and Binary Integrity in Linux Environments

The Friday Afternoon Audit

It was 4:30 PM on a Friday when a routine audit of our legacy jumpserver cluster turned into a forensic deep-dive. I was looking for a specific stale key, but what I found was a mess of "prod-keys" shared across five hundred instances—a classic case of Jugaad persistence common in some of our older local data center deployments. When everyone shares a single key for convenience, attribution goes out the window, and backdoors hide in plain sight.

SSH backdoor detection isn't just about looking for a weird string in a text file. It’s about verifying the entire chain of trust from the binary on disk to the environment variables set during session initialization. I started with the obvious, but as usual, the obvious was a dead end.

---

Auditing SSH Configuration Files for Unauthorized Access

I initially tried a quick and dirty check across the home directories. I ran this, thinking it would be enough:

# The "I'm in a hurry" approach that failed me
grep -r "ssh-rsa" /home/*/.ssh/authorized_keys

I hit exit code 0; nothing looked out of the ordinary. But the logs were still showing successful logins from an IP in a range we don't own. The problem with relying solely on grep or cat of ~/.ssh/authorized_keys is that sophisticated persistence bypasses this file entirely. If an attacker has root, they can modify /etc/ssh/sshd_config to fetch keys from anywhere.

The AuthorizedKeysCommand Trap

I checked the config and found a sneaky directive that pointed to a script in /usr/local/bin. This is a common way to maintain access without leaving a footprint in the user's home directory.

# Snippet found in /etc/ssh/sshd_config
AuthorizedKeysCommand /usr/local/bin/get-keys.sh AuthorizedKeysCommandUser nobody
# Also check for non-standard key locations
AuthorizedKeysFile /etc/ssh/keys/%u .ssh/authorized_keys

The AuthorizedKeysCommand fetches keys from an external URL or a local script every time someone tries to log in. The local authorized_keys remains completely empty and clean, making traditional audits useless. We also have to watch for AuthorizedKeysFile being redirected to a global directory like /etc/ssh/keys/%u, where an admin might not think to look.

Mass Fingerprinting Keys

To get a real sense of what’s authorized across the board, I stopped grepping for strings and started fingerprinting. This command is much more reliable for identifying exactly what keys are present:

find /home /root -name 'authorized_keys' -exec ssh-keygen -lf {} \;

This gives you the bit length, the fingerprint, and the comment for every key. If you see a 4096-bit RSA key with no comment or a suspicious email address, that’s your first red flag. We also found some keys were set with chattr +i, making them immutable. Even root couldn't delete them until I ran chattr -i. It’s a simple trick, but it stops automated cleanup scripts in their tracks.

---

Detecting Backdoors via System Log Analysis

Logs are usually the first place to look, but they are also the easiest to flood or rotate out. I needed to see who had been getting in over the last 24 hours without scrolling through megabytes of noise.

journalctl -t sshd --grep='Accepted' --since '24h ago'

I was looking for "Accepted password" when we supposedly only allow keys, or "Accepted publickey" for users that shouldn't exist. In Indian MSP environments where turnover is high, we often see "Ghost Sessions"—logins for users who left the company months ago but whose keys were never rotated because of the "prod-key" sharing culture.

Spotting Unusual Patterns

Session duration is another giveaway. An automated backdoor script might log in, execute a command, and drop in three seconds, repeated every hour. I use lastlog to see when accounts were last accessed, but keep in mind that attackers often clear /var/log/wtmp or /var/run/utmp to hide their presence.

If last shows a user logged in but who or w doesn't show an active TTY, you might be looking at a process that has unlinked itself from the terminal to stay hidden.

---

Advanced Detection: Binary Integrity and Process Monitoring

If the config files look clean and the keys are rotated, but the box is still compromised, you have to look at the binaries. The sshd binary itself can be replaced or hooked.

Verifying the sshd Binary

On RHEL-based systems, I use rpm -V. On Debian/Ubuntu, it’s debsums. I ran this on a suspected box:

# For RHEL/CentOS/Alma
rpm -V openssh-server
# For Debian/Ubuntu
debsums -s /usr/sbin/sshd

If debsums returns nothing, the binary matches the package maintainer's hash. If it spits out a filename, the file has been altered. We saw this recently with CVE-2024-3094, where the XZ Utils backdoor targeted liblzma to compromise sshd authentication. It wasn't the sshd binary itself that was modified, but a library it linked against.

Identifying Malicious PAM and LD_PRELOAD

This is where it gets nasty. An attacker can drop a malicious library and use LD_PRELOAD to hook into sshd. This allows them to accept a "magic password" regardless of what is in authorized_keys.

I checked /etc/pam.d/sshd for any unusual modules. A common persistence technique is replacing pam_unix.so with a version that logs passwords or allows a specific backdoor password.

# Checking for active network sockets
ss -tulpn | grep sshd

I saw an extra listener on a high port that looked like SSH but wasn't managed by systemd. That’s usually a second instance of sshd running with a custom config file hidden in /dev/shm or /tmp.

---

Essential Tools for Automated Detection

We can't do this manually for every server. I've started leaning on rkhunter and Lynis for the initial sweep. They aren't perfect, but they catch the low-hanging fruit like known rootkit signatures and suspicious file permissions.

For real-time monitoring, we’ve been testing Wazuh. It has decent File Integrity Monitoring (FIM) that alerts us the second /etc/ssh/sshd_config is touched. It’s better than the old days of running a cron job to md5sum the config files, though I still trust a manual auditctl rule for high-value targets:

auditctl -w /etc/ssh/sshd_config -p wa -k sshd_configs

This logs every write or attribute change to the config file to the audit log, which is much harder for an attacker to scrub if you're shipping those logs to a central collector.

---

Incident Response: What to Do if a Backdoor is Found

When I found the rogue key on that jumpserver, my first instinct was to delete it. Bad move. If you delete it immediately, you lose the chance to see where else that key is used in your environment.

1. Isolate: Use iptables or your cloud NSG to restrict SSH access to a single management IP. Don't kill the session yet; you might need to see what processes are running.
2. Dump Memory: If you suspect a sophisticated rootkit, a memory dump might be the only way to see the LD_PRELOAD hooks.
3. Check Environment Variables: Look at /etc/environment and ~/.ssh/rc. Attackers love putting execution triggers in sshrc because it runs before the user gets a shell.
4. Restore and Patch: Reinstall the openssh-server package to ensure binaries and PAM modules are clean. Then, for the love of all that is holy, rotate every single key that was on that box.

We also found that attackers were using CVE-2023-38408 to exploit agent forwarding. If your admins are forwarding their SSH agents to compromised boxes, the attacker can jump back to the admin's local machine or other servers. Disable agent forwarding unless absolutely necessary.

---

Preventing Future Persistence

The "Jugaad" way of managing keys has to end. We're moving toward a short-lived certificate model, but in the meantime, standard OpenSSH security hardening helps. We've started enforcing MFA via OATH-TOTP for all external SSH entries. Even if an attacker gets a key into authorized_keys, they still need the secondary token.

I also set up a strict whitelist for the sshd_config:

# Hardening the daemon
PasswordAuthentication no PubkeyAuthentication yes PermitRootLogin prohibit-password AllowGroups ssh-users MaxAuthTries 3

It’s simple, but it cuts down the attack surface significantly. The MaxAuthTries setting is particularly useful for slowing down brute-force attempts that try to cycle through multiple keys in a single session.

Next on my list is to automate a daily check for lsattr on all .ssh directories. If I see that +i attribute appearing on files I didn't lock down myself, I'll know something is up. I'm also curious about how the new eBPF-based monitoring tools handle sshd syscall filtering, but that's a project for a Monday morning, not a Friday afternoon.