The SSH Attack Surface: Beyond Authentication
During a recent red-team engagement, I observed a developer bypass standard logging by using a simple base64 encoding trick to exfiltrate database credentials via an interactive SSH session. Standard syslog and auth.log captured the login event, but the actual data movement remained invisible. This visibility gap is where SSH session recording becomes mandatory for any hardened environment. Implementing team SSH access management ensures that every interaction is logged outside the target host, preventing local log tampering.
Recent vulnerabilities like CVE-2024-6387 (regreSSHion) have reminded us that the SSH daemon itself is a high-value target. A signal handler race condition in OpenSSH's server (sshd) allows for unauthenticated remote code execution as root on glibc-based Linux systems. Even if your patching cycle is aggressive, an attacker who gains access needs to be monitored at the keystroke level.
We also have to contend with the Terrapin Attack (CVE-2023-48795). This prefix truncation attack allows a Man-in-the-Middle (MitM) attacker to downgrade connection security by breaking the integrity of the SSH extension negotiation. While this targets the handshake, the subsequent session behavior must be logged to detect anomalous commands executed under a compromised, downgraded session.
What is an SSH Session? Defining the Meaning and Scope
An SSH session is more than just a remote shell. It is a secure tunnel that can encapsulate interactive TTY sessions, SFTP file transfers, X11 forwarding, and TCP port forwarding. For most security teams, the primary concern is the interactive TTY (Teletype) session, where a user executes commands directly on the host.
When a user connects via ssh user@host, the server allocates a pseudo-terminal (pty). Recording this session involves capturing the input (stdin) and output (stdout/stderr) streams associated with that pty. This is distinct from command logging, which only records the command executed, often missing the context of pipes, redirects, or interactive sub-shells.
Non-interactive sessions, such as those used by Ansible or hardened CI/CD pipelines, present a different challenge. While these are often excluded from full TTY recording to save disk space, their execution paths should still be audited via auditd to ensure the automation hasn't been subverted.
The Critical Role of Session Recording in Modern DevOps
In a high-velocity DevOps environment, "root" access is often replaced by "sudo" access, but the risk remains the same. I have seen production outages caused by a single rm -rf executed in the wrong directory because a developer had multiple terminal tabs open. Session recording provides the "black box" flight recorder data needed to reconstruct these events.
Beyond security, session recording acts as a training and troubleshooting tool. When a senior engineer performs a complex live-patching operation, the recorded session serves as a literal step-by-step guide for junior members. It eliminates the "it worked on my machine" ambiguity by showing exactly what was typed and what the system returned in real-time.
We use session recording to bridge the gap between Infrastructure as Code (IaC) and emergency manual interventions. If a change is made outside of Terraform or Pulumi, the session log is the only source of truth for what actually happened on the instance.
Meeting Regulatory Requirements: SOC2, HIPAA, and PCI-DSS
Compliance frameworks increasingly demand granular auditing of administrative access. PCI-DSS Requirement 10 focuses on monitoring and tracking all access to network resources and cardholder data. Standard bash history is insufficient, a fact highlighted by the OWASP Top 10 as a major risk factor in broken access control.
SOC2 Type II audits require evidence that access controls are functioning as intended. Providing a random sample of recorded SSH sessions to an auditor demonstrates that your organization doesn't just have an access policy, but an active monitoring capability. It proves that "privileged actions" are being logged in a non-repudiable format.
For healthcare entities governed by HIPAA, protecting ePHI (Electronic Protected Health Information) extends to the systems hosting the data. If a DBA accesses a database server via SSH, the session recording provides the audit trail required to prove no unauthorized data was viewed or exported.
The Indian Context: CERT-In and DPDP Act 2023
Indian SMEs and local ISPs frequently utilize unmanaged VPS instances from providers like E2E Networks or Netmagic. Often, these instances lack centralized logging, leaving them vulnerable to undetected lateral movement. Under the CERT-In Cyber Security Directions (April 2022), all Indian entities are mandated to maintain logs of ICT systems for a rolling period of 180 days.
The Digital Personal Data Protection (DPDP) Act 2023 further emphasizes the responsibility of "Data Fiduciaries" to protect personal data. If a breach occurs via an SSH entry point, the lack of session logs could be interpreted as a failure to take "reasonable security safeguards," leading to significant financial penalties (up to ₹250 crore).
For the BFSI sector, RBI security audits are even more stringent. Vendors providing services to Indian banks must often provide "keystroke-level" audit trails. Implementing pam_tty_audit and tlog is the most cost-effective way to meet these mandates without investing in expensive proprietary PAM (Privileged Access Management) suites.
Native Linux SSH Session Recording Techniques
Before jumping to third-party tools, I always recommend looking at what the Linux kernel and standard utilities provide. The most basic tool is the script command, which makes a typescript of everything displayed on your terminal.
To force a user to record their session using script, you can modify their .bash_profile or .profile, but this is easily bypassed by a savvy user. A more robust way is using the ForceCommand directive in sshd_config.
# Example of using script in a wrapper
/usr/local/bin/ssh-recorder
#!/bin/bash SESSION_LOG="/var/log/ssh_sessions/${USER}_$(date +%Y%m%d_%H%M%S).log" exec /usr/bin/script -q -c "$SHELL" "$SESSION_LOG"
The primary limitation of script is that it produces flat text files that are hard to search and can be easily tampered with if the user has write access to the log directory. It also fails to capture timing information accurately, making "playback" look like a wall of text rather than a real-time recreation.
Configuring TTY Logging in Linux Environments
For kernel-level keystroke logging, we use pam_tty_audit.so. This module allows the system to log every keystroke to the audit framework (auditd), even if the user is using a non-interactive shell or attempts to hide their input.
I configure this by editing the PAM configuration for SSH. This ensures that even if a user manages to kill their shell, the keystrokes captured by the kernel up to that point are preserved in /var/log/audit/audit.log.
# /etc/pam.d/sshd
Enable TTY auditing for specific high-privilege users
session required pam_tty_audit.so disable=* enable=root,sysadmin,devops
To view the captured keystrokes, we use the ausearch utility. This is particularly useful for identifying what a user typed in a password prompt or an interactive editor like vim, which script might obfuscate with escape codes.
$ ausearch -k ssh_logs -i
Using tlog for Standardized Terminal I/O Recording
tlog is the current gold standard for open-source session recording. It records everything that passes through the terminal between the user and the shell. Unlike script, tlog outputs data in JSON format, which includes timing information and terminal window size changes.
We typically implement tlog as a login shell wrapper. When a user logs in, tlog-rec-session is started, which then starts the actual shell. This makes it very difficult to bypass. The logs are sent to syslog, which can then be forwarded to a centralized log management system or a modern SIEM for real-time threat detection.
# Install tlog on RHEL/CentOS/Fedora
$ sudo dnf install tlog
Configuration snippet for /etc/tlog/tlog-rec-session.conf
{ "shell": "/bin/bash", "notice": "Your session is being recorded for security purposes.", "writer": "journal" }
To force tlog for a specific group of users, we modify the /etc/ssh/sshd_config file. This is a cleaner approach than changing the user's default shell in /etc/passwd, as it specifically targets SSH access.
# /etc/ssh/sshd_config
Match Group recorded_users ForceCommand /usr/bin/tlog-rec-session PermitTTY yes
Playback and Analysis of tlog Sessions
The beauty of tlog is the playback capability. Since it records timing data, we can watch the session exactly as it happened. This is invaluable during incident response when you need to see the speed and cadence of a user's typing, which can help distinguish between a human and an automated script.
We can pipe logs from journalctl directly into the tlog-play utility. This allows for real-time monitoring of an active session if the logs are being streamed to a central server.
# Playback a specific session from the journal
$ journalctl -t tlog-rec-session -f -o export | tlog-play --reader=export
One challenge we encountered was the volume of data. Interactive sessions generate a lot of JSON. We handle this by using logrotate on the local journal and aggressive filtering at the log collector level to only store sessions involving sudo or access to sensitive directories.
Asciinema: Sharing and Embedding Terminal Sessions
While tlog is for security, asciinema is for collaboration. I use asciinema to record complex setup procedures that need to be shared with the team. It produces a lightweight, JSON-based format (.cast) that can be played back in a web browser using a JavaScript player.
It is not a security tool because the user must explicitly start the recording. However, in a transparent DevOps culture, we encourage engineers to record their "on-call" sessions using asciinema and attach the resulting file to the incident ticket.
# Start recording a session
$ asciinema rec incident-123.cast
Stop with Ctrl-D and play back locally
$ asciinema play incident-123.cast
The .cast files are text-based, meaning you can grep them for specific commands before playing them back. This makes finding the "smoking gun" in a 30-minute recording much faster than scrubbing through a video file.
Enterprise-Grade Tools: Teleport SSH Session Recording
For large-scale infrastructure, manual tlog deployments become a management nightmare. This is where Teleport comes in. Teleport is a proxy-based access plane that replaces sshd with its own agent. It records sessions at the proxy level, meaning the user cannot stop the recording even if they have root access on the target node.
Teleport's architecture is built for compliance. It stores session recordings in an encrypted S3 bucket or a similar object store. The audit logs are centralized, providing a single pane of glass for SSH, Kubernetes, and Database access.
# Teleport configuration snippet (teleport.yaml)
auth_service: enabled: "yes" session_recording: "proxy" # Records at the proxy level proxy_checks_host_keys: "yes" ssh_service: enabled: "yes"
The "proxy-based" recording mode is particularly powerful. Even if an attacker exploits a zero-day in the Linux kernel on the target node, the recording is happening on the Teleport Proxy, which remains uncompromised. This provides a level of integrity that node-local tools like tlog cannot match.
Tailscale SSH Session Recording: Identity-Based Access
Tailscale takes a different approach by integrating SSH into its mesh VPN. With Tailscale SSH, the identity of the user is verified via your OIDC provider (Okta, Google, Azure AD) before the connection is even established. Session recording is handled by a "recorder" node that acts as a witness to the session.
I prefer Tailscale for distributed teams where users are connecting from untrusted networks. The recording is node-based but managed centrally. Tailscale handles the distribution of the recording policy, ensuring that every node in the tailnet adheres to the same audit rules.
The main advantage here is the lack of a central proxy bottleneck. While Teleport Proxies can become a single point of failure or a latency source, Tailscale's mesh approach keeps the data path as direct as possible while still shunting a copy of the TTY stream to a recorder.
Comparing Architecture: Proxy-based vs. Node-based Recording
When choosing a strategy, we evaluate the trade-off between security and complexity. Proxy-based recording (Teleport) is the most secure because the logs never reside on the target machine. However, it requires all SSH traffic to flow through a central cluster, which can be a challenge for multi-region deployments.
Node-based recording (tlog, Tailscale) is easier to deploy incrementally. You can enable it on a per-server basis. The risk is that a compromised node could potentially allow an attacker to delete the logs before they are shipped to a remote server. We mitigate this by using systemd-journal-remote to stream logs in real-time.
| Feature | tlog (Node-based) | Teleport (Proxy-based) | Tailscale (Node-based) |
|---|---|---|---|
| Bypass Risk | Moderate | Low | Low |
| Latency | Negligible | Depends on Proxy location | Low |
| Searchability | Via Journald/ELK | Built-in UI | Centralized Logs |
| Setup Complexity | High (Manual) | High (Infrastructure) | Low (SaaS) |
Ensuring Secure Storage and Encryption of Audit Logs
Recorded sessions often contain sensitive information, including environment variables, database queries, and occasionally inadvertently typed passwords. These logs must be treated with the same level of security as the data they protect. We use GPG encryption for logs at rest.
For non-repudiation, I recommend timestamping the logs. Using OpenSSL, we can create a Time Stamp Query (TSQ) and send it to a Time Stamping Authority (TSA). This proves that the log existed in its current state at a specific point in time and has not been altered.
# Generate a timestamp query for a log file
$ openssl ts -query -data /var/log/tlog/session.log -out /var/log/tlog/session.log.tsq
Verify the timestamp (assuming you have the TSA certificate)
$ openssl ts -verify -in session.log.tsr -queryfile session.log.tsq -CAfile tsa_cert.pem
In the Indian context, the DPDP Act's focus on "Data Minimization" means we shouldn't keep these logs forever. We implement a tiered retention policy: 180 days for all sessions (to satisfy CERT-In), but 2 years for sessions involving "Critical Information Infrastructure" (CII) as defined by NCIIPC.
Balancing Developer Privacy with Security Requirements
One common pushback from engineering teams is the "Big Brother" feel of session recording. To maintain trust, we are transparent about what is recorded. We use the notice feature in tlog to display a banner upon login, reminding the user that the session is being audited.
We also implement "Privacy Zones." For example, we can configure pam_tty_audit to disable logging when a user is in a specific group that only handles non-sensitive tasks, though this is rarely recommended for production systems. A better approach is to use automated scanning to redact PII (Personally Identifiable Information) from the logs before they are indexed in our SIEM.
Developer privacy is also protected by strictly controlling who can view the recordings. Access to the session playback UI or the log files is restricted to the Security Operations Center (SOC) and is only granted during an active investigation or audit.
Automating Session Cleanup and Retention Policies
Session logs can grow to terabytes quickly if you are recording hundreds of developers. We use a combination of find and s3cmd to move logs older than 30 days to cold storage (AWS Glacier or Azure Archive Storage).
# Daily cron job to move old logs to S3
#!/bin/bash find /var/log/ssh_sessions/ -name "*.log" -mtime +30 -exec s3cmd put {} s3://my-audit-bucket/ssh-logs/ \; -exec rm {} \;
We also implement a "metadata-first" search strategy. We extract the commands used in a session and store them in a searchable database (like Elasticsearch), while the full TTY recording remains in object storage. This allows us to search for "who ran docker kill" without needing to scan every byte of every recorded session.
Forensic Analysis: Reconstructing a Breach
When an alert triggers, the first thing I do is pull the session ID from the audit log. Using tlog-play, I can see the attacker's thought process. Did they try multiple passwords? Did they check /etc/shadow? The cadence of their typing often reveals if they are using a copy-paste list of exploits or manually exploring the system.
We combine tlog data with auditd system call logs. If the session recording shows the user running cat /etc/passwd, the auditd log will show the corresponding open() system call. This dual-layer approach makes it nearly impossible for an attacker to hide their tracks entirely.
# Search for file access by a specific user in audit logs
$ ausearch -u username -f /etc/shadow -i
If an attacker attempts to delete the tlog binary or stop the service, the auditd rules we have in place will trigger an immediate high-severity alert. We monitor for any modification to the sshd_config or the tlog configuration files using auditctl.
# Monitor SSH configuration for unauthorized changes
$ auditctl -w /etc/ssh/sshd_config -p wa -k sshd_config_changes
Next Command: tlog-rec-session --help to explore the --latency and --payload flags for fine-tuning your recording overhead.