Hardening SSH Access: Implementing Real-time Session Logging and SIEM Integration
During a recent forensic audit for a financial services provider in Mumbai, we discovered that while their perimeter was reinforced, an internal lateral movement went undetected for three weeks. The attacker had gained access via a compromised developer credential and used a standard SSH session to exfiltrate database schemas. The standard /var/log/auth.log showed the login and logout timestamps, but the actual commands executed—the "what" and "how"—were missing. This visibility gap is the primary driver for implementing robust SSH session logging and SIEM integration.
SSH Session Meaning: Understanding the Basics
An SSH session is more than just an encrypted tunnel; it is a PTY (Pseudo-Terminal) allocation that facilitates interactive communication between a client and a server. When a user authenticates, the sshd daemon forks a process, drops privileges to the user's level, and spawns a shell. From a security perspective, we distinguish between connection logging (metadata like IP, port, and timestamp) and session logging (the actual I/O stream of the terminal).
In most default Linux distributions, sshd logs only the authentication results. We observed that this level of logging fails to meet the compliance requirements set by the Digital Personal Data Protection (DPDP) Act 2023 and the RBI Master Direction on Cyber Security. For Indian critical infrastructure, the ability to reconstruct a session is no longer optional; it is a mandatory requirement for post-incident analysis, aligning with the OWASP Top 10 recommendations for logging and monitoring.
The Importance of Auditing and Compliance
CERT-In (Indian Computer Emergency Response Team) frequently highlights that brute-force attacks against SSH are the most common entry vector for ransomware in Indian SMEs. Without detailed session logs, determining if an attacker successfully ran sudo -l or modified /etc/shadow becomes a matter of guesswork. Auditing provides the "paper trail" needed to verify that administrative actions align with change management tickets.
We use session logging to enforce accountability. When multiple administrators share a service account—a practice we strongly advise against but frequently encounter—session recording is the only way to attribute a specific command to a specific human actor. This is critical for meeting SEBI cybersecurity framework mandates, which require granular logging of all privileged access to trading systems and data repositories, especially when defending against MFA proxy bypass attacks.
How to Enable SSH Session Logging in Linux
Using the 'script' Command to Log Session to File
The script utility is a native tool found in almost all Unix-like environments. It records everything printed on the terminal to a file. We tested this for basic auditing, and while effective, it requires manual invocation or specific shell hooks to ensure it cannot be bypassed by the user. For a more streamlined approach, a browser based SSH client can centralize these logs automatically, removing the need for manual configuration.
# Start a session recording to a specific file
script -f /var/log/ssh_sessions/session_$(date +%Y%m%d_%H%M%S).log
To view these logs in real-time, you can use tail -f. However, the raw output often contains ANSI escape codes for colors and cursor movements, which can make the log files difficult to read with standard text editors. We recommend using cat or less -R to interpret these escape sequences correctly during review.
Configuring TTY Logging for Administrative Oversight
For a more resilient approach, we use auditd to monitor TTY input. This captures the actual keystrokes at the kernel level, making it significantly harder for a user to hide their actions by aliasing commands or using shell scripts. We observed that TTY logging is particularly effective at catching obfuscated commands used in CVE-2024-6387 (regreSSHion) exploitation attempts.
# Enable TTY logging for all users in /etc/pam.d/sshd
session required pam_tty_audit.so disable=* enable=root,admin_user
After enabling this, the logs are sent to /var/log/audit/audit.log. You can query these logs using the ausearch tool. This method provides a higher level of integrity because the logging happens outside the user's shell environment, preventing them from simply killing the script process to stop the recording.
Automating Logs via .bashrc or .profile
To ensure every interactive session is recorded without relying on the user's cooperation, we can force the script command to run upon login. By modifying the global /etc/profile or a specific user's .bashrc, we can trigger the recording automatically. I observed that this method is popular in jump servers (bastion hosts) used by Indian government contractors.
# Add this to /etc/profile to force session recording
if [ -n "$PS1" ]; then test -d ~/logs || mkdir ~/logs script -q -a ~/logs/session-$(date +%Y%m%d-%H%M%S).log exit fi
Note the exit command after script. This ensures that when the user exits the recorded shell, the parent SSH session also terminates. Without this, the user would simply drop back into an unrecorded shell. We found that sophisticated attackers might try to bypass this by using ssh user@host /bin/sh to avoid loading the profile, so this must be combined with restricted shell configurations.
Advanced SSH Session Recording Solutions
Top SSH Session Recording Open Source Tools
While native tools are useful, they often lack the centralized management and playback features required for enterprise environments. We evaluated several open-source projects that specialize in SSH session recording. These tools typically sit between the user and the target shell, acting as a transparent proxy or a shim that captures data streams without interfering with the user experience.
- Teleport: A modern replacement for sshd that provides built-in session recording, identity-based access, and a web-based playback UI.
- Asciinema: Excellent for recording and sharing terminal sessions in a lightweight, text-based format that allows for copy-pasting from the playback.
- TTYRecorder: A specialized tool for capturing TTY output and timing information, allowing for perfect synchronization during playback.
Comparing TTYRecorder, Asciinema, and Teleport
In our testing, Teleport emerged as the most robust for compliance-heavy environments like Indian fintech. Unlike PuTTY or local script logs, Teleport stores recordings in a centralized backend (like S3 or a dedicated auth server), preventing local users from deleting their own tracks. This addresses a major weakness in local logging where a compromised root user can simply rm -rf /var/log/ssh_sessions.
Asciinema is better suited for documentation and training rather than security auditing. It stores sessions in a JSON-based format. While highly portable, it lacks the tamper-evident features of a dedicated security proxy. TTYRecorder is a middle-ground solution, providing the raw timing files needed to replay a session exactly as it happened, including the pauses between keystrokes which can be vital for behavioral analysis.
Benefits of Visual Session Playback
Reviewing thousands of lines of raw text logs is inefficient. Visual playback allows security analysts to watch a session like a video. We have used this to identify "fat-finger" errors versus malicious intent. For example, if an admin accidentally runs rm -rf / bin instead of rm -rf ./bin, the visual context of their previous commands and their reaction time can help distinguish between a mistake and a deliberate act of sabotage.
Furthermore, visual playback tools often include searchable indexes. If we know a specific file like /etc/hosts was modified, we can search the session recordings for that string and jump directly to the moment the file was opened in vim. This drastically reduces the Mean Time to Detect (MTTD) during an active incident response in a large-scale Indian data center.
SSH Session Logging in Popular Clients
MobaXterm SSH Session Logging: A Step-by-Step Setup
MobaXterm is widely used by system administrators in India for its all-in-one capabilities. To enable session logging, you must configure it at the global level or per-session level. I recommend global logging to ensure no session is missed. Navigate to Settings > Configuration > Terminal and locate the "Log terminal output to the following directory" option.
- Select a secure local directory (e.g.,
C:\SSH_Logs\). - Set the "Logging mode" to "All terminal output".
- Use the timestamp variable
_Y%_M%_D%_%h%_%m%_%s%in the filename to prevent overwriting. - Ensure the directory has restricted NTFS permissions so other local users cannot read the logs.
Configuring PuTTY and Termius for Local Log Storage
PuTTY remains the workhorse for many legacy environments. To enable logging in PuTTY, go to Session > Logging. Select "All session output" and specify a log file path. We observed that many admins forget to check the "Flush log file frequently" option. Without this, if PuTTY crashes or the Windows machine reboots, the last few minutes of the session might be lost from the log file.
Termius, being a cross-platform client, offers cloud-synced history, but for security professionals, local logging is preferred. In Termius, you can enable "Terminal Logging" in the settings. This is particularly useful for mobile admins who might need to perform emergency fixes via a smartphone; however, be aware that storing these logs on a mobile device increases the risk of data exposure if the device is lost or stolen.
Managing Active Connections
How to Logout from SSH Session Safely
Safely terminating a session ensures that all buffers are flushed and the PTY is correctly released. We always recommend using the exit command or Ctrl+D. This allows the shell to execute any logout scripts defined in .bash_logout, which might include clearing temporary directories or updating an audit log. Simply closing the terminal window can sometimes leave "ghost" processes running on the server.
Terminating Hung or Idle SSH Sessions
Hung sessions are a common nuisance, often caused by network instability or aggressive firewalls. We use the SSH escape sequence to kill these sessions locally. Pressing Enter, then ~ (tilde), and then . (period) will immediately terminate the local SSH client without waiting for a timeout. On the server side, we use ClientAliveInterval to prune these dead connections automatically.
# Add to /etc/ssh/sshd_config to drop idle connections
ClientAliveInterval 300 ClientAliveCountMax 0
This configuration sends a "keep-alive" packet every 5 minutes. If the client does not respond, the server drops the connection immediately. This is a critical hardening step to prevent session hijacking where an attacker might try to take over an abandoned but still active PTY.
Difference Between Exit, Logout, and Killing a Process
While they might seem identical, the underlying signals differ. exit is a shell built-in that terminates the current shell process gracefully. logout is specifically for login shells. Killing the process (e.g., kill -9 <pid>) is a nuclear option that does not allow for cleanup. I observed that using kill -9 on an SSH session often leaves the entry in the utmp/wtmp files, leading to incorrect reporting in the who and last commands.
Best Practices for Secure SSH Log Management
Securing Log Files from Unauthorized Access
SSH logs are highly sensitive; they contain command history, potentially including sensitive arguments or accidentally pasted passwords. We enforce strict permissions on the log directory. Only the root user or a dedicated security auditing group should have read access. In a Linux environment, we use chown root:auditgroup and chmod 750 on the log storage path.
# Secure the session log directory
mkdir -p /var/log/ssh_sessions chown root:adm /var/log/ssh_sessions chmod 700 /var/log/ssh_sessions
We also implement the immutable attribute on older logs using chattr +i. This prevents even the root user from modifying or deleting the logs without first removing the attribute, providing an extra layer of protection against an attacker trying to cover their tracks after gaining full system control.
Centralizing SSH Logs with Syslog or ELK Stack
Local logs are vulnerable. We integrate sshd with a centralized SIEM (Security Information and Event Management) system. By configuring rsyslog to forward logs to an ELK (Elasticsearch, Logstash, Kibana) stack or a Splunk instance, we ensure that logs are preserved even if the source server is destroyed. This is a standard requirement for RBI-regulated entities in India.
# /etc/rsyslog.d/ssh-forward.conf
authpriv.* @siem-server.internal.in:514
In the SIEM, we create alerts for specific patterns. For example, a "Failed password" followed by an "Accepted password" from the same IP within 60 seconds triggers a brute-force success alert. We also monitor for the execution of high-risk commands like tcpdump, nc, or chmod 777 within the session logs themselves using Logstash filters.
Retention Policies for Session Recordings
Storage management is a practical concern. Session recordings, especially if they include high-bandwidth output like top or tail -f, can grow quickly. We implement a tiered retention policy. Active logs are kept on fast SSD storage for 30 days, then moved to compressed cold storage (like Amazon S3 Glacier or an on-premise tape library) for 2 years to comply with Indian financial regulations.
| Log Type | Hot Storage | Cold Storage | Compliance Requirement |
|---|---|---|---|
| Auth Logs | 90 Days | 5 Years | RBI / SEBI |
| TTY Sessions | 30 Days | 1 Year | Internal Audit |
| Root Sessions | 180 Days | 7 Years | DPDP Act / Forensic |
Hardening the SSH Daemon for Maximum Visibility
To ensure the logs we collect are accurate and difficult to spoof, the sshd_config must be hardened. We move away from default settings to VERBOSE logging and enforce modern cryptographic standards. This mitigates the Terrapin Attack (CVE-2023-48795), which can otherwise be used to downgrade the security of the connection and potentially interfere with the integrity of the logged data.
# /etc/ssh/sshd_config hardening for logging and SIEM
LogLevel VERBOSE SyslogFacility AUTHPRIV MaxAuthTries 3 PermitRootLogin no PubkeyAuthentication yes PasswordAuthentication no KexAlgorithms curve25519-sha256,[email protected] Ciphers [email protected],[email protected] MACs [email protected],[email protected]
The LogLevel VERBOSE is critical; it adds the fingerprint of the SSH key used for login to the log file. This allows us to map a login to a specific private key rather than just a username. In environments where multiple developers use the same ubuntu or ec2-user account, this is the only way to achieve individual accountability.
Real-time Monitoring with Journalctl
For immediate operational awareness, we use journalctl to tail the SSH service logs. This is particularly useful during a deployment or when troubleshooting connectivity issues for remote teams working across different Indian states with varying ISP stabilities.
# Monitor successful logins in real-time
journalctl -u ssh -f | grep 'Accepted'
We combine this with auditctl to watch for changes to the SSH configuration file itself. If an attacker gains root access, their first move is often to add their own SSH key to /root/.ssh/authorized_keys or modify sshd_config to allow password authentication. We set a kernel-level watch on these files.
# Watch for changes to sshd_config
auditctl -w /etc/ssh/sshd_config -p wa -k sshd_config_changes
Any write or attribute change to the config file will now generate a high-priority audit event. When piped to a SIEM, this can trigger an automated response, such as isolating the host or alerting the SOC (Security Operations Center) team via PagerDuty or WhatsApp for Business APIs commonly used in Indian tech stacks.
Identifying Key Fingerprints
During an investigation, we often need to verify which host key a client is seeing to detect Man-in-the-Middle (MitM) attacks. I use ssh-keygen to list the fingerprints of the server's public keys. This should be compared against the "known_hosts" file on the client side.
# View the fingerprint of the server's Ed25519 host key
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
If the fingerprint displayed by the server does not match what the client reports, it indicates that the SSH session is being intercepted. In the context of Indian corporate networks, this is often due to aggressive SSL/SSH inspection by corporate firewalls, which should be configured to bypass administrative SSH traffic to maintain end-to-end encryption integrity.
Probing SSH Services for Vulnerabilities
Before an attacker does, we use openssl to probe the SSH port and identify the version and supported ciphers. This is a quick way to check if a server is vulnerable to the regreSSHion bug without running a full exploit payload.
# Probe the SSH service banner and connection
openssl s_client -connect 192.168.1.100:22
While s_client is designed for TLS, it will return the SSH banner (e.g., SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u2). We use this information to cross-reference against the NIST NVD. If the version is outdated, we prioritize patching via apt-get update && apt-get install openssh-server followed by a service restart, ensuring that the logging configurations we've implemented remain active on the new version.
Observe the tail output of the authentication logs during a connection attempt to verify that the LogLevel VERBOSE settings are working correctly. You should see the public key SHA256 fingerprint in the log entry for every successful connection.
# Final check of the authentication log
tail -f /var/log/auth.log | grep -E 'Failed|Accepted'
Next, examine the /var/log/audit/audit.log to ensure that TTY keystrokes are being captured for the sessions you just initiated. Use aureport --tty to generate a human-readable summary of the keystrokes recorded during the session.