During a recent incident response engagement for a Mumbai-based manufacturing firm, we observed that 92% of their 1,500 daily "Critical" alerts were false positives triggered by legacy ERP software. The SOC team was suffering from chronic alert fatigue, missing a genuine lateral movement attempt that utilized a Tally.exe side-loading technique. This scenario is common across Indian enterprises where flat networks and legacy software create a high-noise environment. Traditional SOC analysis, which relies on reactive monitoring of pre-defined vendor rules, is failing. We are seeing a mandatory shift toward Detection Engineering—a disciplined, code-centric approach to building, testing, and automating threat detection logic within a modern SIEM.
What is Detection Engineering? Definition and Core Concepts
Detection Engineering is the process of designing, developing, and maintaining the logic used to identify malicious activity within an environment. Unlike traditional SOC analysis, which treats the SIEM as a black box of vendor-provided rules, Detection Engineering treats detection as a software development lifecycle (SDLC). We write code to catch code.
The Evolution of Modern Threat Detection
In the early 2010s, detection was synonymous with signature-based antivirus and basic SQL-like queries in a SIEM. As attackers shifted to Living off the Land (LotL) techniques—using native tools like PowerShell, WMI, and Certutil—static signatures became obsolete. Understanding OpenSSH security best practices is vital for securing these entry points. Modern detection must focus on behavior and telemetry. We no longer look for a specific file hash; we look for the behavior of a process spawning a shell to download a payload from an external IP.
Detection Engineering vs. Traditional SOC Analysis
Traditional SOC analysts are "consumers" of alerts. They follow a runbook to determine if an alert is a true positive. Detection Engineers are "producers." We build the logic that generates the alert. If a SOC analyst finds a false positive, they might ignore it; a Detection Engineer modifies the code to ensure that specific false positive never fires again.
Why Detection Engineering is Critical for Cybersecurity
The Indian threat landscape is unique due to the widespread use of localized ERP systems and unlicensed software in SME sectors. Standard global detection sets often fail here because they don't account for the "normal" noise of these environments. Detection Engineering allows us to customize our defenses. With the Digital Personal Data Protection (DPDP) Act 2023 mandating strict breach reporting, the cost of a missed detection or a slow response has shifted from a technical failure to a massive legal and financial liability (potentially up to ₹250 crore in penalties).
The Detection Engineering Lifecycle
We treat the detection lifecycle as a continuous loop. If a detection is not tested, it does not exist.
Phase 1: Threat Modeling and Hypothesis Generation
We start by identifying what we are trying to catch. For an Indian enterprise, we might model the threat of a container breakout in a cloud-native environment (CVE-2024-21626) or a side-loading attack on accounting software. We generate a hypothesis: "If an attacker uses WinRAR to execute a malicious script via CVE-2023-38831, we will see a child process of WinRAR.exe spawning cmd.exe or powershell.exe with suspicious arguments."
Phase 2: Data Collection and Telemetry Analysis
Without telemetry, detection is impossible. We verify that our endpoints are sending the correct Event IDs. For Windows environments, we prioritize Sysmon and Windows Event Logs (Security, System, and Application).
# Checking if Sysmon is running and collecting Event ID 1 (Process Creation)
Get-Service | Where-Object {$_.Name -like "sysmon"} wevtutil qe Microsoft-Windows-Sysmon/Operational /c:1 /f:text /q:"*[System[(EventID=1)]]"
Phase 3: Detection Development and Logic Creation
We use Sigma as a vendor-neutral language to write our detections. This allows us to share logic across different platforms like Elastic, Splunk, or Microsoft Sentinel. Below is a Sigma rule we developed to catch suspicious Certutil usage, a common LotL technique.
title: Suspicious Certutil Download
id: 5f3243d5-1234-4a5b-8c9d-1234567890ab status: experimental description: Detects certutil.exe downloading a file from the internet logsource: product: windows category: process_creation detection: selection: Image|endswith: '\certutil.exe' CommandLine|contains: - '-urlcache' - 'split' condition: selection falsepositives: - Administrative scripts (rare) level: medium
Phase 4: Testing, Tuning, and False Positive Reduction
Once the rule is written, we convert it to the target platform's query language using sigma-cli.
# Converting the Sigma rule to an Elasticsearch query
sigma-cli convert -t elasticsearch -c windows-audit rules/windows/process_creation/proc_creation_win_powershell_susp_download.yml
We then run the query against historical data to check for false positives. If a legitimate update script for an Indian ERP tool triggers the rule, we add an exclusion based on the file path or the signing certificate.
Phase 5: Deployment and Continuous Monitoring
We deploy the rule using a CI/CD pipeline. We treat our detection repository like any other codebase, using pytest to validate the YAML structure before deployment. For teams managing these environments, using a browser based SSH client simplifies access without compromising security.
# Running validation tests on the rule set
pytest --sigma-validation ./rules/production/
How to Start a Career in Detection Engineering
The transition from a SOC Analyst to a Detection Engineer requires a shift in mindset from "investigation" to "automation."
Essential Skills for Aspiring Detection Engineers
- Log Analysis: Deep understanding of JSON, XML, and EVTX formats.
- Query Languages: Proficiency in KQL (Azure/Sentinel), SPL (Splunk), and Lucene (Elastic).
- Programming: Python is non-negotiable for automating triage and rule deployment.
- Threat Research: Ability to read a CVE report or a vendor blog and extract actionable TTPs.
Top Detection Engineering Courses for Beginners and Pros
For those in India looking to upskill, we recommend focusing on platforms like the WarnHack Academy that provide hands-on labs.
- Detection Engineering Academy: Advanced, specialized training for practitioners.
- SANS SEC555: Logical Analysis and Security Analytics (Expensive, usually corporate-sponsored).
- Blue Team Labs Online (BTLO): Excellent for practical incident analysis.
Hands-on Practice: Detection Engineering on TryHackMe
TryHackMe has several rooms dedicated to the "Detection Engineering" and "SOC Level 2" pathways. We recommend starting with the "Windows Event Logs" and "Sysmon" rooms to understand the underlying telemetry before moving to the "Sigma" room.
Earning Your Detection Engineering Certification
While there is no single "Detection Engineering" certification that dominates the market yet, the following are highly regarded:
- GIAC Certified Detection Analyst (GCDA): Focuses on the analysis of modern logs.
- Certified Blue Team Level 1/2 (BTL1/BTL2): Highly practical and gaining traction in the Indian market.
- Splunk Core Certified Advanced Power User: Essential if you are working in a Splunk-heavy environment.
Best Resources for Detection Engineers
Must-Read Detection Engineering Books
- "Practical Threat Intelligence and Data-Driven Threat Hunting" by Isaac Khoo.
- "Detection Engineering for Beginners" (Various community-led e-books).
- "The Practice of Network Security Monitoring" by Richard Bejtlich (Core fundamentals).
Leveraging Detection Engineering GitHub Repositories for Open-Source Tools
The community is the greatest asset in this field. We frequently use and contribute to:
- SigmaHQ/sigma: The de-facto standard for detection rules.
- Neo23x0/signature-base: Florian Roth's repository of YARA and Sigma rules.
- OTRF/DetectionLab: A great tool for spinning up a local lab to test detections.
Staying Current with Detection Engineering Weekly Newsletters
The landscape moves too fast for books alone.
- Detection Engineering Weekly: Curated links on the latest detection techniques.
- Unsupervised Learning: Daniel Miessler’s newsletter covers the intersection of security and AI.
- CERT-In Advisories: Critical for Indian context-specific threats (e.g., localized ransomware campaigns).
The Job Market: Detection Engineering Jobs and Salary
The demand for Detection Engineers in India has surged by 40% year-on-year, particularly in the BFSI (Banking, Financial Services, and Insurance) and ITES sectors.
Common Roles and Responsibilities in Detection Engineering Jobs
- Developing custom detections for cloud-native applications (AWS/Azure/GCP).
- Managing the "Detection as Code" pipeline.
- Collaborating with Red Teams to validate detection coverage (Purple Teaming).
- Mapping organizational visibility to the MITRE ATT&CK framework.
Detection Engineering Salary Trends by Experience and Region
In India, salaries vary significantly based on the city and the complexity of the tech stack.
| Experience Level | Region | Annual Salary (Approx. INR) |
|---|---|---|
| Junior (1-3 Years) | Bangalore/Pune | ₹8,00,000 - ₹15,00,000 |
| Mid-Level (4-7 Years) | Hyderabad/NCR | ₹18,00,000 - ₹35,00,000 |
| Senior/Lead (8+ Years) | Remote/MNC | ₹40,00,000 - ₹75,00,000+ |
How to Prepare for a Detection Engineering Interview
Expect technical screens that involve:
- Writing a Sigma rule on a whiteboard for a specific TTP (e.g., Process Hollowing).
- Explaining the difference between a "False Positive" and a "Benign Positive."
- Discussing how to handle log ingestion costs versus detection coverage.
- A practical test involving log analysis in a SIEM of your choice.
The Future of Detection Engineering
The future is not just about more rules; it is about smarter triage.
Automation and AI in the Detection Lifecycle
We are moving toward "Agentic SOC" models where Large Language Models (LLMs) act as reasoning engines for alert triage. This is similar to building an AI-powered audit log analyzer for automated triage. Instead of a human analyst looking at a Certutil download alert, an LLM agent can pull the file hash, check VirusTotal, analyze the command line context, and decide if the alert should be escalated or suppressed.
import openaifrom sigma.collection import SigmaCollection from sigma.backends.elasticsearch import LuceneBackend
def agentic_triage(event_data): """Agentic function to triage alerts using LLM and Sigma conversion""" rule_yaml = """title: Suspicious Certutil Download logsource: product: windows category: process_creation detection: selection: Image|endswith: '\\certutil.exe' CommandLine|contains: '-urlcache' condition: selection"""
# Convert Sigma to Query backend = LuceneBackend() query = backend.convert(SigmaCollection.from_yaml(rule_yaml))
# Agentic Decision Logic prompt = f"Analyze this event: {event_data}. Cross-reference with query: {query}. Is this a False Positive for an Indian SME environment?" response = openai.ChatCompletion.create(model="gpt-4-turbo", messages=[{"role": "user", "content": prompt}]) return response.choices[0].message.content
We can also use local LLMs like Llama3 via Ollama to analyze Sysmon logs directly on-premise, ensuring data privacy in compliance with the DPDP Act.
ollama run llama3:8b "Analyze the following Sysmon Event ID 1 for TTPs and map to MITRE ATT&CK: [Log Data]"The Shift Toward Detection as Code (DaC)
Detection as Code (DaC) means applying software engineering best practices to detections. This includes:
- Version Control: Storing all rules in Git.
- Automated Testing: Using tools like
atomic-red-teamto trigger the TTP and verify the alert fires. - Peer Review: Every new detection requires a pull request and code review.
This approach is essential for Indian SOCs that are scaling rapidly. It ensures that the knowledge of a senior engineer is captured in code and doesn't leave the company when they change jobs.
Technical Insight: API Key Management for Agentic Workers
When building these automated pipelines, never hardcode credentials. Use the SIEM's API for secure communication between your agentic workers and the data lake.
curl -X POST "https://elastic-soc.internal:9200/_security/api_key" -d '{"name": "agentic-soc-worker"}'The next step in your detection journey should be the implementation of a validation framework. If you cannot prove your detection works by simulating the attack, the detection does not exist in a production-ready state.
kubectl logs -l app=detection-engine-agent -f --tail=100