Building effective threat hunting and detection rules in Elastic Security

Detection engineering is about operationalizing threat detection efforts based on available data, knowledge of the threat landscape, and the attack surface of your environment.

The security community has a long history of sharing detection rules and collaborating to strengthen defenses across organizations; Elastic has always participated in these community collaborations. For example, Elastic Security opened its detection rules repository in 2020 and its endpoint behavior protections in 2022. These resources provide access to the rule logic behind our out-of-the-box SIEM detection and endpoint rules even if you are not using our tooling.

While this broad community effort provides robust coverage, you will likely encounter situations where you have detection use cases that are unique to your environment due to custom data sources, specific operational needs, or novel detection ideas. In these cases, you will need to fill coverage gaps by creating your own detections.

Before we begin, we need to define our focus — for example, the technology we want to detect threats for or an attack scenario we want to detect.

Then, we will follow this process to create and deploy detection rules:

Step 1. Create and refine detection logic
Step 2. Create a rule with detection and response context
Step 3. Preview and test the rule
Step 4. Deploy rule to production

Our focus area will be AWS, so let’s walk through a couple examples of threat detections for AWS. 

Amazon CloudTrail is an audit logging service that provides a record of nearly every API call made within your AWS environment. CloudTrail events include information about who made the API call (IAM user, role, or service principal), when the call was made, from what location, and what parameters were passed. It is an essential data source for auditing and monitoring AWS operations.

Once ingested, CloudTrail logs provide a massive amount of information for security analysts to sift through. Without meaningful threat hunting queries or detection logic in place, threat actors can blend in with the noise and remain undetected.

When developing detection rules, we prioritize capturing underlying attacker techniques rather than relying solely on static indicators or signatures that can be easily bypassed. Focusing on behavioral patterns that reflect what adversaries are actually doing ensures our detections remain relevant even as tooling evolves. You should always consider how an attacker might evade your rule, such as by altering command-line arguments or abusing trusted processes. At Elastic, we practice what we preach and design our prebuilt rules detection logic to be resilient to such evasions.

We strive to balance precision and coverage by tuning for realistic false positives while avoiding overly narrow logic that might miss variations of the threat. We document known limitations or expected noise clearly in the rule itself as additional context. If our approach generates high-value signals but also produces too many false positives to reliably alert on, we consider implementing it as a threat hunting query instead. This allows analysts to investigate patterns of interest without overwhelming them with alerts. 

By taking this thoughtful, technique-focused approach, you can develop effective detections that provide intended coverage while maintaining analyst trust and avoiding alert fatigue.

We’ll go through two examples in the next sections to demonstrate our approach and process. In the first example, we’ll develop a threat hunting query to capture behavior that might not always be indicative of an attack but will certainly give us events that should be further investigated. And in the second example, we’ll move on to a high-fidelity detection that is more indicative of threat behavior and rises to the standard for a true alert.

Good quality detections stem from good quality threat research. For example, Elastic Security Labs’ Exploring AWS STS AssumeRoot and Rhino Security’s AWS IAM Privilege Escalation – Methods and Mitigation both describe how attackers can exploit misconfigurations in AWS Identity and Access Management (IAM) to elevate a low-privilege account to full administrative rights. 

IAM lets you define who (users, groups, or roles) can authenticate to your AWS account and control exactly what they can do via JSON policy documents. These documents specify which AWS actions are allowed or denied and on which resources. Although IAM is designed to enforce least-privilege access, overly permissive or risky permission combinations can also be abused.

Like all AWS services, CloudTrail records IAM API calls, providing a detailed audit trail of activity. We can query CloudTrail for high-risk IAM API calls as a first step in hunting for potential threats in our environment. Because these API calls can also appear in legitimate administrative workflows, using a threat hunting query rather than an automated detection helps avoid unnecessary alerts while enabling proactive investigation.

Let’s start by querying our CloudTrail datastream for all IAM API calls made over the last 30 days.

An interesting research article by Permiso “CloudTrail Logging Evasion: Where Policy Size Matters” outlines a defense evasion technique that abuses CloudTrail’s logging size constraints.

The requestParameters field of a CloudTrail log typically displays IAM policies when they’re created, modified, or attached to an identity. This includes details, such as the policy name and the full policy document content. However, when policies padded with large amounts of insignificant whitespace (spaces, tabs, or line breaks) reach a size range of 102,401 to 131,072 characters, they are omitted from CloudTrail logs and are instead rendered as "requestParameters too large." By exploiting this gap, threat actors can bypass monitoring performed through CloudTrail and can effectively conceal unauthorized permissions changes. 

We can detect this behavior by looking for IAM API calls with the requestParameters property containing reason:”requestParameters too large” and omitted:true. This is not a common set of circumstances to expect in normal operations and would almost certainly be an intentional attempt to abuse this gap in CloudTrail logging. We can confidently use this query to create a detection rule, knowing that our analyst will receive a true positive alert indicative of suspicious activity. 

Detection rules not only define the detection logic and how often it is executed but also provide crucial context for detection engineers and security analysts.

Here are best practices for rule creation, focused on reducing mean time to respond (MTTR) by providing useful information to analysts:

• Name the rule: Capture concisely what the rule will detect.

• Describe the rule: Add a clear description for those unfamiliar with the specific threat or for team members reviewing and tuning the rule.
• Apply appropriate severity and risk scores: Help analysts prioritize alerts during triage and influence entity scoring within Elastic Security.
• Map your detection to MITRE ATT&CK®: Track and maintain detection coverage, and easily find similar detections.
• Add custom highlighted fields: Ensure security analysts have relevant context readily available when reviewing an alert.
• Provide an investigation guide: Offer insights from detection engineers, presenting key checks and attack scenario context in a standardized form to assist analysts under pressure.
• Outline false positive examples: Provide valuable information for weeding out false positive alerts.
• Add reference URLs: Include links for further learning about the threat or for future reference.
• Provide a setup guide: Capture any specific settings like logging policies that must be enabled for proper rule operation, aiding seamless future data source onboarding.

By following the four key steps — creating and refining detection logic, adding detection and response context, previewing and testing the rule, and finally deploying it to production — you can build effective threat detection capabilities. This structured approach combined with best practices and available Elastic features like ES|QL and Elastic AI Assistant empower security teams to enhance their operations and maintain a robust defense against evolving threats. We highly recommend reading about all the detection engineering features beyond ones related to detection creation in this detailed post.