Rule-based detection

How it works

Where signature detection says “this specific thing is bad” and anomaly detection says “this is unusual,” Rule-Based Detection Security monitoring approach using predefined rules to identify suspicious or malicious activity based on specific conditions or patterns. says “this combination of conditions is suspicious in our environment.” A rule combines multiple indicators (a process name, a parent, a destination, a time window, a User An individual who interacts with a system, network, or application. privilege) and fires when the combination matches. The rule encodes a scenario the team has decided is worth a human looking at.

Rules let security teams encode their own institutional knowledge and their own policies into the detection layer. Most modern SIEMs and SOAR platforms run on rules as their primary Detection Mechanism Tools or methods employed to identify security incidents, including signature, anomaly, and behavioral detections. , supplemented by anomaly and behavioral models for what rules cannot easily express.

What a rule looks like in practice

⚠️ Office macro execution

If a Microsoft Office process spawns PowerShell and establishes an outbound network connection within 60 seconds, then fire a high-severity alert.

🚪 Sensitive system access from unmanaged device

If a user accesses three or more sensitive systems and the source device is not enrolled in the asset inventory, then trigger an investigation.

🔑 Privilege escalation with bulk access

If an account is granted elevated privileges and performs bulk data access within an hour and the activity is outside business hours, then escalate to the on-call engineer.

Representative platforms: Splunk Enterprise Security, IBM QRadar, Microsoft Sentinel analytics rules, Sigma A vendor-neutral, open standard for describing SIEM detection rules. Sigma rules can be translated into the query languages of multiple SIEMs, which makes them portable across detection platforms. rules deployed across any compatible SIEM, custom KQL or SPL queries.


Example: an Office macro chain

Walk through a rule-based detection

A SIEM rule produces the following alert:

Alert: Office process spawned PowerShell with outbound traffic
Severity: High
Rule: rule_off_macro_pwsh_egress
Confidence: rule-defined
Host: laptop-rgreen
Parent: winword.exe (PID 4128)
Child: powershell.exe (PID 4132)
Network: TLS connection to mail-storage-09.example.com:443
Window: All three observations within 28 seconds

The analyst’s first question is which family fired. The “Rule: rule_off_macro_pwsh_egress” field answers it. This is a rule-based alert, which means a security engineer wrote the logic that combined three observations into one verdict.

What the analyst checks next:

  • Read the rule. Understanding exactly which conditions had to be true for the alert to fire tells the analyst what scenario the engineer was trying to catch. A well-named rule is half the investigation.
  • Validate each condition. Is the parent process really Word? Is the network destination really an unfamiliar host? Process Telemetry Collection and transmission of security-relevant data from remote sources for monitoring and analysis. and DNS history confirm each condition independently of the rule.
  • Look at the user. Did rgreen open a document recently? The mail logs and the user’s recent inbox give context for whether the document came from a Phishing Deceptive messages (usually email; sometimes SMS, voice, or chat) that impersonate a trusted sender to lure the recipient into clicking, opening, or entering credentials. The bait is the email; the line is the impersonation; the catch is initial access. message or a legitimate sender.
  • Check for siblings. Did the same rule fire on other hosts in the same window? Multiple hits on one rule across an environment is usually a campaign, not a coincidence.

Rule-based detection is precise about what it caught, and that precision is what makes the rule’s wording the starting point of every rule-based investigation.


Strengths and limitations

Strengths

  • Highly customizable. The team encodes its own policies, thresholds, and scenarios.
  • Captures complex attack patterns that a single-indicator signature would miss.
  • Transparent. An analyst can read the rule and explain exactly why the alert fired.
  • Efficient at runtime when rules are well-written and the underlying queries are indexed.

Limitations

  • Requires deep domain expertise to design and maintain. Bad rules are worse than no rules.
  • Produces false-positives when misconfigured or when business context shifts and the rule does not.
  • Operational overhead grows with the rule library. Conflicting rules, stale rules, and duplicate rules all happen.
  • Continuous updates required as adversary techniques evolve.

Operational considerations

  • Review and test rules on a regular cadence. Quarterly is a reasonable minimum. The rule that fired well last quarter may be silent or noisy this quarter.
  • Maintain version control and lifecycle documentation. Every rule should have an owner, a documented purpose, and a history of changes.
  • Watch for conflicting rules. Two rules covering the same scenario can produce duplicate alerts, confused analysts, and lost trust.
  • Train analysts to interpret rule-driven alerts. A rule’s name and description should be enough for an analyst to know what the rule is trying to catch and what to verify.

Next up

Behavioral analytics

The final family. Multi-event sequence analysis that catches the patterns no single event would reveal.

Read behavioral analytics