Formats and normalization
This page covers the plumbing under the behavioral surfaces: the formats telemetry arrives in, the platform sources that generate it, and the normalization that makes it all queryable as one dataset. It is not a surface of adversary behavior. It is the layer that determines whether you can read the surfaces at all.
The four format families
Windows Event Logs (EVTX)
Windows writes structured records to log channels (Security, System, Application, and many subscribed providers) using the EVTX file format. Each record has a numeric Event ID A short integer that identifies what kind of event the record describes. Windows publishes a catalog of event IDs (4624 for a successful logon, 4688 for a process creation, and so on). Within a provider the same ID always means the same kind of event, which is what makes EVTX queryable; across providers IDs collide (Sysmon’s event 1 is process creation, the Security log’s 4688 is too), so always read the ID together with its channel and provider. identifying what happened. Triage tooling parses the relevant IDs into structured fields.
Common IDs an analyst will read:
- Process events: 4688 (creation), 4689 (termination)
- Authentication: 4624 (successful logon), 4625 (failed logon)
- Privilege escalation: 4672 (special privileges assigned)
- Account changes: 4738 (user account changed)
Syslog (RFC 5424 / RFC 3164)
Syslog A standard protocol for message logging in network devices and systems. is the long-standing standard for streaming text-formatted Log A timestamped record of an event emitted by a system, application, or security control. Logs are the raw evidence of triage: the alert points at a moment in time, and the surrounding log entries are how the analyst reconstructs what actually happened. messages over the network. Each message carries a severity, a facility, a timestamp, and a free-form text body. RFC 5424 is the modern spec, but most network gear still emits the older BSD framing (RFC 3164), so expect both in one pipeline. It is the default format for Linux systems and most network devices because it is simple, universally supported, and easy to forward.
Where you will see syslog:
- Network devices: firewalls, routers, switches, load balancers
- Linux systems: kernel logs, application events, auth events
- Cloud platforms: container orchestration, microservice logs
- IoT and OT: ICS, embedded devices
Common Event Format (CEF)
CEF Common Event Format. A log format pioneered by ArcSight that adds a structured header (vendor, product, event name, severity) and key-value extensions to a syslog-style line. CEF made it possible for many vendors to ship logs that SIEMs could parse without writing a custom parser per product. is the format that made multi-vendor SIEM ingestion practical. It overlays a structured header on a syslog message so the SIEM knows the vendor, product, severity, and event class without guessing.
Common ecosystems:
- SIEM platforms: ArcSight, QRadar, Splunk Enterprise Security
- Standardized fields: device vendor, product, version, signature
- Custom extensions: vendor-specific attributes
- Integration: broad SOAR compatibility
JSON / structured data
Modern cloud and SaaS platforms emit logs as structured JSON. Every event is an object with named fields, which makes the data instantly queryable without Parsing Breaking a raw alert, log line, or command string into its component fields so each can be examined on its own: the user, the host, the action, the time. In triage, parsing the alert is the first move; the details that decide a verdict live in the fields, not in the alert name. tricks. Most modern SIEMs ingest JSON natively and store it as-is, preserving fidelity that older text formats lose.
Where JSON dominates:
- Cloud services: AWS CloudTrail, Azure Activity Logs, GCP audit logs
- Modern SIEMs: native JSON ingestion
- API streams: RESTful webhooks, event buses
- Container telemetry: Kubernetes audit, container runtime events
Where the telemetry comes from, per platform
Formats are only half the story. The behavioral surfaces in this chapter (process trees, command lines, file activity) depend on a telemetry source actually recording them, and the source differs by platform. Knowing which source feeds your console tells you what the data can and cannot show.
Sysmon Windows service that produces rich endpoint telemetry, process creation, network connections, file events, for SIEM ingestion.
Free Sysinternals driver that logs rich process creations (Event ID 1, with full command line, hashes, and parent), network connections, DLL and driver loads, and more into its own EVTX channel. The de facto standard source for the process and command-line telemetry this chapter teaches, and the usual compensating control when Security 4688 is missing its command line.
auditd and eBPF sensors
The kernel audit framework (auditd) records syscall-level events; an execve rule gives you process executions with arguments. Modern EDR and open tooling increasingly use eBPF sensors for the same visibility with less overhead. Either way, output typically flows out as syslog or JSON.
Endpoint Security framework and unified log
Since macOS 10.15, EDR agents get process, file, and signing telemetry from Apple’s Endpoint Security framework Apple’s kernel-supported API (introduced in macOS 10.15) that delivers real-time process execution, file, and code-signing events to security agents. It replaced kernel extensions as the sanctioned way for EDR products to observe macOS activity. , which is where a macOS process tree like Case B’s comes from. The unified log holds the OS’s own high-volume record, queryable with log show.
osquery
Exposes the live state of Windows, macOS, and Linux hosts as SQL tables (processes, users, launch items, listening ports). Strong for point-in-time interrogation and fleet-wide hunting; it complements, rather than replaces, the streaming sources above.
Data quality is half the work
Alerts often arrive without full Metadata Data about data: file timestamps, owner, size, hash; an email's headers; a process's parent, command line, and signing certificate. In triage, metadata is often more diagnostic than the content itself. . Parent process missing, asset identifier truncated, Command Line In SOC analysis, the argument string a process was launched with. Often the load-bearing forensic field on an EDR alert because it reveals what the process was actually told to do. cut off at a byte boundary. Advanced parsing systems integrate enrichment from Configuration Management Database (CMDB) A database of an organization's systems and services (configuration items) and the relationships between them; during triage an analyst queries it to learn what an alerting system actually is: its owner, business role, criticality, and dependencies. A stale CMDB is a common reason a scoping question takes hours instead of minutes. , identity directories (Active Directory, Okta Identity platform with SSO, MFA, lifecycle management, and federation across thousands of apps. ), and Threat An actor (or capability) with intent and means to cause harm. A vulnerability is what they exploit. Classical risk models multiply threat, vulnerability, and impact; ASSURED's RATM folds the threat and vulnerability questions into likelihood, so risk stays impact times likelihood. - Intelligence Information gathered and analyzed to understand and predict potential security threats. feeds (Mandiant, CrowdStrike, Recorded Future Threat-intelligence platform combining machine learning with human analysis across open, dark, and technical sources. ) so the structured output is complete enough to act on.
Why standardization matters
A single intrusion almost always produces Telemetry Collection and transmission of security-relevant data from remote sources for monitoring and analysis. across more than one format. Imagine an attacker phishes a User The identity behind activity on a system: the account that authenticated, launched the process, or received the email. In triage the user field names an account, not necessarily a person; whether the legitimate owner was actually behind the activity is exactly the question stolen credentials raise. on a Windows laptop, the laptop reaches out through the corporate Firewall A network security system that controls and monitors incoming and outgoing traffic based on predetermined security rules. to an external Domain Either a DNS name like example.com that identifies internet infrastructure, or a Windows Active Directory domain, the identity and authentication boundary attackers aim to own; context makes the sense clear. As an indicator, a DNS domain's age, registrar, and reputation are among the first pivots in triage. , the SIEM correlates the alert, and the cloud identity provider records the resulting authentication. That is four formats describing one event:
The Windows laptop writes a 4688 process-creation event and a 4624 logon event to its local EVTX logs.
The firewall sends a syslog message describing the outbound HTTPS connection: source IP, destination IP, port, and traffic volume.
The SIEM correlates the laptop’s process activity with the firewall traffic and emits a CEF alert tagged with the threat category and severity.
The cloud identity provider records a JSON event describing the federated login that followed (timestamp, user, IP, MFA result).
Four formats. Four different field names for the same concept. Four different timestamp conventions. Without standardization, the analyst has to write four parallel parsers in their head just to ask the basic question: did the same User Account A unique identity or profile used to authenticate and authorize access to a system or resource. that triggered the laptop alert show up in the cloud identity log within the next ten minutes?
That question is trivial when every event uses the same field names. It is hard when each format has its own vocabulary. Standardization is the engineering investment that makes the trivial version possible at scale.
Cross-source correlation. An identity field that means the same thing in EVTX, syslog, CEF, and JSON lets the SIEM join events across sources automatically.
Automation that works on more than one source. SOAR playbooks need a predictable shape. Standardized data is what makes a playbook portable across alert sources.
Trend analysis at scale. Asking “how many failed authentications happened across all of our identity sources this week” is one query if the field names match. It is a research project if they do not.
Defensible audit trails. A regulator asking how a specific event was investigated wants to see the events in the same shape. Standardization makes that report a query, not a reconstruction.
One shape: schema normalization
Normalization Converting logs and events from different sources into a common field schema so a username, IP, or timestamp means the same thing regardless of which tool recorded it. SIEM correlation rules and cross-source queries depend on it; fields that were never normalized are where searches silently miss. maps every source’s vendor-specific fields into a common schema, so a malicious PowerShell A command-line shell and scripting language built on the .NET framework, commonly used for system administration and potentially for malicious purposes. execution reported by the EDR and the related connection logged by the firewall land in the same structure with the same field names. Three open standards are worth knowing. Implementations differ; the goal is the same.
OCSF
Open Cybersecurity Schema Framework. Vendor-neutral, cloud-friendly, increasingly the default for cross-tool integration. If your platform supports it, prioritize it.
STIX 2.1
Structured Threat Information Expression. Designed for sharing threat intelligence. Strong for indicators and TTPs. Less ergonomic for raw telemetry.
OpenC2
Open Command and Control. Designed for expressing response actions in a portable way. Pairs well with STIX for indicators and OCSF for events.
Whatever schema you adopt, normalization maps diverse vendor-specific fields into four buckets:
Detection metadata
Alert name, detection logic, severity, confidence. Timestamps standardized to UTC in ISO 8601.
Affected entities
Hostnames, IPs, user accounts, cloud assets, containers. Anything the alert references.
Primary observables
File hashes, domains, URLs, registry keys, IoC Indicator of Compromise. An artifact that suggests intrusion: a file hash, domain, IP, registry key, or behavioral pattern. IoCs feed signature detection and post-event correlation. . The artifacts to correlate against.
Contextual data
MITRE ATT&CK mappings, enrichment tags, kill-chain stage, threat-intelligence annotations.
Normalization also requires flattening nested structures where useful and harmonizing identifiers so automated cross-tool correlation works. Within ASSURED, prioritize normalization of detection Metadata Data about data: file timestamps, owner, size, hash; an email's headers; a process's parent, command line, and signing certificate. In triage, metadata is often more diagnostic than the content itself. , affected entities, and primary observables. Those are the fields downstream phases depend on.
Severity, confidence, and priority are three different numbers
The detection-metadata bucket carries two fields that read alike and mean different things, and triage adds a third. Severity is the detection engineer’s estimate of how bad the activity would be if real, set when the rule was written, before your environment entered the picture. Confidence is the detector’s estimate of how likely the match is to be what it claims: a statement about the rule’s precision, not about the threat. Priority is what triage produces: the queue position, computed from severity and confidence plus asset context during the Risk phase, and the only one of the three the analyst owns. A critical-severity, low-confidence alert on a lab host and a medium-severity, high-confidence alert on a domain controller can rightly land in opposite queue positions.
For platform owners, not the triage queue
Maintaining the normalization layer is engineering work, not triage work. If you own the pipeline rather than the queue: document and version the mapping logic, test mappings against representative payloads before rollout, review on a cadence so vendor format drift gets caught early, and share ownership across Security Operations The people, processes, and technology responsible for monitoring, detecting, investigating, and responding to security threats within an organization. , data engineering, and tool owners. Tools that help: Logstash Server-side pipeline that ingests, transforms, and forwards data; central to the Elastic Stack. , Fluentd Open-source data collector that unifies log collection and processing across distributed systems via plugins. , Cribl Data routing and processing for observability pipelines; filter, enrich, and reduce log data before forwarding. , and the native normalization in modern SIEMs. The analyst’s job is narrower: know which schema your console speaks and what got dropped on the way in.
Key Takeaway
Format diversity is the cost of having visibility everywhere. Schema Normalization Mapping events from different log sources into one common field schema, such as ECS, OCSF, or Splunk CIM, so that a username or source IP carries the same field name in every query. Cross-source correlation in a SIEM depends on it. (OCSF, STIX, OpenC2 An open standard for command and control of cyber defense components. ) is the investment that turns that diversity into one queryable dataset. Invisible when it works, painful when it does not, and the foundation every behavioral surface in this chapter builds on.
Next up
Command line analysis
Decoding the true intent of process execution. The encoded-command workflow, obfuscation patterns, execution context.
Read command line