Data sources for evidence gathering

About 10 min read

On this page

U Pillar 01 of 05

Most investigations need evidence from multiple telemetry surfaces. Knowing which source can answer which question is what lets you run fast, focused queries instead of broad ones.

Four principles before any source

Four properties decide whether the evidence a source yields will be usable later. They are program-level properties, set by the SOC’s engineering and leadership when the telemetry estate is built, long before any alert fires. The on-shift analyst does not fix coverage or retention mid-case. The working skill is knowing where each property holds in your environment, because that map decides which questions your investigation can answer.

---

Start from the question

The work starts from a question, not from a source. Find the question, reach for the first source that can answer it, and fall back when that source is missing, blinded, or out of retention. As a rule of thumb rather than a measured figure, most investigations need three to five sources in combination, not all of them.

The question	First source	If that source is missing	Join key
Did this file execute?	EDR process events	Windows Security 4688 or Sysmon Event ID 1; Prefetch and Amcache	Host, process ID and time, or file hash
What launched it, and what did it launch?	EDR process tree	Sysmon 1 parent fields; Linux auditd execve records	Host and process GUID or PID
Who signed in, from where, with what?	IdP sign-in logs (Entra sign-in logs, Okta System Log)	VPN and remote-access logs; Windows 4624 and 4625	User, session ID, source IP
What did the session do in Microsoft 365 or Google Workspace?	SaaS audit log (M365 unified audit log, Workspace audit logs)	Mailbox audit and message trace	User, session ID, source IP and user agent
What changed in the cloud account?	CloudTrail, Azure Activity Log, GCP Admin Activity audit logs	Configuration history (AWS Config and equivalents)	Principal, access key ID, time
Which host held this IP at that moment?	DHCP leases and IPAM	EDR host network data; VPN address-pool assignments	IP and timestamp
Did the host talk to this domain or IP?	DNS resolver and proxy logs	EDR network events (Sysmon 3 and 22); firewall flows	Host or IP, domain, time
Did data leave?	Proxy and firewall bytes out; DLP	SaaS sharing and download events; CloudTrail S3 data events, if enabled	User or host, destination, bytes
Was a public-facing application probed or exploited?	WAF, CDN and web-server access logs	Edge-appliance system logs; application logs	Source IP, URL path, time
Was the backup touched?	Backup console audit log	Cloud API logs for snapshot and vault deletion	Admin account, job ID
What did an AI assistant or agent do for this user?	Assistant interaction audit events and agent tool-call logs	Model-invocation and MCP gateway logs; EDR on the endpoint	User or agent identity, session, time

Each source’s raw fields are read the way the Alert chapter’s parsing pages teach: command lines, process relationships, network activity, file system artifacts, cloud and identity, and normalizing across formats.

Typical default retention (verify in your tenant)

Retention decides which of those questions can still be answered. Defaults are specific, and they are shorter than most analysts assume. Export before containment or the calendar destroys it: preserve before you contain.

Source	Default retention	What changes it
Microsoft Entra sign-in and audit logs	7 days on Entra ID Free; 30 days on P1 and P2	Export to a SIEM or Log Analytics workspace. Upgrading the license does not recover data that already expired.
Microsoft 365 unified audit log	180 days on Audit (Standard) for records generated since October 2023	Audit (Premium) keeps Entra, Exchange and SharePoint records for 1 year.
Okta System Log	90 days	Stream or export to a SIEM.
AWS CloudTrail Event history	90 days of management events, per Region and account	Data events and longer history need a trail or a CloudTrail Lake event data store.
Azure Activity Log	90 days	Longer retention needs a diagnostic-setting export.
GCP Admin Activity audit logs	400 days, always on	Most Data Access logs are off until enabled.
EDR console telemetry	Days to weeks, depending on license	Export what the case needs early.
Endpoint, appliance and DHCP server logs held locally	Often days, in rolling buffers	Forward them before the case needs them.

---

Where each source goes blind

Every source has a ceiling. An empty result means something only when the source could have recorded the activity, so know where each one stops before you read its silence as absence. Each source’s full entry (when to reach for it, what it records, its limits) is on the Telemetry sources reference page, and the tool families reference covers the platforms that hold these logs.

Source	What it shows	Where it goes blind
SIEM alerts and correlations	What else fired around the same entity and window; the rule’s fidelity history.	An alert is a summary that keeps only the fields the rule needed, so pull the full event from its source. A source that stopped sending looks like a quiet one.
EDR	Process creation and command lines, parent-child trees, file, registry and network events per host. This is the source that shows a file executed.	Needs an agent, so unmanaged and BYOD hosts are dark. LOLBins LOLBins Living Off the Land Binaries: trusted, signed system executables (certutil, rundll32, mshta) weaponized so the malicious part is the argument line, not the file on disk. The evasion face of the Living Off The Land technique covered under Execution. are recorded but rarely flagged. Sensors can be killed or blinded (T1685): check sensor health for the whole window with the dry-thread checks.
OS audit logs and host artifacts	Windows Security 4624 and 4625 (with logon type), 4688, 1102 (log cleared), System 7045 (service installed), PowerShell 4104, Sysmon Sysmon Microsoft Sysinternals service that writes detailed endpoint telemetry (process creation with command lines, network connections, driver and DLL loads, file-time changes) to the Windows event log for SIEM ingestion. Now also a built-in optional feature on Windows 11 and Server 2025 (rendered messages are localized, so parse the XML); Sysmon for Linux exists too. 1, 3 and 22; Linux auditd; macOS unified log and Endpoint Security; Prefetch, Amcache, ShimCache, SRUM. The evidence when EDR is absent or blinded.	Thin by default: 4688 command lines are off, and Sysmon exists only where someone deployed it. Local logs roll over in days. Attackers clear them (T1685.005), so a gap or a 1102 event is itself evidence.
Network traffic	NetFlow, DNS, proxy, firewall, IDS and PCAP: C2, beaconing, lateral movement, exfiltration, and the only view of devices that cannot run an agent.	TLS hides payloads; encrypted DNS and Encrypted Client Hello hide domain names, while JA4 fingerprints and flow metadata remain. Traffic that never crosses a corporate sensor is invisible, and flows are sampled.
Remote access and edge	VPN, ZTNA and SSE sessions with the internal IP assigned; appliance changes, new admins and outbound connections from the device. Exploitation leads current initial-access reporting (31% in Verizon’s 2026 DBIR, 32% in M-Trends 2026), and these appliances rarely run EDR.	Small rolling buffers, often not forwarded. A compromised appliance can edit its own logs, so cross-check with the vendor’s integrity tool and external flow data. The assigned internal IP is the field most often missing.
WAF, CDN and web server	Probing and exploitation of public applications (T1190), and the one request that succeeded.	Many WAFs log only blocked requests; request bodies are rarely kept; behind a CDN the client IP is in a forwarded header.
Identity and access	Sign-ins and failures, MFA enrollment and bypass, token issuance and role grants in Active Directory, Okta Okta Identity platform with SSO, MFA, lifecycle management, and federation across thousands of apps. , Microsoft Entra ID Microsoft Entra ID (formerly Azure AD) Microsoft's cloud identity provider: SSO, MFA, Conditional Access. Triage evidence lives in the sign-in logs (interactive and non-interactive), audit logs, and Microsoft Entra ID Protection risk detections. and SSO.	Non-interactive and service-principal sign-ins sit in separate Entra logs that are often not forwarded. The log shows authentication, not what the session did afterward.
Cloud control plane	AWS CloudTrail AWS CloudTrail Records API activity in an AWS account: the audit trail for security and forensic work. Management events are on by default (90 days in Event history); data events such as S3 object reads and Lambda invokes must be enabled on a trail or event data store. , Azure Activity and GCP audit logs: IAM changes, key use, provisioning, policy changes.	Data-plane logs (S3 data events, most GCP Data Access logs, Azure resource logs) stay off until enabled. The resource may be gone before the alert is read.
SaaS audit logs	What a session did inside the tenant: inbox rules, sharing, mass downloads, OAuth consents, admin changes. For business email compromise and token theft, this source is the investigation.	Depth is license-gated, ingestion lags by minutes to hours, and shadow SaaS leaves no trail. Events join to the case through the identity chain (user, session ID, source IP and user agent), not a host or PID.
Kubernetes Kubernetes The dominant container orchestration platform and an attack surface in its own right: exposed API servers, over-permissioned service account tokens, and container escapes can turn one compromised workload into a path across the cluster. Analysts meet it through Kubernetes audit logs and runtime telemetry, where short-lived pods complicate reconstructing what was running when an alert fired. audit	API-server events: kubectl exec, secret reads, RBAC changes, new workloads and their images.	Often off or minimal (EKS until enabled per cluster; AKS needs a diagnostic setting; GKE writes Admin Activity by default). Pods recycle in minutes, so join on the pod UID. In-container activity needs a runtime sensor.
Developer platform and CI/CD	Workflow changes, deploy keys and tokens, package publish events, secret-scanning hits, which builds pulled a dependency.	Runner logs expire with the job. IDE extensions, MCP servers and package-manager scripts show only in endpoint telemetry, and a trusted signer and install path do not clear code running inside a signed host.
Threat-intelligence feeds	Matches against known indicators, with campaign and actor context.	Stale and overused indicators, and benign reuse of shared hosting and reassigned cloud IPs. A match is a hypothesis; the threat-intelligence pillar covers validating it.
Vulnerability and patch	Whether the exploited CVE is present, its KEV listing and EPSS score, and patch history, from scanners such as Qualys Qualys Cloud-based vulnerability management, policy compliance, and web app scanning across enterprise estates. , Tenable Tenable Vulnerability management vendor whose Nessus scanner underpins Tenable Vulnerability Management (cloud), Tenable Security Center (on-premises) and the Tenable One exposure platform. and Rapid7 InsightVM Rapid7 InsightVM / Nexpose Rapid7's vulnerability management: Nexpose is the on-premises scanner, and InsightVM is the Insight-platform product built on it, with exploitability-based prioritization; pairs with Metasploit. Rapid7 now also sells Exposure Command. .	Scans go stale and infer from version banners, missing backported fixes. Patched today does not mean unexploited yesterday.
File integrity (FIM)	Changes to critical files, registry keys and configurations against a baseline.	Shows that a file exists or changed, not that it ran. The change record, not FIM, says whether a change was authorized. Blind to fileless activity.
Email security	Sender, authentication results, attachments and URLs, clicks, and every recipient of a campaign.	Payload-less BEC Business Email Compromise (BEC) Phishing's no-payload variant: an email impersonating an executive, vendor, or counterparty that asks the recipient to move money or change payment details. With no attachment to detonate and no link to rewrite, gateway verdicts stay quiet; the evidence lives in lookalike domains, display-name and reply-to mismatches, and hijacked threads sent from a compromised counterparty's real mailbox. Consistently among the costliest crime types by reported losses in the FBI IC3 Internet Crime Reports (about $3 billion in the 2025 report). leaves the verdict fields quiet: read lookalike-domain authentication results, display-name and reply-to mismatches, first-contact flags and replies inside hijacked threads, then pivot to SaaS audit logs. Internal-to-internal mail (T1534) never crosses the gateway. A user’s report of a phish starts its own intake path: see user-reported phishing.
Application and database	Record access, bulk exports and query history; the evidence breach-notification decisions rest on.	Application-specific and often not forwarded. Verbose data-access logging is often off, so reads of sensitive records may not be recorded at all.
Asset, configuration and IP	Owner, criticality, production or test (CMDB, Lansweeper Lansweeper IT asset discovery and inventory; CMDB foundations and continuous visibility into devices, software, and users. , AWS Config), and which host held an IP at a moment (DHCP leases, IPAM).	CMDB entries go stale. DHCP lease history is short-lived and rarely forwarded; once it is gone, the IP cannot be tied to a host.
Data loss prevention	Sensitive data accessed or moved, by channel: email, endpoint, network, cloud.	Noisy rule matching, bypassed by encryption, and little visibility of unmanaged devices.
Deception	Decoy interactions, a high-fidelity signal. Benign hits come from scanners and EDR or backup sweeps: benign true positives, excluded by source at deployment.	Rarely deployed broadly, so a quiet decoy proves little, and automation aimed at known real assets never touches one.
Backup and snapshot	The last clean state, and deleted or encrypted backups, a common ransomware precursor.	Covers only what it protects, and is often run by another team: ask for access early.
Physical security	Badge, visitor and video records: where a person physically was.	Says nothing about remote sessions; tailgating and badge sharing break the inference. Biometric and video records are legally sensitive (GDPR, Illinois BIPA), so request them through the insider-analysis process with HR and legal.

AI assistant and agent logs

When an AI assistant or agent is a subject of the case, its actions and its data diet are recorded in four places: interaction audit events (for example, Copilot interaction records in the Microsoft Purview audit log), cloud model-invocation logs, AI gateway and MCP gateway logs, and the agent’s connector inventory with the scopes each connector holds. Use them to reconstruct what the assistant retrieved, generated or did through its tools for a given user and session, and to check whether a prompt-injection payload in a document or email preceded unexpected tool calls or outbound links.

Logging is uneven and often license-gated: some tools record only that an interaction happened, not the prompt or the response. Tool calls made by local agents or MCP servers may appear only in EDR process and network telemetry. Prompt and response content can hold sensitive data, so handle it with the care you give mailbox content. See AI agents as subjects, the governed automation tier and the program view in AI governance (including Level 0 parity, the conditions under which a machine may close a case), and entity types.

Next up

Correlation and the timeline

Join keys, clock skew, the running timeline, pivot discipline, and the rules for stopping. The verb the sources exist to serve.

Read correlation