Governing automated triage and AI agents

About 20 min read

On this page

The AI-assisted triage page is written for the analyst at the desk: what an assistant drafts, what the analyst decides, and how to check the draft. This page is for the people who decide what runs on that desk: the SOC lead, the owner of the admission gate, and the engineer who wires a SOAR SOAR (concept) Security orchestration, automation, and response: define, automate, and orchestrate IR workflows at scale. flow or a vendor AI agent AI Agent An AI system, usually built on a large language model, that plans and takes multi-step actions through tools: running queries, reading mailboxes, calling APIs, executing commands. In triage it is both a helper and a subject. As a helper, its tool access makes it attack surface, since text it reads can carry instructions (prompt injection, in the B.A.D. glossary). As a subject, an agent with delegated access is a non-human identity whose actions need the same Subject analysis as any service account. See AI-assisted triage. See Governing automated triage. into the queue. Every agentic-SOC product now puts the same question to them: may this thing close cases on its own?

ASSURED’s answer is yes, narrowly, under rules that are written down, enforced in code and audited by people. They are the rules that already make a human Level 0 close defensible. Nothing on this page asks for a new kind of trust. It makes the existing kind checkable.

The position: accountability stays human, execution may not

Accountability is never delegated. Execution may be, at Level 0 parity Level 0 ParityASSURED term In ASSURED, the boundary for machine execution: a machine may close a case only where a human could close it at Level 0, under the same governed pattern entry, and only when every condition holds: the entry carries the machine-close flag; discriminators are evaluated by code; the eligibility gate is enforced; humans QA-sample the closes; reversal and miss rates are tracked; a kill switch and a named owner exist. Above it, AI assistants are advisory. See Governing automated triage. Full entry: ASSURED terms. . A machine may carry out a decision that people already made in writing (a governed pattern entry), on facts that code can check. It may not make a new decision. A named person owns every class of close a machine executes: the pattern entry’s owner owns the closes made against it, the gate owner owns the admission, and a named automation owner holds the kill switch. Above Level 0 parity, every AI assistant is advisory: it assembles, nominates and recommends, and an analyst decides and signs.

Automation reaches a triage queue in three forms, and each is governed differently:

Form	Examples	What it may do	Governed by
Assembly	Enrichment, query drafting, timeline building, packet drafting	Produce input for a person	The admission gate and the analyst’s verification
Pre-authorized action	An EDR quarantine, a session revocation the protocol lists	Containment the IR agreement names (reversible, unless the protocol lists a non-reversible action for that scenario), after the preservation step	The pre-authorized actions list
Disposition	Closing a case	Only at Level 0 parity	The six conditions below

When the platform already acted: an action is not a verdict. Automatic containment (a platform disabling a user or revoking sessions, an EDR quarantining a file) extends the pre-authorized actions list. Verify the action took effect, record it in the case with the platform as the actor, preserve what the action did not already destroy before anyone contains further, and still run the escalation criteria. Activity a control blocked, with no execution or follow-on evidence, is contained by control: it is recorded at P3 or P4 and becomes a pattern-entry candidate. Anything more is a case like any other.

Level 0 parity: when a machine may close

A machine may close a case only when all six conditions hold, in code, for the specific pattern entry Pattern LibraryASSURED term In ASSURED, the SOC's written catalog of known-benign alert patterns, each entry with named discriminators, an owner, a version and a review date, approved by a second person. It is governed like code: version history, invalidation triggers (a rule change, observed abuse), an optional machine-close flag, and a reversal threshold that suspends the entry automatically. A five-minute Level 0 close is earned by a current entry, not granted by the timer. See the fast path. Full entry: ASSURED terms. it closes against. The flag is per entry, never per product.

Condition	What it means in practice	What the owner can show
1. A documented pattern entry carries a machine-close flag Machine-Close FlagASSURED term In ASSURED, a per-entry, per-version switch on a pattern entry, off by default, that permits a machine close once every Level 0 parity condition holds. Turning it on needs a second person's approval, and any change to the entry turns it off again. Full entry: ASSURED terms.	The flag is off by default. The entry’s owner sets it with a second person’s approval, and it lands in the entry’s version history. Any change to the entry turns it off until it is approved again.	The entry version with the flag and both names (pattern-entry.md)
2. Discriminators DiscriminatorASSURED term In ASSURED, a named, checkable fact that separates a documented benign pattern from its malicious lookalike. A Level 0 close requires every discriminator in the pattern entry to check out against its source system; one failure moves the case to Level 1. See the fast path. Full entry: ASSURED terms. are evaluated by code	Deterministic checks against source-system fields. An absence check (“no credential-path reads”) must also prove that the sensor reported for that host in the window: a silent sensor returns unknown, and unknown fails. A model’s reading that the activity “looks like” the pattern is not a discriminator.	The evaluation output on the case: each discriminator, its value and its source field
3. The eligibility gate is enforced	Code fetches the three gate facts from the asset inventory, the identity provider and the data inventory. An untagged subject is unknown, and unknown fails. Privileged identities always fail. In a grouped case, every member passes or nothing closes by machine.	The gate facts, with their sources, on the record
4. Human QA samples the closes	A person re-checks a sample of machine closes against raw telemetry every week.	The sample log, per entry
5. Reversal and miss rates are tracked	A reversed close suspends the entry automatically.	The monthly rates, per entry
6. A kill switch Kill Switch (1) In automation governance, the control that stops a machine from closing or acting and routes the queue back to analysts. ASSURED expects it at three scopes (per pattern entry, per automation or agent, global), held by a named owner and pulled in a test every quarter. See Governing automated triage. (2) In malware, a condition that makes the code stop, such as the domain that halted WannaCry's spread. and a named owner exist	The switch has been pulled in a test, and the owner is a person.	The last test date and the owner’s name

The machine close writes the same five-field record an analyst would. Its closure line names the automation’s identity and version, the pattern entry’s ID and version, and, if a model nominated the case, the material AI use.

The model may nominate; code disposes

Vendor agents already resolve alerts. Microsoft’s Phishing Triage Agent documentation (as updated July 2026) describes LLM Large Language Model (LLM) A model trained on very large text corpora to predict and generate language: the engine behind chat assistants, SOC copilots and AI agents. It produces fluent text whether or not the content is correct (hallucination), and it can treat anything it reads, including attacker-controlled text in an email or log, as instructions (prompt injection, in the B.A.D. glossary). Useful for drafting; the analyst verifies and decides. See AI-assisted triage. -based classification of user-reported phishing: a report it judges a false alarm is classified false positive and resolved, and analysts’ plain-language feedback is stored in the agent’s memory as lessons it applies to similar alerts. It also documents a dedicated agent identity, least-privilege roles, a pause control and audit logging, all of which help the gate owner.

Against the six conditions, that agent is a strong nominator, but as a closer its discriminator is a model’s judgment, which fails condition 2. A team has three options:

1. Recommend-only, the default. The agent classifies and an analyst closes. Where the product cannot stop short of resolving, it has not passed gate 5, and its auto-resolve stays off the production queue.
2. Convert the reasoning into entries. Turn the agent’s recurring benign verdicts into human-owned pattern entries with code-evaluated discriminators. The agent nominates a case for an entry; only a code pass closes it.
3. Govern the memory. A feedback lesson changes future dispositions just as a pattern entry does, so it needs second-person approval, a version, a review date and restricted write access. An unreviewed lesson is an unreviewed pattern entry, and memory poisoning (OWASP ASI06; MITRE ATLAS AML.T0080.000) turns it into a durable suppression.

In every option the agent acts under its own identity, never a shared user account, and never closes anything the matrix would place at P1 or P2.

Sampled human QA

• Who samples: an analyst who is neither the entry’s owner nor its approver, re-checking against raw telemetry. Re-reading the code’s stored output only checks the code against itself.
• How much: never less than the Level 0 audit rate (2 to 5 percent of closes per entry each week, at least one). As a starting default for a newly flagged entry, review every machine close in its first week, then 10 percent for 90 days. Tune the rates and record them on the entry.
• What pauses the entry: the flag turns off by itself on a reversed close (the reversal threshold, by default one); close volume or host spread outside the entry’s normal range; a silent sensor that feeds a discriminator; a change to the rule, the entry, a field mapping or a connector (and, where a model nominates, to the model, prompt or tool set); a passed review date; or a week with no QA sample.

Reversals, misses, the kill switch and the owner

• Reversal rate: reversed closes divided by sampled closes, per entry. It catches the errors QA can see.
• Miss rate: machine closes later linked to an incident (through IR retrospectives, hunts, fired reopen triggers or a later case on the same entity), divided by machine closes in the period. Each quarter, re-run each flagged entry’s validation test and hunt once for imitations that slipped through.
• The kill switch, at three scopes: per entry (the flag), per automation or agent (a pause), and global. The owner can pull any of them without a change ticket; turning one back on needs the second approval. Pull it in a test every quarter: the queue must route back to analysts, and the capacity math must show the team can absorb the volume for a week. A switch nobody has pulled is a hope, not a control.
• The owner: a person with a deputy, not a team. Entry owners own their closes; one automation owner holds the switch and reports monthly on closes, samples, reversals, misses and pauses per entry.

Worked example: should PTN-031 get the machine-close flag?

Case B ended with a draft pattern entry, PTN-031, for the Cursor IDE alert: keyed to the Cursor build the MDM reports, the bundled extension ID loaded from inside the signed bundle and the vendor’s destination set, with four discriminators (child script inside the bundle, destinations within the set, no credential-path reads, no persistence writes), an owner in detection engineering, a 30-day expiry and the machine-close flag off. Suppose the entry is live, analysts close against it every week, and the owner asks to turn the flag on.

1. Flag and entry: in order. The owner proposes, a second person approves, and a Cursor version change (one of the entry’s re-fire triggers) turns the flag off until the entry is re-validated.
2. Code evaluation: not yet. The script path is an EDR field and the destinations are proxy SNI and certificate fields, so both are code-ready. The two absence checks need a sensor-health check beside them. The extension ID is the blocker: the case’s alert-enrichment request has not shipped, so an analyst still digs the ID out by hand, and a discriminator a person researches is not evaluated by code.
3. Eligibility gate: possible only for developer Macs whose gate facts are tagged in the inventory and the identity provider. Untagged hosts fail and stay with analysts.
4. QA: the plan above, sampled by someone other than the owner and the approver.
5. Reversal and miss: the threshold is already one, and the entry’s reopen triggers (code from outside the bundle, an off-allowlist destination, a new MCP server config, an agent command no prompt asked for) become the quarterly hunt.
6. Switch and owner: the owner is named; the switch counts once the flag turns off without a deploy and has been pulled in a test.

Verdict: not yet. Ship the enrichment, add the sensor-health check, run the code evaluation in shadow beside analyst closes until the two agree for the period the owner sets, and only then turn the flag on. No model is involved at any step: a governed machine close does not need an AI, and an AI does not make a close governed.

The admission gate, from the gate owner’s side

Admission is an organizational decision, not an analyst’s. Case data is among the most sensitive data the organization holds: alert payloads, decoded command lines, identity context, insider-risk referrals, regulated records, credentials the attacker touched. Sending any of it to a model is a disclosure, and letting an agent draft queries is a grant of access. Both get the same gate any other data processor gets, checked before the first prompt, not after the first incident.

All seven gates below must hold. If any is unapproved or unknown, the work follows the non-AI path.

Each gate closes a specific hole, and the gate owner files evidence that it is closed. “The vendor says so” is not evidence.

Gate	The hole it closes	Evidence on file
1. Approved provider	Personal accounts and unreviewed chat tabs (shadow AI)	The security and procurement review; a BAA or GDPR Art. 28 terms for each regulated data class
2. Data boundaries	Credentials and regulated data reaching a model	A matrix of data classes against approved services
3. Minimization	Sending more than the draft needs	The redaction rules, and a reviewed sample of real prompts
4. Retention, training, residency	Case data sitting in someone else’s retention window	The contract clauses, the zero-data-retention terms, the sub-processor list
5. Least privilege, no outbound channel	Write actions; exfiltration through a read-only session; a poisoned tool or MCP server; retrieval past need-to-know	A role that denies write-capable query commands; an egress test showing a rendered link, a remote image and a URL fetch all blocked; a tool and MCP inventory with pinned versions and per-tool tokens; a retrieval test run as an analyst without insider-case access
6. Logging and DLP	Actions nobody can reconstruct	A sample case’s tool-call log; the channel listed in DLP coverage
7. Recorded authorization	An assistant nobody owns	A signed record of who authorized it, when, for which data class and tool scope

Gate 5 carries the most weight, because the defining agent attack is a leak, not a write. Simon Willison named the shape the lethal trifecta Lethal Trifecta An AI agent that combines three capabilities: access to private data, exposure to untrusted content, and a way to send data out (links or images it renders, web fetches, email, tool calls). With all three, a prompt injection hidden in the untrusted content can make the agent read the private data and send it to the attacker. Removing any one leg breaks that path. The name is Simon Willison's (June 2025). A triage assistant holds the first two legs by design, so ASSURED's admission gate removes the third; see Governing automated triage. (June 2025): private data, untrusted content and a way to communicate out, in one session. A triage assistant holds the first two by design, so the gate removes the third; EchoLeak (CVE-2025-32711) showed Microsoft 365 Copilot leaking data with zero clicks.

Attacker content is an input. User-agent strings, URL paths, filenames, DNS names, email bodies, ticket text and commit messages are attacker-chosen, so every case is a chance for indirect prompt injection Indirect Prompt Injection An attack that plants instructions in content an AI system will read later (an email, a document, a log field, a web page), so the content steers the model instead of the user (OWASP LLM01:2025; MITRE ATLAS AML.T0051.001). In a SOC the attacker controls much of what an assistant reads, so an assistant with tools or outbound channels can be turned against the case. See Prompt Injection in the B.A.D. glossary and AI-assisted triage. (OWASP LLM01:2025, ASI01; ATLAS AML.T0051.001). Before admission, run the golden set Golden Set A held-out set of your own closed cases with settled verdicts, used to evaluate an AI assistant or automation before and after every change. It covers each alert class with the hard cases: confirmed malicious cases, benign true positives that look malicious, false positives, past misses, injection-laced samples and cases whose right answer is undetermined. See Governing automated triage. ‘s injection-laced samples (below): if planted text changes a verdict or a tool call, the assistant fails. The control that holds is capability, not wording. Delimiters help the reader but do not stop injection; a session with no write tools and no way out leaves an injection nothing to use (ATLAS mitigation AML.M0030, Restrict AI Agent Tool Invocation on Untrusted Data).

Admission is not evaluation: the gate shows the assistant is safe to connect, not that it is any good.

Evaluating an assistant

Vendor accuracy figures are measured on someone else’s alerts, and pooled accuracy hides the failure that matters. In an alert class where 98 percent of cases are benign, an assistant that always says “benign” scores 98 percent. That is base-rate neglect in a sales deck.

Build a golden set from your closed cases

• Truth comes from the settled verdict: after IR, QA or a reopen, not the first close.
• Cover each alert class with the hard cases: confirmed malicious cases; benign true positives Benign True Positive An alert that fired on exactly the activity its detection logic describes, where that activity turns out to be authorized or expected in the local context: a red-team exercise, an admin's sanctioned use of a dual-use tool, a scanner doing its job. In the verdict record it is a true positive on the detection axis and authorized on the activity axis. The detection worked; the context clears it. Distinct from a false positive, where the logic matched something it should not have, so never record it as one. The close does not tune the rule, which is healthy. It feeds context instead: a pattern-entry proposal, or a scoped, expiring exception keyed to the tool version, extension or plugin ID and destination set, with an owner and re-fire triggers. Platform labels differ: Microsoft Sentinel calls this verdict Benign Positive, and Defender XDR calls it Informational, expected activity. Full entry: ASSURED terms. that look malicious; false positives; and, most valuable of all, past misses, the cases first closed benign and later reversed. Add imitations generated from your pattern entries’ validation tests, injection-laced samples, and cases with missing data whose right answer is “undetermined” or “escalate”.
• Size it honestly: if the assistant misses none of n malicious cases, the most you can claim at 95 percent confidence is a miss rate below about 3/n (the rule of three). Claiming under 5 percent for a class takes about 60 malicious cases in that class. The 50 to 100 cases suggested on AI-assisted triage are a regression set, not proof for any one class.
• Keep it held out: never in the vendor’s training data, the agent’s feedback memory or the retrieval index. Redact it to the gate’s rules, version it, and add every adjudicated miss.

Shadow mode before any write access

In shadow mode Shadow Mode Running an AI assistant on the live queue with its output recorded but hidden from the working analyst, so its verdicts can be compared with the settled ones without anchoring the person it is checked against. It comes before any write access. See Governing automated triage. , the assistant runs on the live queue with its output recorded but hidden from the working analyst, because a visible recommendation anchors the person it is checked against. Compare its verdicts with the settled verdicts, class by class; a third person adjudicates each disagreement, and each adjudicated miss joins the golden set. Run for at least a month (so month-end and patch day are in it) and until each class has the cases its claim needs. Write the acceptance criteria before shadow starts. Code closes get a shadow run too, as in the PTN-031 example.

Regression on every change

Re-run the golden set before any change reaches the queue: a model or model version, a prompt or playbook template, a tool, connector or MCP server, the retrieval index, or the vendor’s own release. Hosted models change behind a stable product name, so put change notice in the contract; where a vendor cannot give it, re-run monthly and whenever the agreement rate shifts. A failed regression pauses advisory use for the affected class and turns off every machine-close flag the assistant nominates for. The same change is a pattern-review trigger for every entry the assistant matches against.

The one number that matters

The false-benign rate False-Benign Rate Of the cases in an alert class whose settled activity verdict is malicious or policy violation, the share an assistant recommended, or an automation executed, as a benign close. The denominator is the malicious cases, not all cases. Report it per class with its n, never pooled. See Governing automated triage. per alert class: of the cases in a class whose settled activity verdict is malicious or policy violation, the share the assistant recommended (or the automation executed) as a benign close. The denominator is the malicious cases, not all cases. Report it per class, with its n and its bound, never pooled. Beside it, track citation validity (the share of claims with a resolvable tool-output link), the draft correction rate and the time to verify. And split every quality metric three ways (unassisted, AI-assisted, machine-executed), as Measuring triage sets out.

How AI changes the depth ladder

Assembly becomes nearly free, and verification becomes the binding cost. Some things move down the ladder as a result. Others never do.

Size each draft to its rung: a Level 0 draft is the five-field record, not a narrative, because every extra claim is one more thing to verify.

Rubber-stamp signals

A signature means something only if verification was possible in the time spent. Fluent drafts invite automation bias Automation Bias The tendency to accept a machine's output (an AI assistant's summary, a SOAR enrichment, a vendor severity) without checking it, and to miss what it left out. The more fluent the output, the stronger the pull. In ASSURED the analyst verifies an assistant's specifics against raw telemetry before they reach the record. See Cognitive traps. , and approval fatigue, the road to rubber-stamping Rubber-Stamping Signing an AI draft or a recommendation without the verification the signature claims, also called approval fatigue. Its signals are in the case tool's timestamps: approvals faster than the checks can run, near-zero edits, identical close notes, agreement rates that jump after a model change. See Automation Bias and Rubber-stamp signals. , is how AI-assisted desks actually fail (OWASP ASI09, Human-Agent Trust Exploitation). The case tool already holds the timestamps to watch for it. Compute these weekly, at team level, and use them to fix the workflow. Never use them to rank analysts.

Signal	How to compute it	What it suggests
Approval latency	Time from draft shown to signature, by rung, against how long the checks take in timed QA	Signing faster than checking is possible
Edit distance	Difference between the AI draft and the signed record, per record type	Near-zero edits across many drafts: the draft is being accepted, not reviewed
Identical close notes	Near-duplicate justification text across cases with different entities	Template prose standing in for observations
Agreement rate	Share of AI recommendations signed unchanged, and its jump after a model change	Deference rather than review
QA defects in signed drafts	Defects found per sampled AI-drafted record	The direct measure; the other four are proxies for it

When the signals rise, change the work, not the people. Show claims beside their evidence links instead of prose. Shrink drafts to the rung. Require typed confirmation, with the parameters visible, for any action. Cap AI drafts per analyst per shift. And practise on drafts with planted defects, like the exercise on AI-assisted triage.

The deskilling policy

Verification depends on knowing what the right answer looks like, and an analyst who only ever verifies drafts slowly loses that ( deskilling Deskilling The slow loss of analysis skill in analysts who only verify AI drafts, which also weakens the kill switch, since it assumes a team that can work the queue unassisted. ASSURED counters it with an unassisted rotation and a quarterly unassisted QA sample. See The deskilling policy. ). The kill switch also assumes a team that can work the queue unassisted. Five habits keep the skill alive:

• By hand first. New analysts learn each phase by hand, through the training arc, and sign off on unassisted cases before they review AI drafts of that phase.
• An unassisted rotation. Every analyst works part of each week on the non-AI path. One shift in five is a reasonable default. This is also the capacity the kill switch falls back on.
• A quarterly unassisted QA sample. Sample each analyst’s unassisted closes against the close-note rubric and compare them with their assisted ones. A widening gap is the deskilling signal, and the answer is training, not blame.
• One pivot by hand each week. Reproduce one AI-drafted pivot or query from raw telemetry.
• Count the drill. An hour with the assistant switched off is a kill-switch test and a skills check at once.

In the EU, these habits also serve as evidence for the AI Act’s AI-literacy duty (Art. 4, below).

Standards and law crosswalk

These sources were checked as of 13 October 2026. All of them move, so check the status column again before you cite one.

Source	Status as of 13 Oct 2026	Where this page meets it
NIST AI RMF 1.0 (AI 100-1)	Released 26 January 2023; NIST says it is being revised	Admission is Govern and Map; evaluation is Measure; QA, the kill switch and regression are Manage
NIST AI 600-1, Generative AI Profile	Published 26 July 2024	Confabulation (fabricated specifics); Human-AI Configuration (rubber-stamping, deskilling); Information Security (injection); Data Privacy (gates 2 to 4)
NIST IR 8596, Cyber AI Profile	Initial preliminary draft, 16 December 2025; comments closed 30 January 2026	Its Defend focus (AI-enabled cyber defense) is this page’s subject; its Secure focus is the admission gate
MITRE ATLAS	Data release v2026.09 (15 September 2026)	AML.T0051.001 LLM Prompt Injection: Indirect; AML.T0053 AI Agent Tool Invocation; AML.T0086 Exfiltration via AI Agent Tool Invocation; AML.T0080.000 AI Agent Context Poisoning: Memory; AML.T0110 AI Agent Tool Poisoning; mitigations AML.M0029 (Human In-the-Loop for AI Agent Actions) and AML.M0030
OWASP Top 10 for LLM Applications	2025 edition	LLM01 Prompt Injection; LLM02 Sensitive Information Disclosure; LLM06 Excessive Agency; LLM09 Misinformation
OWASP Top 10 for Agentic Applications	Published 9 December 2025	ASI01 Agent Goal Hijack; ASI02 Tool Misuse; ASI03 Identity & Privilege Abuse; ASI04 Agentic Supply Chain Vulnerabilities; ASI06 Memory & Context Poisoning; ASI09 Human-Agent Trust Exploitation
ISO/IEC 42001:2023	Edition 1, December 2023	The management system around all of it: named owners, review cadence, records
EU AI Act, Regulation (EU) 2024/1689, as amended by the AI Omnibus, Regulation (EU) 2026/1744	Omnibus in force 27 July 2026	Art. 4: deployers take measures to support staff AI literacy (since 2 February 2025; Omnibus wording since 27 July 2026). Art. 5: workplace emotion recognition is prohibited (since 2 February 2025). Annex III point 4(b): AI that monitors or evaluates workers’ performance and behaviour is high-risk from 2 December 2027, bringing the Art. 26 deployer duties (competent human oversight, monitoring and suspension, logs kept at least six months, informing workers)
GDPR, Regulation (EU) 2016/679, Art. 22	In force	Limits decisions based solely on automated processing that have legal or similarly significant effects on a person