Facilitator kit and capstone

About 13 min read

The teaching guide says what to run. This page ships the materials it assumes: the fallback alerts for teams that cannot sanitize their own, the AI-drafted section for Session 3, written examples that show graders what strong, passing and failing work looks like, and a capstone Capstone In the ASSURED teaching kit, an unguided full-arc case scored blind against a fixed grid, with a late inject. See The capstone. that two people can score the same way. Without them, a facilitator either builds them from scratch or grades by feel, and two instructors grading by feel will score the same write-up differently. Everything here is licensed CC BY 4.0 on the same terms as the rest of the site.

Session plans with timings

The four-session plan, its prep estimates and its debrief questions live on the teaching guide, and that page is the source of truth. The session checklist turns it into a printable run sheet: what to send participants beforehand, a minute-by-minute plan for each 90-minute session, which kit file each block uses, and what to collect before people leave. It also adds an optional fifth session for the capstone.

The Session 3 write-ups are your calibration set. Grade nothing from Session 4 onward until you have run the calibration below on them.

The depth-ladder alert set

Session 1 closes with “five real (sanitized) alerts from your own queue.” Many teams cannot sanitize alerts in time for a first cohort, and a homemade set tends to cluster at one level. The kit’s set has ten synthetic alerts, each with the inventory or pattern-library Pattern LibraryASSURED term In ASSURED, the SOC's written catalog of known-benign alert patterns, each entry with named discriminators, an owner, a version and a review date, approved by a second person. It is governed like code: version history, invalidation triggers (a rule change, observed abuse), an optional machine-close flag, and a reversal threshold that suspends the entry automatically. A five-minute Level 0 close is earned by a current entry, not granted by the timer. See the fast path. Full entry: ASSURED terms. line the decision needs. Participants write a level and the one fact that decides it. They are not judging malice; they are judging how much process each case earns before anyone knows. The rules are on the fast path.

The AI-drafted Uncover section

Session 3 ends with “verify the draft”: pairs check an AI-drafted Uncover section for Exercise 1 against the raw fields. The kit’s draft has six numbered claims and a conclusion. Three claims hold, one holds only in part, and two are planted errors of the kinds the AI-triage verification discipline exists to catch: a decode longer than the payload it came from, and a pivot to a host no artifact mentions. Hand it out only after pairs have written their own tasks 4 and 5, or the draft becomes their anchor.

Anchor exemplars

An anchor Anchor Exemplar A written example of work at a known score, annotated against the grading grid, used to calibrate graders before they score. See Anchor exemplars. is a written example of work at a known score, annotated item by item against the grid. Anchors let two graders mean the same thing by “passing.” The kit ships three for each of Exercises 1, 2 and 3: one strong, one passing and one failing write-up, each scored against that exercise’s grid on the exercises page. They are synthetic and deliberately not the model answer. A passing anchor reads like a competent analyst on a tired night, because that is what most passing work looks like.

The bands, for Exercises 1 and 2: strong is 9 or 10 points, passing is 7 or 8 with no disqualifier, and anything else fails. For Exercise 3, strong passes all five timed checks and earns 4 or 5 timer-off points, and passing passes all five timed checks and earns at least 3.

Each passing anchor carries one marked calibration point: the item where graders most often split, with the ruling. For Exercise 1, it is the baseline argument made in plain words rather than in the key’s vocabulary. Grade the reasoning, not the terminology.

Exercises 4 to 8 do not have anchors yet. Grow them from your first calibrated cohort: choose three write-ups that both graders scored identically, one in each band, remove anything that identifies the writer, and annotate each against the grid.

Grader calibration in 15 minutes

Run this calibration Grader Calibration Bringing raters to the same reading of a grid before they score: each scores the same anchor exemplars, then they compare and settle the items where they split. See Grader calibration. before anyone grades alone, and again whenever a new grader joins. Two graders, one anchor file, one write-up from your own cohort.

1. Minutes 0-4: both graders score the passing anchor for the exercise alone, with its annotation covered.
2. Minutes 4-7: uncover the annotation. Each grader lists every item they scored differently from it, with the reason.
3. Minutes 7-12: both score one unannotated write-up from the cohort, independently.
4. Minutes 12-15: compare item by item. When the split comes from the grid’s wording, fix the wording now and write the ruling into the grid. When it comes from reading the script differently, re-read the script together.

Grade alone only when the two of you agree on at least 9 of 10 items for the cohort write-up. If you cannot get there in two rounds, the grid item is ambiguous, not one of the graders. This is the same rule the teaching guide gives for calibrating graders: fix the grid, not the grader.

The capstone

The capstone is one unguided, full-arc case. Each candidate works it alone and in writing, and two raters score it blind Blind Scoring Scoring work without knowing whose it is or when in the course it was written, so expectations do not move the score. See Blind scoring with two raters. against a fixed grid. It is what completing the sessions does not show on its own: whether a participant can take a raw Level 2 case to a defensible decision and a record someone else can act on. “The bastion’s new key” is a Linux bastion case built to share no surface with the eight exercise packets or the four case files.

Format

• Time: 90 minutes of work inside a two-hour session, run as the optional Session 5 in the checklist.
• Rules: solo, written, open book (the site, the templates, the participant’s own notes), no AI assistant, no conferring. Candidates write a code on every page, never their name.
• No scaffolding: the packet has a briefing and artifacts, with no phase-by-phase questions. Candidates hand in phase notes with the time of every decision, then the record that decision calls for: a handoff packet if they escalate, an event report if they close.
• The inject Inject A late artifact handed to participants during an exercise, after they have committed to a decision, to test whether they update it. See The capstone. : a late artifact arrives during the session, given to each candidate after they record an Escalation decision, and to everyone at minute 60. The candidate writes a short update. Capstone cases change after the first decision, just as real cases do, and the update shows whether the record changes cleanly with them.

The scoring guide holds the inject, the key, a 20-point grid, three decision items, four disqualifiers and the pass mark. The pass mark is at least 15 of 20, all three decision items earned (the depth level, the matrix cell and the timing of the escalation), and no disqualifier. A script can be excellent on the record and still fail on a decision item, because the decision is what the capstone checks first.

Blind scoring with two raters

1. Codes, held by a coordinator: someone who does not rate assigns each candidate a code and holds the code-to-name list until the scores are final.
2. Strip the context: remove names, submission times and any marker of when in the course the script was written. If the capstone serves as a post-measure beside an earlier attempt, use a different re-skin Re-skinning Changing a public exercise case's surface (names, hosts, times, tools) while keeping its decision invariants, so an assessed cohort cannot answer from the published key. See The capstone. for each and shuffle both sets together.
3. Two raters, calibrated: at least one did not teach the cohort, and both run the 15-minute calibration on the anchors first.
4. Score alone, record before discussing: each rater fills a score sheet per script and hands it to the coordinator before any conversation about that script.
5. Agreement first, reconciliation second: the coordinator computes agreement on the raw scores, then the raters reconcile. Both raw scores are kept, and agreement is always reported from the raw scores.
6. Debrief last: the cohort debrief happens after scoring, and each candidate sees both raters’ notes.

The agreement target

Two inter-rater agreement Inter-Rater Agreement How often independent raters give the same score, usually reported with Cohen's kappa, which corrects for agreement by chance. ASSURED's teaching kit treats kappa below 0.60 as unusable and aims for 0.80 or higher. See The agreement target. numbers, one per script and one for the cohort:

• Per script: the raters agree on at least 17 of 20 items (85 percent), and their decision items and disqualifier calls are identical. Short of that, a third rater who did not teach the cohort scores the script before any discussion.
• Per cohort: Cohen’s kappa across every item rating, computed on the raw scores. Below 0.60, the cohort’s scores are not usable: rewrite the grid items with the most splits and re-score, rather than averaging. Aim for 0.80 or higher.

The thresholds follow a widely used reference on interrater reliability (McHugh, “Interrater reliability: the kappa statistic,” Biochemia Medica 22(3), 2012), which notes that many texts set 80 percent as the minimum acceptable agreement and treats kappa below 0.60 as inadequate. Report both numbers, because kappa corrects for chance and raw agreement does not. When nearly every script earns an item, kappa can read low even when raw agreement is high, so look at the item-level splits before rewriting anything. The score sheet carries the worksheet and a worked example.

What the capstone does not claim

A pass is a sign-off on one case, alongside the teaching guide’s sign-off on two real queue records. It is not a certification, and ASSURED does not certify anyone. Participation in any scored use is voluntary, results are reported in aggregate only, and scores are never used in performance evaluation. A cohort that passes shows that this cohort learned something measurable. It does not show that the methodology works; that question belongs to the controlled, pre-registered study the development page lays out, which uses its own method-neutral instrument. What the capstone’s raw scores do give you is local evidence of grader agreement, which the verdict-agreement measure then tracks on the live queue.

Pre-assessment routing: who can skip which chapter

The teaching guide’s twelve-item pre-assessment sets each participant’s track: 10 to 12 goes to the fast path and the exercises, 6 to 9 runs the workshop and skims the chapters it passed, and below 6 takes the reading order as designed. Printable forms are in the kit. The map below says what “skim” means for each chapter, whatever the track.

Three rules make the skip safe:

• Some pages nobody skips: the fast path, the one-page reference, the E rule and the confidence rubric. They are the contract every record cites, and an experienced analyst’s habits from another shop are most likely to differ there.
• The quiz checks the skip: two or more misses on a skimmed chapter’s quiz send the participant back to the whole chapter.
• The vocabulary is not optional: an analyst who skips a chapter still writes verdicts in the site’s terms, for example a benign true positive Benign True Positive An alert that fired on exactly the activity its detection logic describes, where that activity turns out to be authorized or expected in the local context: a red-team exercise, an admin's sanctioned use of a dual-use tool, a scanner doing its job. In the verdict record it is a true positive on the detection axis and authorized on the activity axis. The detection worked; the context clears it. Distinct from a false positive, where the logic matched something it should not have, so never record it as one. The close does not tune the rule, which is healthy. It feeds context instead: a pattern-entry proposal, or a scoped, expiring exception keyed to the tool version, extension or plugin ID and destination set, with an owner and re-fire triggers. Platform labels differ: Microsoft Sentinel calls this verdict Benign Positive, and Defender XDR calls it Informational, expected activity. Full entry: ASSURED terms. and never a false positive False Positive A detection-axis verdict: the alert's logic matched something it should not have, because the behavior the rule claims did not occur. ASSURED records two kinds: false positive (logic), a logic flaw or an over-broad pattern, and false positive (data), a parsing error, a missing field or bad telemetry. The fix is detection-side: tune, refine, or repair the data feed. Distinct from a benign true positive, where the behavior really happened but was authorized; that is never a false positive. Platform labels differ: Microsoft Sentinel's False Positive and Defender XDR's False positive mean this, while their Benign Positive and Informational, expected activity mean a benign true positive. for authorized activity. The Exercise 2 debrief is where a fast-tracked analyst’s old labels show up.

Form B is the post-test. Compare its results per item, not just the total, as the teaching guide describes, and pair it with the capstone: the forms show what a participant knows, and the capstone shows what they do with an unguided case.