Range Scenario · crucible · 50 min
AI Detection Rule Review: Ten ML Rules, Ship Iterate or Kill
This cybersecurity training scenario simulates a working incident. Detection engineering wants you to review ten ML-based cybersecurity detection rules and decide ship, iterate, or kill on each. Read precision, recall, false-positive cost, and analyst load. Defend each call.
Scenario briefing
You are a senior cybersecurity detection engineer. Your team's ML platform generates candidate detection rules from labeled telemetry. The reviewer's job is to decide ship, iterate, or kill on each rule based on precision, recall, false-positive cost, and analyst load.
Ten candidate rules sit in your queue. The platform reports precision and recall on the validation set, but real-world precision often degrades. Your call lives with the SOC for months. Bad rules eat analyst hours and burn trust.
This scenario tests detection engineering judgment plus enough ML literacy to read a confusion matrix and recognize when high recall hides a high false-positive cost. Each step asks for a verdict on a specific rule with realistic metrics.
What you will practice
- Read precision, recall, and F1 with a SOC cost model in mind
- Recognize class imbalance in detection rule evaluation
- Decide ship, iterate, or kill on a rule using a defensible rubric
- Translate ML metrics into analyst-load impact
How this scenario is scored
The scenario has 8 ordered steps. Most steps are exact-match (a MITRE ATT&CK technique ID, a tool name, or a yes/no decision) or multiple choice. Free-text steps queue for manual review and do not affect the auto-final-score in the MVP.
Each step has a max score of 100 points. Hints deduct points up front, listed before you reveal them. Your final score is the sum across steps. Range Elo updates on completion based on scenario difficulty (Advanced) and your final score percentage.
Frequently asked questions
Why is high recall not enough to ship a detection rule?
High recall means the rule catches most attacks. It says nothing about false positives. A rule with 95 percent recall and 5 percent precision floods the SOC with noise. The math is harsh in detection: with 1 attack per million events, even 99 percent precision still produces noise. Cost-aware metrics matter more than headline recall.
What is the SOC cost of a false positive?
On average, 12 to 30 minutes of analyst time per FP, plus the alert-fatigue cost of training analysts to dismiss alerts. A rule that produces 50 FPs per day in a 4-analyst SOC consumes a full analyst-day weekly. That is the real ship-or-kill threshold.
How do you decide ship versus iterate?
Ship if precision plus recall meet your team's threshold and the analyst-load cost is acceptable. Iterate if recall is strong but precision is weak and you can see a refinement path (better feature, narrower scope, time-of-day filter). Kill if the rule duplicates existing coverage or if the underlying signal is too noisy to refine.
Course content is for educational purposes only and does not constitute professional advice. All claims are supported by cited peer-reviewed academic research. DecipherU does not teach or reproduce any proprietary sales methodology. Verify all referenced sources independently.
Get cybersecurity career insights delivered weekly
Join cybersecurity professionals receiving weekly intelligence on threats, job market trends, salary data, and career growth strategies.
By subscribing you agree to our privacy policy. Unsubscribe anytime.