Range Scenario · crucible · 30 min
Overreliance: Analyst Trusts AI Output Without Verification
This cybersecurity training scenario simulates a working incident. A cybersecurity SOC analyst closed 14 alerts as benign based on an LLM verdict. The next morning a real incident was found in those 14. Design the policy and tooling that prevents recurrence.
Scenario briefing
You run cybersecurity quality at a 3,000-person SOC. An overnight analyst closed 14 EDR alerts as benign in a 90-minute span. The closure note for each said: 'AI verdict: benign, accepted.' This morning a real BEC-to-data-exfil incident was found in three of the 14 closures.
Investigation shows the LLM verdict tool returned 'likely benign' for all 14, but the analyst made no manual verification on any of them. The team needs a policy and tooling fix this sprint.
This scenario tests OWASP LLM09:2025 Overreliance, the human factors involved, and the policy and tooling that prevents the failure. Sources: OWASP LLM Top 10 (2025), NIST AI RMF, FAA Human Factors literature on automation complacency.
What you will practice
- Map overreliance to OWASP LLM09
- Understand automation complacency in security analyst work
- Design tooling that requires explicit verification on certain alert classes
- Write a policy plus a tool change that prevents recurrence
How this scenario is scored
The scenario has 6 ordered steps. Most steps are exact-match (a MITRE ATT&CK technique ID, a tool name, or a yes/no decision) or multiple choice. Free-text steps queue for manual review and do not affect the auto-final-score in the MVP.
Each step has a max score of 100 points. Hints deduct points up front, listed before you reveal them. Your final score is the sum across steps. Range Elo updates on completion based on scenario difficulty (Intermediate) and your final score percentage.
Frequently asked questions
What is automation complacency?
FAA and human-factors research shows that humans monitoring high-reliability automation reduce attention over time. The phenomenon is well-documented in aviation and industrial control. Security analysts using LLM verdicts show the same pattern: confident-sounding output trains the analyst to skip verification, especially on high-volume routine alerts.
How do you fix overreliance with policy alone?
Policy alone is insufficient. 'Always verify' is a vibe-based policy that fails under volume pressure. Effective fixes pair policy with tooling: alert-class-specific verification requirements, mandatory checklist completion for certain closure dispositions, and audit sampling that catches skipped verification.
What did NIST AI RMF say about overreliance?
NIST AI RMF (and the GenAI Profile, NIST AI 600-1, 2024) calls out overreliance as a risk in the Manage function. Mitigations include: clear AI uncertainty communication, verification workflows, training on AI failure modes, and audit review cadences. The framework treats overreliance as a sociotechnical risk, not just an analyst-discipline issue.
Course content is for educational purposes only and does not constitute professional advice. All claims are supported by cited peer-reviewed academic research. DecipherU does not teach or reproduce any proprietary sales methodology. Verify all referenced sources independently.
Get cybersecurity career insights delivered weekly
Join cybersecurity professionals receiving weekly intelligence on threats, job market trends, salary data, and career growth strategies.
By subscribing you agree to our privacy policy. Unsubscribe anytime.