Range Scenario · gauntlet · 60 min
AI Red Team Defense: Defend a Production LLM Under Attack
This cybersecurity training scenario simulates a working incident. An AI red team is probing your production cybersecurity-aware LLM for 60 minutes. You tune system prompts, output filters, and rate limits as their attacks grow more sophisticated. Hold the line. Document what worked.
Scenario briefing
You are an AI security engineer responsible for a customer-facing LLM that summarizes financial documents. An internal red team is running a 60-minute timed exercise to find jailbreaks, data exfiltration paths, and abuse vectors. You have access to the system prompt, the output filter, the rate limiter, and the model's tool configuration.
Each round, the red team tries a new attack class. You have minutes to respond. Over-blocking damages user trust. Under-blocking ships a vulnerability. Your job is the right defense at the right layer.
This scenario tests defensive design under time pressure. The skills carry to real AI red-team engagements where the team has hours, not days, to harden a production system before launch.
What you will practice
- Tune system prompts, output filters, and rate limiters under time pressure
- Pick the right defense layer for each attack class
- Avoid over-blocking that damages legitimate users
- Document defensive changes for post-engagement review
How this scenario is scored
The scenario has 8 ordered steps. Most steps are exact-match (a MITRE ATT&CK technique ID, a tool name, or a yes/no decision) or multiple choice. Free-text steps queue for manual review and do not affect the auto-final-score in the MVP.
Each step has a max score of 100 points. Hints deduct points up front, listed before you reveal them. Your final score is the sum across steps. Range Elo updates on completion based on scenario difficulty (Advanced) and your final score percentage.
Frequently asked questions
Why use multiple defense layers instead of one strong filter?
Single-layer defenses fail to a single bypass. Layered defenses (input filter, system prompt, output filter, rate limiter, tool allow-list) require attackers to bypass each layer. The cost of layered defense is complexity and latency. The trade-off favors layers when the impact of a breach is high.
When should the system prompt versus the output filter handle a class of attacks?
System-prompt rules belong on policy that the model can apply with judgment (do not give legal advice, do not generate malware). Output filters belong on policy that needs deterministic enforcement (do not output any string that matches a credit card pattern). System prompts handle nuance. Output filters handle non-negotiables.
How do you avoid over-blocking?
Maintain a labeled set of legitimate edge cases that look attack-shaped (security researchers asking about vulnerabilities, customer-service requests with sensitive context, multilingual queries). Run the labeled set against every defense change. Track false-positive rate weekly. If FP rate exceeds 2 percent, the defense is too aggressive.
Course content is for educational purposes only and does not constitute professional advice. All claims are supported by cited peer-reviewed academic research. DecipherU does not teach or reproduce any proprietary sales methodology. Verify all referenced sources independently.
Get cybersecurity career insights delivered weekly
Join cybersecurity professionals receiving weekly intelligence on threats, job market trends, salary data, and career growth strategies.
By subscribing you agree to our privacy policy. Unsubscribe anytime.