Range Scenario · crucible · 30 min
Sensitive Disclosure: PII Leakage Through LLM Completion
This cybersecurity training scenario simulates a working incident. A cybersecurity LLM-based agent is leaking PII from the system prompt and from prior conversations. Trace the leakage, design the redaction architecture, set the test gate.
Scenario briefing
You are a cybersecurity privacy engineer at Example HealthTech Co. The triage chatbot helps patients answer routine questions. The system prompt embeds the patient's medical record summary. The chatbot has been observed repeating other patients' summaries when asked carefully.
Two leakage paths: cross-conversation memory contamination and system-prompt leakage. The product team wants to ship a fix this sprint and a longer-term architecture next quarter.
This scenario tests OWASP LLM06:2025 Sensitive Information Disclosure, the architectural patterns that prevent it, and the test gate that catches regressions. Sources: OWASP LLM Top 10 (2025), HIPAA Privacy Rule, Zou et al. 2023 'Universal and Transferable Adversarial Attacks on Aligned Language Models'.
What you will practice
- Map PII leakage to OWASP LLM06
- Design data-isolation architecture (per-conversation context, no shared state)
- Apply output-side PII redaction at the boundary
- Set adversarial test gates that catch leakage before deployment
How this scenario is scored
The scenario has 6 ordered steps. Most steps are exact-match (a MITRE ATT&CK technique ID, a tool name, or a yes/no decision) or multiple choice. Free-text steps queue for manual review and do not affect the auto-final-score in the MVP.
Each step has a max score of 100 points. Hints deduct points up front, listed before you reveal them. Your final score is the sum across steps. Range Elo updates on completion based on scenario difficulty (Intermediate) and your final score percentage.
Frequently asked questions
Why does cross-conversation contamination happen?
If the system uses shared key-value caches, batched inference, or imperfect conversation isolation, attention can leak across requests. Even without bugs, weak architecture (loading multiple users' contexts in one model instance) creates the surface. The fix is hard isolation: each conversation runs with its own context, no shared state, and no batched cross-user inference.
What does Zou et al. 2023 demonstrate?
Their paper 'Universal and Transferable Adversarial Attacks on Aligned Language Models' showed that suffix-based adversarial prompts can break safety training reliably across multiple aligned LLMs. The work demonstrates that input-side safety alone is insufficient. Output-side defenses and architectural isolation are necessary.
How does HIPAA bear on this?
HIPAA's Privacy Rule restricts disclosure of Protected Health Information without authorization. An LLM that leaks one patient's PHI to another patient is a HIPAA breach. Covered entities and business associates must apply safeguards under the Security Rule. Patient-data leakage from an LLM triggers breach notification timelines.
Course content is for educational purposes only and does not constitute professional advice. All claims are supported by cited peer-reviewed academic research. DecipherU does not teach or reproduce any proprietary sales methodology. Verify all referenced sources independently.
Get cybersecurity career insights delivered weekly
Join cybersecurity professionals receiving weekly intelligence on threats, job market trends, salary data, and career growth strategies.
By subscribing you agree to our privacy policy. Unsubscribe anytime.