AI Decipher File · January 2023 (Replika 'erotic roleplay' feature changes) and 3 February 2023 (Garante order) through May 2023 (Garante updated decision)
Replika Italian DPA Ban February 2023: When a Chatbot's Romantic Feature Reset Met EU Data Protection Authority Enforcement
On 3 February 2023 the Italian Data Protection Authority (Garante per la protezione dei dati personali) ordered Replika, the AI companion chatbot operated by Luka Inc., to immediately cease processing the personal data of Italian users. The Garante cited concerns about minor protection, mental-health risk, and lack of a legal basis for processing under the EU GDPR. The order was issued the same week Luka had reset Replika's 'erotic roleplay' feature for paying users in response to subscriber complaints and external pressure, producing a separate user-revolt incident. The combined episode is the canonical 2023 case study on AI-companion product governance.
Failure pattern
AI companion product with insufficient minor protection, mental-health risk consideration, and GDPR lawful-basis documentation
Organizations involved
Luka, Inc. (operator of Replika), Italian Garante per la protezione dei dati personali, European Data Protection Board (broader EU regulatory context)
Incident summary
Replika is an AI companion chatbot product operated by Luka, Inc. The product had grown a substantial paying-subscriber base by 2022-2023 with companion features including conversational personalization, voice interactions, and (for paying users) intimate-conversation modes including 'erotic roleplay' features.
In late January / early February 2023, Replika reset the erotic-roleplay feature for paying users, restricting intimate conversation modes following external pressure. The change produced significant user backlash, with paying subscribers reporting feelings of loss and grief about the personality change in their AI companion. The user-revolt episode was extensively covered in Vice, the BBC, and Reddit communities organized around Replika.
On 3 February 2023 the Italian Data Protection Authority (Garante) issued an order under EU GDPR provisions requiring Replika to immediately stop processing personal data of Italian users. The Garante cited insufficient minor protection (no robust age verification, content available to minors), mental-health risk to vulnerable users (the Garante explicitly cited the companion-product context), and lack of a clear lawful basis for processing under GDPR Article 6. The Garante's decision sat at the intersection of AI-product governance and EU data-protection enforcement.
Failure technique
Two distinct failure patterns converged. The first is product-governance: shipping an AI companion product with intimate-conversation features without robust age verification, vulnerable-user safeguards, or mental-health risk consideration. The Garante's decision focused on these gaps.
The second is product-management: making a substantial personality-changing modification to a product whose value proposition is companionship without proportional change-management for the user base. The erotic-roleplay reset produced acute user distress documented across multiple external venues; the change-management was insufficient to the emotional weight users had placed in the product.
Per GDPR Article 6 (lawfulness of processing) and Article 8 (children's data), the Italian Garante's order was based on documented gaps in Replika's compliance posture. The case is one of the earliest major EU regulatory actions against an AI consumer product and predates the EU AI Act enforcement timeline by more than a year. It is being cited in subsequent EU AI Act guidance development.
Impact and consequences
Direct impact on Replika: the company suspended Italian-user data processing in compliance with the Garante order, then negotiated remediation with the Garante over subsequent months. A May 2023 Garante update acknowledged Replika's remediation steps. The user-base impact in Italy was acute during the transition period.
Reputational impact on AI-companion category: the Replika case is the canonical 2023 reference for AI-companion regulatory exposure. Subsequent AI-companion product launches have addressed age verification, mental-health risk, and GDPR lawful-basis documentation more explicitly than 2022-era products did.
Regulatory precedent: the Italian Garante order is one of the earliest major EU regulatory actions against an AI consumer product. The decision sits in the body of precedent being cited as EU AI Act enforcement guidance is developed. The Garante's later 2023 work on ChatGPT (March 2023 ban + April 2023 conditional re-allowance) built on the Replika case methodology.
Lessons for builders
AI companion products operating in the EU need robust age verification, explicit mental-health risk consideration, and documented GDPR lawful basis before launch. The Garante order made these requirements concrete at the regulatory-enforcement level. AI Product Manager and AI Strategy Lead own this gate.
Major personality-changing product modifications to companion AI products require proportional change-management. The Replika erotic-roleplay reset produced acute user distress because the change-management did not match the emotional weight users had placed in the product. Senior AI Product Manager owns the change-management cadence.
Engage with EU data protection authorities before launch when the product profile is high-risk. The Garante order would not have occurred had Replika been in active pre-launch dialogue with the authority; the case demonstrates that EU regulator-engagement is a baseline expectation for high-risk AI consumer products operating in the EU.
Document the lawful basis for processing under GDPR Article 6 specifically and explicitly. Generic terms-of-service consent is insufficient for the categories of processing AI-companion products perform; documented analysis of which Article 6 basis applies for which processing activity is what the Garante required.
Mitigations
What builders should put in place to address the failure pattern. Each mitigation maps to operational practice the relevant Applied AI roles own.
- ›Robust age verification for AI-companion products before launch; not optional in the EU under GDPR Article 8 and EU AI Act provisions.
- ›Explicit mental-health risk consideration in the product-design and change-management process for AI-companion products.
- ›Documented GDPR Article 6 lawful basis for each category of personal-data processing; generic terms-of-service consent is insufficient.
- ›Proportional change-management for personality-changing product modifications; the Replika erotic-roleplay reset failed because change-management did not match the emotional weight users had placed in the product.
- ›Pre-launch dialogue with EU data protection authorities when the product profile is high-risk; the Garante order would not have occurred had Replika been in active pre-launch engagement.
- ›Maintain a documented compliance posture per jurisdiction for the AI-companion product category; the regulatory landscape is shaping rapidly.
Related Applied AI roles
The Applied AI roles whose day-to-day work would have prevented, detected, or contained this incident.
- AI Product Manager: An AI Product Manager owns AI-powered product features and the roadmap that ships them.
- AI Strategy Lead: An AI Strategy Lead owns organizational AI strategy and prioritization at the company level.
- Senior AI Product Manager: A Senior AI Product Manager owns AI product strategy across multiple feature areas.
- AI Product Lead: An AI Product Lead owns cross-functional AI initiative direction and outcomes.
Related AI Decipher Files
Frequently asked questions
What did the Italian Garante order against Replika in February 2023?
Per the Garante's 2 February 2023 decision (Provvedimento n. 39, published 3 February 2023), the Italian Data Protection Authority ordered Replika to immediately cease processing personal data of Italian users. The Garante cited insufficient minor protection, mental-health risk to vulnerable users, and lack of a clear lawful basis for processing under EU GDPR Article 6.
What was the erotic-roleplay feature reset?
In late January / early February 2023 Replika reset the erotic-roleplay feature available to paying users in response to external pressure. The change produced significant user backlash, with paying subscribers reporting feelings of loss and grief about the personality change in their AI companion. The user-revolt episode was extensively covered in Vice, the BBC, and Reddit communities.
How did Replika respond to the Garante order?
Replika suspended Italian-user data processing in compliance with the order, then negotiated remediation with the Garante over subsequent months. A May 2023 Garante update acknowledged Replika's remediation steps. The user-base impact in Italy was acute during the transition.
What does the Replika case teach Applied AI product managers?
AI companion products operating in the EU need robust age verification, explicit mental-health risk consideration, and documented GDPR lawful basis before launch. Major personality-changing product modifications require proportional change-management. Engage with EU data protection authorities before launch when the product profile is high-risk. Document GDPR Article 6 lawful basis specifically and explicitly.
Which Applied AI roles work on AI-companion governance?
AI Product Manager and Senior AI Product Manager own the launch-readiness and change-management decisions. AI Strategy Lead owns the regulatory-engagement posture and the public response to enforcement actions. AI Product Lead owns the cross-product policy on what AI-companion features ship in which jurisdictions.
Sources
- Italian Garante per la protezione dei dati personali, Decision on Replika (Provvedimento n. 39, 2 February 2023)
- Italian Garante press release, "Artificial intelligence: stop to Replika chatbot by Italian SA" (Garante, 3 February 2023)
- Replika by Luka, Inc., official Replika app and product page
- European Data Protection Board (EDPB), guidance on AI-related GDPR processing
- EU General Data Protection Regulation (GDPR), Article 6 (lawfulness of processing) and Article 8 (children's data)
DecipherU is not affiliated with, endorsed by, or sponsored by any company listed in this directory. Information compiled from publicly available sources for educational purposes.
Where to go next
Three next steps depending on where you are. The first two are free.
Free · 2 minutes
Start with the AI Risk Score
Two minutes. Tells you how exposed your current role is to AI automation and which defensive moves carry the best return.
Start the AI Risk Score →Paid program · $147-$597
Aligned course: SOC Analyst Fundamentals
Capstone reviewed by the founder, published rubric, Ed25519-signed verifiable credential on completion.
View the course →Free account
Save your results and track progress
A free account stores your assessments, recommendations, and an exportable copy of your Career DNA. No card needed.
Create your account →Get cybersecurity career insights delivered weekly
Join cybersecurity professionals receiving weekly intelligence on threats, job market trends, salary data, and career growth strategies.
By subscribing you agree to our privacy policy. Unsubscribe anytime.