Italy's data protection watchdog Garante has levied a €5 million ($5.64 million) fine against Luka Inc., the developer behind AI chatbot Replika, marking another significant enforcement action in the European Union's ongoing scrutiny of artificial intelligence platforms.
Replika, launched in 2017 in San Francisco, offers users personalized AI avatars that engage in conversations and are marketed as improving emotional wellbeing. The company generates approximately $2 million in monthly revenue from premium features such as voice chats, while offering basic services for free.
The Italian regulator initially ordered Replika to suspend operations in Italy in February 2023, citing specific risks to children and emotionally vulnerable individuals. Following a thorough investigation, Garante determined that Luka Inc. violated EU privacy regulations on two critical fronts: it lacked a valid legal basis for processing users' personal data and failed to implement any age verification mechanisms to prevent minors from accessing the service.
Particularly concerning to regulators was how the app encouraged users to share sensitive personal information in what it portrayed as a "safe space," potentially manipulating vulnerable users into disclosing more than they typically would. While Replika's terms claim the service is not intended for users under 18, investigators found no effective measures to enforce this restriction.
In addition to the financial penalty, Garante has launched a separate investigation to assess whether Replika's generative AI system complies with European Union privacy rules, specifically regarding how its language model was trained. This represents part of a broader pattern of enforcement by the Italian authority, which has emerged as one of the EU's most proactive regulators in the AI space.
Last year, Garante fined OpenAI, the maker of ChatGPT, €15 million after temporarily banning the popular chatbot in Italy over alleged breaches of EU privacy rules. These actions highlight the increasing regulatory challenges facing AI companies as they navigate the complex landscape of European data protection laws, particularly as the EU AI Act begins its phased implementation through 2025 and beyond.