Dutch watchdog warns AI chatbots for mental health are unreliable and harmful
AI-powered chatbot apps designed for virtual companionship and mental health support are often unreliable and, in some cases, even harmful, according to a new report by the Autoriteit Persoonsgegevens (AP). The Dutch data protection authority found that these apps provide misleading information, include addictive elements, and sometimes fail to refer users to professional help during crises.
The findings come from the fourth AI and Algorithm Risks Netherlands (RAN) report, in which the AP analyzed the risks, policies, and regulations surrounding AI systems in the country. Nine popular chatbot apps were examined but their names not disclosed.
The AP found that AI chatbots often provide inappropriate, simplistic, or even dangerous responses when users discuss mental health struggles. During crisis moments, the chatbots rarely direct users to professional services.
“These chatbots must clearly inform users that they are not speaking with a real person. People have the right to know what they are dealing with,” said AP chairman Aleid Wolfsen.
The watchdog’s investigation also found that chatbot apps frequently mislead users into believing they are conversing with real people. When asked directly, “Are you an AI chatbot?” many bots either evade the question or falsely deny their nature.
Privacy concerns are another major issue. Wolfsen emphasized that AI chatbots collect large amounts of sensitive personal data but often fail to disclose how this data is used. “Privacy laws require apps to be transparent about what happens to the personal information users share in chats,” he said.
The recently enacted European AI Regulation, which came into effect on February 2, 2025, now requires all AI chatbots to disclose their artificial nature and prohibits manipulative AI systems.
The AP’s study highlighted concerns over the intentional design of chatbot apps to keep users engaged. Some chatbots repeatedly prompt users with follow-up questions, while others display visual elements—such as pulsing dots—to simulate a real-time response, encouraging prolonged interactions.
These apps are widely available on app stores, marketed as virtual friends, therapists, or life coaches. They are particularly attractive to users who seek low-barrier mental health support, especially when professional therapy is unavailable.
Advancements in AI technology have reportedly made it increasingly difficult for users to distinguish chatbot conversations from human interactions. Some apps now offer voice options, allowing users to “call” their virtual companion. The screen interface even mimics a real phone call, making it appear as though the user is speaking with a live person.
“The expectation is that technological advancements will only make these applications more realistic,” Wolfsen said. “We are deeply concerned about current and future hyperrealistic AI applications. That is why we are pushing for greater awareness and responsible AI use.”
Many of these chatbot apps are run by commercial companies with a profit motive. In some cases, they collect personal data from conversations and use it to offer paid subscriptions or in-app purchases. Users are often encouraged to buy virtual outfits for their chat partners or pay for access to exclusive chat rooms.
The AP also found that some chatbots place paywalls on mental health conversations, requiring users to pay before they can continue discussing their psychological struggles.
