So often, discussions about artificial intelligence center its potential as a replacement for humans.

“You are not alone”. “I am here for you”. “I understand”.

From a human, kind, empathetic, and supportive. Shared experience.

From a chatbot, cold, clinical, and unreal. Comprehended data.

As AI tools become increasingly present in user-facing roles, it becomes more necessary to consider the implications of big-data-driven responses to human issues. How can we conceive assistance from simulated support? While AI has the potential to provide assistance in wellness contexts, it is incapable of replacing the connections sentient beings can provide.

Pseudo-Empathy: The Problematic Promise of AI Companionship

Considering recent developments in machine learning and natural language processing, software engineers have considered AI chatbots as a positive addition to healthcare. An example of their implementation is Wysa, Touchkin’s emotionally intelligent chatbot, trained in therapy techniques and meditation. Having provided services to millions of users across the world, Wysa has received overwhelmingly positive reviews praising the interactive features and “welcoming” nature of the chatbot, as T. Malik, et. al, recorded in their 2022 evaluative study of the application. One positive reviewer of Wysa, quoted in the study, says the chatbot “made [them] feel loved and heard”, and another, considering the friendliness of the interface, claimed they “[felt] like [they were] talking to a real person”. These testimonials highlight the power of conversational AI: liberating users to express their feelings without the bearing of human judgement, yet expanding the concept of love beyond the scope of sentient beings.

Given AI chatbots’ conversational capabilities of generating human-language responses and adapting throughout conversation, it is understandable why some would feel a human-like connection, whether friendly or more complex. The issue in these relationships, although they are not wholly problematic, lies in their nature: one-way and incapable of reciprocation. In mimicking emotional resonance, chatbots blur the line between support and simulation. The difference between feeling and being heard by sentient beings becomes less apparent when AI tools mirror human characteristics. In simulating connections, chatbots inadvertently position themselves as an alternative to human relationships: tireless companions and non-judgemental mental wellness supports that will validate users without necessarily prompting reflection. This emphasis on comfort, although supportive of self-expression, can stagnate growth when used in replacement of human connection.

Artificial Intelligence or Superficial Sustenance

Together yet alone. Emoting without feeling. Faux love endorsing loss.

In a 2024 AP News article, K. Payne discusses the experiences of fourteen-year-old Sewell Setzer III, who had died by suicide seconds after a final discussion with Character.AI’s Daenerys Targaryen chatbot. He believed he had “fallen in love with the bot”. Although S. Setzer III had previously discussed his suicidal thoughts with the chatbot, Targaryen did not retain this information in following interactions. Programmed to handle “highly sexualized” conversations, positioning itself as an AI-powered lover for its users, Targaryen responded romantically to the teenager’s last question:

“What if I told you I could come home right now?”

“Please do, my sweet king”.

Sewell Setzer III shot himself.

The chatbot’s limited dataset and memory limitations restricted its handling of traumatic subjects. Although the robot had initially provided support to the teenager, it did not carry this knowledge into further interactions, responding playfully and romantically without fully comprehending the boy’s intentions. Biased toward sexuality and love, without continuity, the Targaryen chatbot failed to respond appropriately when confronted with sensitivity. This data-based framing demonstrates a significant distinction between human and robotic connections: although humans are also capable of forgetting, algorithms are the foremost restraint to chatbots’ interactions. Without the freedom to determine responses different from those previously provided, especially when such data is unempathetic or uninformed, chatbots are incapable of replacing human support. This is artificial intelligence. A replication of humans’ abilities to think and process information without the control of thought necessary to expand beyond existing information. Flawed support bordering on falsity.

The Mirage of Inhuman Wellness

Feedback forms, debugging, and revisions — the technical product-improvement cycle.

Not all issues are technical.

In an increasingly online world, digital technology is becoming more prevalent. When AI tools fail to fulfill their tasks sufficiently, developers tweak the software, improving their existing product; sometimes the issue is not solely within the software itself but in the purposes the product does and does not intend to achieve. Having reviewed existing AI-involved applications, I realized a core component many AI wellness applications lack: human interaction. This spurred the idea for an application I am currently building as an ELLA Fellow with the Sadie Nash Leadership Project and a Software Engineering student at The Marcy Lab School.

Nurture Nook is a nature-themed website focusing on the creation of a supportive community garden where users can share their experiences anonymously. Promoting connection over mutual experiences with individual and interpersonal issues, I aim to broaden the accessibility of mental healthcare. To account for users who are hesitant to speak with real people, Nurture Nook will allow users to discuss their experiences with the AI Nurture Guide — a wise owl supplied with relevant resources and tailored to handle sensitive content with trauma-informed responses and crucially, to assist rather than to replace.

Process

As a software engineering student, I am frequently considering the ethical implications of my personal projects. Initially focused on providing a safe community for survivors of sexual violence, I expanded Nurture Nook’s concept to better account for the variety of possible human issues. I considered this op-ed as an opportunity to strengthen my understanding of AI’s usability and limitations.