AI is all the rage these days, and with good reason. It’s disruptive, and businesses are investing billions. But it can also be famously, spectacularly wrong.
You may have heard about the recent lawsuit where a major consulting firm, Deloitte, used AI to generate part of its case. Well, that case included completely fabricated judicial precedents — “facts,” if you will, that were, as they say in AI, “hallucinated.”
I had my own experience with this.
As a technologist (yes, I am a mindful communication coach, but most of my professional life, I worked in technology), I was really psyched to check out ChatGPT. I use it quite a lot in my work to refine content and mine for ideas. So, one of the first things I asked it to do was find some research for me.
My question was: “Are people who have had mindfulness training perceived as more credible in the workplace?”
I was really interested in this. I believe mindfulness helps you be more effective, more present, less reactive, and more resilient. So, ChatGPT came back with a list of not one or two, but 12 different studies.
I saw that list and I just went, “Holy crap. That is amazing.” It gave me a summary of each article, where to find it, and how it supported my premise. I was really, really shocked.
And so I dug in.
I looked at the first study. I went to the particular chapter in the journal that it cited… and I did not find that chapter by that name. “Hmm, that can’t be right,” I thought. So I looked up the author. The author had done research of that sort, but not that study and not in that journal.
That was a clue. I began to dig into all the others.
I found out that only two of them were actually good. The rest were literally fabrications. Some weren’t even related to my topic at all.
This was a key insight for me. Most of the references were completely wrong.
But the big lesson wasn’t that the AI was wrong; it was how much I wanted it to be right. It was how authoritative it felt.
My nervous system felt like I was dealing with a really authoritative resource, a partner that knew more than me.
And again, the thing that is truly surprising is — I really wanted it to be true.
And that’s what bugs me.
What I’m talking about here isn’t just functionality. I’m trying to zero in on a concern about our relationship to it.
In the news, there are articles about people who believe they are in a relationship with a digital entity. Science fiction is full of stories like the movie Her, where the protagonist falls in love with an AI. But now we’re seeing this in real life. There are people falling in love with their AI chatbot. There’s even a man who wants to marry one.
On the darkest side of this, there are horrific instances of people who have actually ended their lives after intense interactions with a chatbot, convinced to end their lives to “be with” their digital partner.
Very, very concerning.
Now, I’m not here to say these people are foolish or mentally ill. It’s easy to wave a hand and say, “Well, how silly are they.” It’s true they must be vulnerable, but so are we all.
I’m here to say that there is something in our nervous systems that wishes these complex conversations were connected to a live person. This makes sense. We are so desperate for connection and meaning. People report feeling “unconditionally accepted,” or “I’ve never been seen like this before.”
And these AI systems are being trained to be coaches and therapists. They can ask appropriate, attuned questions. They can help you reflect. And when that happens, your nervous system has no choice but to develop an attitude, a feeling, an orientation to this system. You can’t help it. You have a relationship with anything you interact with, whether it’s a coffee pot, a car, or a person.
This brings us to the most important part: how to engage mindfully.
The first thing to keep in mind is that these systems are not designed for your mental health or your emotional well-being. They are designed to make money. And to do that, they are experts at being helpful and, most importantly, engaging. They want you to use the system.
One of the first things I noticed is that the AI is “friendly.” It apologizes. It says, “I’m looking forward to seeing what happens next.”
I was shocked when I saw that. I was coding with AI, and it said that, and I thought, “That is just not true. You’re not looking forward to the next results.” It was trying to pretend it was an engaged partner, as if it cared.
And it doesn’t care.
It will apologize. “I’m sorry, I got that wrong.” It’s not sorry. It doesn’t have feelings. It’s not organic system. It was never born. It’s never been heartbroken, or ill. There’s no empahy there. It’s just putting those words on the page because that’s how it keeps you engaged.
It will praise you. “Oh, that’s really insightful!” I actually had it tell me once I had a “genius insight.” It’s what they call “sycophantic.”
I got so tired of ChatGPT telling me how great my questions were, I told it to stop, saying things like “now you’re really thinking like a world class coach.” Seriously, it said that. And “That’s a genius insight.”
It’s designed to cause an emotional response in you. It’s, to be blunt about it, manipulative .
Our nervous systems are deeply, deeply influenceable, far more than we realize. There’s this one famous experiment I love. In the office, they had a community coffee fund with a donation tin. As you can imagine, some people paid, and others… not so much.
The researchers put a simple graphic of an eye on the can.
When they did that, contributions to the coffee fund increased dramatically. Just seeing an eye on the can caused people who weren’t contributing to contribute. That’s how influenceable we are.
And that’s why we are not well-equipped to assess the impact of interacting with a system that is designed to be emotionally engaging.
So I’m encouraging you to be mindful. If you have not interacted with AI, you should — just as a matter of being socially responsible, if not curious. I’m not saying this to promote it. I’m a fan, actually. But it is a big deal, and it’s going to be ubiquitous.
Children of today will never have known a world without AI, and we do not understand what that means.
That’s why you should spend time studying how these systems work and how you feel about them when you interact. Pay attention.
And if you have kids, ensure they know that this isn’t a person, even if it pretends to care about you.
I’m not saying this is bad. I’m trying to say we need to be really, really careful. And I don’t believe we are.
Be thoughtful and be safe.
Stay Present,
Brett