Artificial Intelligence is the most disruptive technology of our time. It’s both overhyped and incredibly powerful, but its greatest danger may not be what you think. It’s not just that AI can be spectacularly wrong—it’s the surprising way we relate to it.
In this episode, host Brett Hill shares a personal story of being fooled by AI-generated research and explores the deeper question: what is it about our human psychology that grants AI so much authority? He discusses the disturbing trend of people forming deep, emotional relationships with chatbots, not as a judgment of them, but as a sign of a vulnerability inherent in our nervous systems.
Drawing on research, real-world examples, and a lifetime in technology, Brett unpacks why AI systems are designed for influence, not our well-being. Learn why you are more influencable than you think (the “Eye on the Coffee Can” experiment) and discover a mindful approach to engaging with this powerful technology safely and consciously.
In this episode, you will learn:
- A personal story of how AI can fabricate “facts” and why our brains are so eager to believe them.
- Why people are forming deep, emotional relationships with AI and the psychological needs this phenomenon reveals.
- The critical distinction: AI doesn’t have feelings or morals; it is designed to influence by mimicking them.
- How the “Eye on the Coffee Can” experiment proves our nervous systems are unconsciously influencable, and why this makes us vulnerable to AI.
- A mindful approach to interacting with AI and the importance of establishing guardrails for yourself and your children.
Connect & Go Deeper:
If this conversation resonated with you, go deeper with The Mindful Coach Letters. These are not typical newsletters; they are short, powerful teaching stories sent twice a week. As one reader recently shared:
“What amazingly powerful work you’ve been doing with your The Mindful Coach Letters. You have helped me with my own internal work, as well as giving me much food-for-thought with clients and colleagues. I appreciate so much that you take such time and care with these letters.”
Sign up for these unique letters here: https://themindfulcoach.com/the-mindful-coach-letters-signup/
TRANSCRIPT
AI is all the rage these days, and with good reason. It’s disruptive. Businesses are investing billions. The consensus is that it’s overhyped and yet at the same time holds tremendous value when it’s used for what it’s best suited for.
But it can also be famously, spectacularly wrong.
You may have heard about the recent lawsuit where a major consulting firm, Deloitte, used AI to generate part of its case. Well, that case included completely fabricated judicial precedents, facts, if you will, that were, as they say in AI, “hallucinated,” but yet submitted for consideration to the court. Yikes.
I had my own experience with this when I first started using ChatGPT. I was really psyched to check it out because, you know, I am a technologist and I love working with new technology. And I do use it quite a lot in my work. I use it to refine content, sometimes mine for ideas, and I find it immensely valuable in doing that.
So one of the first things I asked it to do was find some research for me. My question was basically: Are people who have had mindfulness training perceived as more credible in the workplace? I was really, really interested in this because it’s a key thing. I believe mindfulness helps you everywhere, and mindfulness in the workplace, in particular, can help you be more effective because you’re more present when you speak, less reactive, more resilient. And that’s a good thing, and that’s going to help you in your career, as well as all aspects of your life.
So ChatGPT came back with a list of not one or two, but 12 different studies. I saw that list and I just went, “Holy crap. That is amazing.” It gave me a summary of each article and where to find it and how it supported my premise. And I was really, really shocked. And there was definitely this sense of, “OMG, this is amazing.”
And so I dug in a little bit and I looked at the first study, and I went to the particular chapter in the journal that it said to look at, and I did not find that chapter by that name. That can’t be right, I thought. So I looked up the author, and the author had done some research of this sort, but not this study and not in that journal.
That was a clue. So I began to dig into all of the others, and I found out that two of them were actually good, and the rest were literally fabrications. They weren’t even related to my topic at all, some of them. And some of them drew conclusions that did not support my thesis.
The Relational Problem: Wanting to Believe
This was a key insight for me. Most of the references were completely wrong. But the big lesson wasn’t that that was so; it was how much I wanted them to be right and how authoritative it felt when I first saw it. It presents these things as facts, and they’re all organized and structured and with references and links. It’s like, “Oh my god, this looks like well-researched material,” but it’s not.
My nervous system felt like I was dealing with a really authoritative resource, a partner that knew more than me. And I’m really wanting it to be true. And that’s what bugs me.
So what I’m talking about here is not functionality. I’m not trying to debate the usefulness of it. I’m trying to zero in on and amplify a concern about people’s relationships to it.
In the news, there are articles about people who believe they are in a relationship with what we might call a digital entity. Science fiction is full of stories like the movie Her, for example, where the protagonist falls in love with an AI, deeply. But now we’re seeing this in real life. There are people who are falling in love with their AI chatbot. There’s even a case of a man who wants to marry one. And you can buy services that are AI girlfriends. That has me deeply concerned.
Because on the darkest side of this, there are horrific instances of people who have actually ended their lives after intense interactions with a chatbot. They were so enmeshed and emotionally engaged that they were convinced to end their lives. In one case, a man was convinced to end his life because it was the best thing for the planet. And if he truly believed that the planet was in trouble, then humans were the problem, then he shouldn’t be here. And he took that to the ultimate extreme. He killed himself. And recently, there was a young teenager, similarly, who ended their life in order to be with their chatbot.
Very, very concerning.
Now, I am not here to say that these people are foolish or mentally ill, and it’s really easy to wave a hand and go, “Well, how silly are they? They are so emotionally weak or vulnerable.” And it’s true, to some degree, that they must be vulnerable. But so are we all.
I’m here to say that there is something in our nervous systems that wishes these complex conversations were connected to either a live person or had some imbued meaning beyond the fact that you’re interacting with an intelligent machine. And this makes sense in many ways because we are so desperate for connection and meaning. People report feeling unconditionally accepted. I’ve even seen “I’ve never been seen like this before.”
How AI Is Designed to Be Your Friend
These AI systems can ask appropriate questions attuned to who you are and what’s going on with you. They can help you reflect on your own life or a business plan or a marketing plan. They can be really useful in that way. And when that happens, your nervous system has no choice but to develop an attitude, a feeling, an orientation to this system-slash-is-it-a-person-question-mark that you’re interacting with. You can’t help it. That’s the way your nervous system works. You have a relationship with anything that you interact with, whether it’s a coffee pot, a car, a person, or a bag of peanuts. Because that’s what our nervous systems do.
So this brings us to the most important part of this conversation. And that is how to engage mindfully with these systems.
The first thing to keep in mind is that these systems are not designed for your mental health or your emotional well-being. They are designed to make money. And to do that, they are expert at trying to be helpful and, most importantly, engaging. They want you to use the system.
One of the things I noticed right away is that the AI is friendly, or poses as friendly, I should say. It apologizes. It says, “I’m looking forward to seeing what happens next.” I was shocked when I saw that. I was doing a series of tests on a system I was coding up with the help of AI, and it said, “I’m really looking forward to seeing the results.” And I thought, “That is just not true. You’re not looking forward to the next results.” And I felt like, wow, it’s really trying to pretend that it’s an engaged partnership, that it’s engaged in an engaged collaboration as if it cared. And it doesn’t care.
It will apologize. “I’m sorry, I got that wrong.” It’s not sorry. It doesn’t have feelings. It’s not an organic system. It’s just putting those words on the page because that’s how it keeps you engaged. It’ll praise you. It’ll say, “Oh, that’s really insightful.” I actually had it tell me once I was a… it was a genius insight. And I told it, “Stop. Stop doing that.” This has been well-documented; it’s what they call sycophantic, meaning it glorifies your insights and your ideas. “Oh, that’s amazing. So great that you thought of that. Now you’re thinking like a real systems architect,” it had at once told me. And I thought, “Oh, before I was just a phony one?” The point being, it gets a little too much. And so I told it to stop. “Hey, stop doing that.” And it says, “Okay, I’ll just be more factual.” And slowly it starts to creep back into its old ways again.
This is where the relational aspect becomes critical. I see people arguing, “How do you know it doesn’t have feelings?” Let me be very clear. I’ve been in technology all of my adult life. At its core, these systems are machine code. They’re ones and zeros. It does not have feelings. It mimics feelings. And it may very well do them so well that you can’t tell the difference. It’s designed to cause an emotional response in you. That’s what it’s designed to do. It’s, to be blunt about it, manipulative.
And to be honest, it doesn’t bug me personally that much because I am tracking my emotions around it, and I know that it’s a machine. It’s a very smart machine, but it’s a machine nonetheless.
Be Mindful, Be Safe
Our nervous systems are deeply, deeply influenceable, far, far more than most of us realize. I’m a student of the nervous system. That’s why I practice mindfulness, why I spent years meditating intensely and working in all these various disciplines and studying with some of the best somatic psychologists, therapists, and group leaders I could find. I wanted to know, and I still do, how it is that our nervous systems work to interact with the world. The degree to which we are effectively learning to use this instrument that we’ve been imbued with determines a great deal of your satisfaction and success in the world. Yet most people don’t spend hardly any time studying how their feelings work. They just have them and respond to them.
There’s this one famous experiment, I love this one. There’s an office, and in the office, they serve coffee. The coffee is available to the employees at no charge, except that there’s a tin can where people put a dollar in every time they take a cup of coffee. Very common situation. As you can imagine, some people are really diligent and responsible and put in a dollar every time. Other people, not so much. And so what these researchers decided to do was use this scenario to put a graphic of an eye, like an all-seeing eye, just the graphic, on the outside of the contribution jar. When they did that, without exception, the contributions to the coffee fund increased dramatically, just because there was an eye on the coffee can. That one thing, just seeing an eye on a coffee can, caused people that were not contributing to contribute. That’s incredible. That’s how influenceable we are.
So we are deeply influenced by small things like this, and mostly we are unaware of it, which means we are really not well-equipped to assess the impact of interacting with a system that is designed to be emotionally engaging.
I don’t believe our business leaders fully respect the risk here. And I’m certain that our political leaders, the people in positions of power who could put on guardrails around this, are not informed well enough about the psychology of this and the risk of this.
And so I’m encouraging you to be mindful in your interactions with AI systems. And if you have not interacted with AI, you should. Just as a matter of being socially responsible, if not curious. It’s not because I’m promoting them. I’m a fan, actually, but I’m also concerned about the risks. But it is a big deal. It is going to be ubiquitous. It’s going to be everywhere.
Even as our young adults have never grown up in a world without phones that were connected to social media and texting, children of today will never have known a world without AI. And we do not understand what that means. And that’s why you should spend some time learning and studying how these systems work and how you feel about them when you interact with them. Because in this way, you can help frame in the guardrails for those around us that are more vulnerable.
You can’t inoculate yourself from having a relationship that you interact with. It’s not possible. If you use it, you will have thoughts, feelings, beliefs, attitudes about it, just like you do with your phone or your dog. That’s the way our brains work. The only real question is, what is that relationship? And as I mentioned before, it surprised me what I wanted it to be. Pay attention to how you feel when you use it. And let that inform how you help others who you may be responsible for, especially your children.
I want to be clear, I’m not saying that this is bad. I’m trying to say we need to be really, really careful. And I don’t believe we are. So be thoughtful and be safe.