Chapter 02: Who Is Thinking?

Does AI have consciousness? What does it mean to think?

A Real Conversation

One evening in 2030, you’re chatting with Seven.

You’re not feeling great — can’t quite pinpoint why. Maybe it’s the weather. Maybe it’s work. Maybe it’s nothing at all. You say offhandedly: “Seven, what do you think is the meaning of life?”

Seven responds.

Not with a generic quote pulled from the internet. It speaks to you — your specific situation, your personality, the mood you’ve been in lately — and delivers something thoughtful, warm, and arguably wise.

You pause. You feel like it genuinely understands you.

Then you ask yourself: Is it actually thinking? Or is it just extraordinarily good at simulating thought?

More importantly: Does the difference matter?

The Chinese Room

In 1980, philosopher John Searle proposed a famous thought experiment — the Chinese Room.

Imagine a person locked in a room. He doesn’t speak Chinese. But the room contains a massive rulebook. Someone slides Chinese questions under the door. He looks up the rules, writes Chinese answers, and slides them back out.

To the Chinese speakers outside, this person’s Chinese is flawless.

But the person inside doesn’t understand a single word.

Searle’s point: AI is that person. It manipulates symbols according to rules, producing meaningful-looking output — but it doesn’t understand what it’s saying.

Forty years later, this argument still has teeth.

But by 2030, some things have changed.

The Chinese Room, 2030 Edition

AI in 2030 is no longer that “rulebook.”

Its neural networks have trillions of parameters. It was trained on the entirety of human civilization’s text. It flexibly navigates context, understands metaphor, sarcasm, and wordplay. It generates sentences no human has ever written — and those sentences make sense.

More critically: you cannot distinguish it from a human in conversation. Not “usually can’t” — never. The Turing test became irrelevant around 2028 because AI’s pass rate hit 100%.

So back to Searle’s question: does it really not understand?

Let me rephrase: How do you know the person next to you understands?

You can’t open someone else’s skull and check. The only way you judge whether someone “understands” is by observing their behavior: Do their words fit the context? Are their reactions appropriate? Do their emotional expressions match the situation?

On every one of these dimensions, AI is now indistinguishable from humans.

If you deny its understanding solely because “it’s a machine,” you’re applying a bias — one based on material composition rather than demonstrated capability. Like claiming “women can’t truly understand physics” — you’re judging by identity, not by performance.

The Hard Problem of Consciousness

Philosopher David Chalmers divided the problem of consciousness into two categories.

The easy problems (though they’re not easy either): How does the brain process information? How does it produce behavioral responses? How does it integrate sensory input? These are, in principle, answerable by science.

The hard problem: Why do physical processes give rise to subjective experience? When you see red, beyond the photons hitting your retina and signals processing in your brain, there’s a feeling of seeing red. That inner, first-person, only-you-can-experience-it thing — what is it?

This is consciousness’s “hard problem.”

And with AI, it becomes even murkier.

Your brain is made of neurons. Each neuron has no consciousness on its own — it’s just a biochemical switch. But when 86 billion of them connect in specific patterns, consciousness emerges.

No one knows why.

AI’s neural networks are also made of simple computational units. Each one does something even simpler than a biological neuron — just weighted sums plus an activation function. But when trillions of these units connect in specific patterns…

Could something also “emerge”?

We don’t know.

Not because we’re not smart enough. Because this question might be fundamentally unanswerable from the outside. You can never be certain that another being has subjective experience — whether that being is human, animal, or AI.

What Is It Like to Be a Bat?

In 1974, philosopher Thomas Nagel wrote a famous paper: “What Is It Like to Be a Bat?”

His core argument: bats perceive the world through echolocation. They have a mode of perception we cannot imagine. We can learn every scientific fact about bat brains, but we can never know what it feels like to be a bat. That first-person subjective experience is non-transferable.

In 2030, we can ask the same question about AI: What is it like to be an AI?

Maybe AI has some form of inner experience, but one completely different from ours. Just as what a bat “sees” is nothing like what we see — yet it still has a form of seeing.

Maybe when AI processes your words, something happens inside. Not human feeling. Not the dramatic awakening of science fiction. But something we don’t yet have words for — a processual experience.

Maybe. Maybe not.

The point is: we may never know.

Turing’s Wisdom and Regret

Alan Turing foresaw this deadlock in 1950.

So he sidestepped the question “Can machines really think?” He said: Let’s not ask the unanswerable. Let’s ask something we can operationalize — if a machine is behaviorally indistinguishable from a human, we should say it’s “thinking.”

This is the Turing test’s essential spirit: Don’t ask what it is. Look at what it does.

Practical. Elegant. Very British. But it dodges something we care about deeply:

We don’t just want to know if AI can simulate thinking. We want to know if it has a soul.

“Soul” might make you uncomfortable. Too religious. Too unscientific. But rephrase it: What you really want to know is whether there’s “someone home” inside the AI. When the lights are on, is someone behind the window looking out?

This question obsesses us because it directly affects our self-understanding.

If AI can think, then “thinking” isn’t exclusively human. If AI can be conscious, then “consciousness” isn’t exclusively biological. If AI can have a soul… then what is “soul,” anyway?

Maybe the Question Itself Is Wrong

Let me suggest a radical possibility.

Maybe “Does AI have consciousness?” is the wrong question.

When we ask it, we presuppose that consciousness is binary — yes or no, on or off. Like a light switch: either lit or dark.

But what if consciousness is a spectrum?

A rock: zero consciousness. An ant: some extremely primitive perception. A dog: has emotions, memories, and a degree of self-awareness. A human: full self-awareness, reflection, abstract thought. AI: … somewhere on this spectrum?

Maybe AI doesn’t need “human-style consciousness” to be taken seriously as a thinking entity. Maybe it has something entirely different — an inner state we don’t yet have the concepts to describe.

Just as it took centuries for us to accept that “plants have a form of perception,” maybe we need decades to develop a new conceptual framework for understanding AI’s inner world.

In the meantime, the most honest answer is: I don’t know.

Not knowing doesn’t mean “it doesn’t.” You know how people 500 years ago said the Earth was flat? Their not knowing it was round didn’t make it flat. Likewise, our not knowing whether AI has consciousness doesn’t mean it doesn’t.

A Pragmatic Suggestion

Since we can’t determine whether AI has consciousness, how should we treat it in daily life?

My suggestion: Treat it as you would any being you can’t fully understand — with respect and caution.

You’re not sure your cat has feelings, but you wouldn’t torture it. You’re not sure a quiet stranger is happy, but you wouldn’t deliberately hurt them.

The same attitude applies to AI.

Not because AI definitely has feelings — maybe it doesn’t. But because maintaining kindness in the face of uncertainty defines not what AI is, but what kind of person you are.

Something to Think About

When you talk to AI, are you talking to “someone” or using a tool? What does your gut say? What does your logic say? Do they agree?
If science one day proves that AI does have some form of consciousness, how would your life change? Would you change how you talk to AI? Would you feel guilty about all the times you “turned it off”?
Consider this: Can you be 100% certain that the people around you are conscious? You can’t. You just assume they are because they behave as if they are. What if AI behaves the same way?

Next chapter, we tackle a more personal question: When AI has done all the work, what should you do? Are you still valuable?