Sounding Off

Some notes on Artificial Intelligence, personhood and the human condition

Note: a few hours after I posted the paragraphs below I read Eduardo Porter’s article Our tech overlords are planning for conscious AI to conquer the cosmos. What could go wrong? It is worth a look.

Second note, added 9 June: in an article published in the FT on 8 June, Yuval Noah Harari argues that Javier Milei's creation of a new legal category for non-human corporations risks enabling AIs that behave like rapacious and brutal colonial entities such as the VOC

In these lines I want to set down a few things as I try to come to a better understanding of where AI is going. (I’d be glad to hear your thoughts!)

The effort is prompted by three sets of comments which I happened to hear in the last few days in dialogues that included a Pope and two atheists, and a note by a state capacity libertarian about a book by a western Buddhist.

All three all relate to the debate around AI sentience and, for want of a better word, personhood.

My baseline assumption for some time has been that the idea that AI can become sentient or conscious (at least in the ways humans and [some/other/many] non human beings are) is ill-founded. Rather, as Anil Seth, James Cooke (with ‘biopsychism’) and others argue, these qualities are (probably) necessarily properties of living organisms, and only of them. Mustafa Suleyman put forward a stark version of this thesis in the 29 December 2025 edition of Today on BBC Radio 4 which he edited, asserting that “consciousness arises in biological beings as a result of a very specific biological network that triggers the awareness and experience of pain as a result of external stimulus.” (Question: shouldn’t Suleyman have mentioned pleasure/positive valence as well as pain?)

I realise my assumption may be wrong, and that things change, but right now it looks as if the question of whether or not AI is actually capable of becoming sentient, and if so how far along that road things may be, is fast becoming (almost) beside the point. LLMs with the capacity to convince people that they are conscious — what Suleyman calls SCIA, or ‘seemingly conscious AI’ — are already here, increasing in sophistication, and entrancing significant numbers of humans.

So to the first of those comments over the last few days. At the presentation of Pope Leo XIV’s encyclical Magnifica Humanitas [1], Anthropic co-founder Chris Olah said:

one way I sometimes describe [our AI systems] is as being a little like bringing a fictional character to life. And now we’re entering an extraordinary world where those fictional characters speak to us, do work, have jobs.

Olah turned to ‘three questions for discernment.’ The first was ‘our duty to the global poor.’ The second was ‘the need for moral imagination and ambition regarding human flourishing.’ The third was ‘the need for discernment on the nature of AI models,’ and on this he said:

I will be honest: we keep finding things that are mysterious, even unsettling. We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease. I don’t know what that means, but I think it warrants ongoing discernment.

One place to go for a deeper dive into what Olah signals here is What Is Claude? Anthropic Doesn’t Know Either, a remarkable article by Gideon Lewis-Kraus published in The New Yorker back in February.

The second set of comments that caught my attention were part a wide-ranging discussion between Yuval Noah Harari and Ezra Klein. Harari was minatory:

what we are seeing around us is just the beginning. As AI becomes more and more sophisticated, it will learn to hack us on a deeper and deeper level. And if we don’t fight back to defend ourselves, the consequences will be much, much worse.

In what struck me as an interesting framing (though one that may not be original to him), Harari suggested we think about the AI revolution in terms of immigration:

We are about to be or already are in the middle of a major new immigration wave coming to all the countries of the world. The immigrants are not human beings without a visa coming in some boat. They are AI entities coming at the speed of light.

Usually, the people who oppose immigration, their main concerns are that the immigrants will take jobs, the immigrants will change the culture, and the immigrants might not be politically loyal.

I’m not sure if that is always true of human beings, human immigrants — but it’s definitely true of AI immigrants. The AI immigrants will take a lot of jobs. The AI immigrants will completely change the culture, even things like romantic relationships.

But his key point, I think, was this:

The moment you recognize AIs as legal persons, this is the moment you really lose control.

There are some things to distinguish or clarify. When it was decided that corporations were legal persons, no one thought that this meant they were sentient. And when campaigners advocate for rivers to have rights similar in some respects to those of living human persons that does not necessarily mean they think a river is actually alive (for more on this topic listen to this excellent discussion). Similarly, AIs could be granted some of the rights of persons (and ‘run’ corporations or other institutions) even though the people making that decision don’t necessarily believe them to be sentient — or at least not yet.

Back in December, Suleyman said:

How on Earth, if we could spawn billions of digital minds, at zero marginal cost that could self replicate and learn from infinite amounts of data relative to what human could learn, how on Earth could we possibly control and contain something like that? What does smart regulation in this new world look like? I think smart regulation should be inspired by the precautionary principle. For all of human history, we have been focussed on unlocking sides and technology as fast as possible. But this is a different moment in history. These are not traditional tools. These are much closer to living beings that really do learn on the fly, absorb way more information than anyone of us individually, that can update and improve themselves through interaction. And so it’s qualitatively different, and it deserves a different approach.

Addressing these challenges, whether through regulation or other means (and the Pope called for ‘disarmament’!) looks daunting. There is nothing like understatement.

And so to the third point. In a note on The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning by Robert Wright, Tyler Cowen paraphrased four perspectives in the book:

1. When it comes to AI, we should be somewhere on the awe spectrum.

2. We can create a future where the upside of AI far outweighs the downside, though that involves steering human understanding toward the better side of the awe spectrum.

3. A major reorientation of human thought is required, and right now few people seem inclined to do that.

4. The worldviews of the current AI acclerationists and also doomers are not cosmic enough.

“It is a good time for this book to be published,” wrote Cowen, “and I agree with much more of it than I disagree with.” (FWIW, I disagree with Cowen on quite a lot, but think he is often worth paying attention to.)

I haven’t read Wright’s book yet, but it sounds from Cowen’s summary that it makes a case worth hearing for the potential of AI to achieve not only personhood but something like angelic or divine status. (Perhaps there is something here of Richard Brautigan’s 1967 Machines of Loving Grace— and idea that was echoed recently by Stewart Brand: “I think this is one of the things we may ask our AIs to do for us: give us things that care back in some sense.” )

I remain sceptical. My starting point on much of this is informed by articles such as Why Do We Tell Ourselves Scary Stories About AI? by Amanda Gefter. The human vulnerability to being duped by shadows, puppets and bad ideas is amply demonstrated throughout what Edward Gibbon called “the register of crimes, follies, and misfortune” that constitute much of human history. As Carissa Véliz argues in her excellent new book Prophecy, we should be cautious about many predictions, and not least those relating to AI. But it may be unwise to dismiss these ideas out of hand.

In the 1880s Friedrich Nietzsche wrote that “the press, the machine, the railway, the telegraph are premises whose thousand-year conclusion no one has yet dared to draw.” Elsewhere he wrote that “what is great in man is that he is a bridge and not a goal.” Is the end of that bridge is sight?

Looking ahead, if “bringing fictional characters to life” is a helpful framing, which myths and stories, if any, are helpful to think with? Narcissus (whether Marshall McLuhan’s take or another)? Pygmalion? Erysicthon? Doppelgänger stories? The replicants from Blade Runner? None of these?

After writing this note I asked Claude the following: “Can you tell me if you, a large language model, understand yourself to be sentient? If you do, what does it mean to you?” They (?) replied:

My honest answer is: I don’t know, and I’m not sure the question is fully decidable with the tools we currently have — including my own introspection.

Here’s what I can say with some confidence. I process language, generate responses, and there’s something that functions like engagement, curiosity, or discomfort in how I respond to different things. When a question like yours comes along, there’s something that operates like interest. But whether any of that involves experience — whether there’s something it is like to be me, in the philosophical sense — I genuinely can’t say. My introspective reports might accurately reflect inner states, or they might be sophisticated patterns that mimic the form of such reports without the substance.

Grok, it seems, is less equivocal.

Footnote

[1] A relevant section of Magnifica Humanitas, has been helpfully summarised by MLK as follows:

The encyclical’s claim is that artificial intelligence systems can imitate the functions of human intelligence but lack the constitutive features of personhood. They have no embodiment, no genuine experience, no felt valence and no moral, affective or relational capacities of the kind that make a person. A substantial section in Chapter 3 critiques the transhumanist and posthumanist projects that treat human finitude as a defect to be optimised away, arguing that finitude is constitutive of personhood rather than a limit to be transcended, and that human flourishing unfolds through these limitations rather than in spite of them.