Claude Passes Daniel Dennett’s Turing Test

 One of our most memorable episodes of the InFi podcast was #15 in which we added audio (in a smart British accent) to ChatGPT-4 to reenact a “conversation” I had had with the Large Language Model (LLM). The point of the episode was to demonstrate to the public just how impressive the latest LLMs had become.

In today’s post, I want to give another update, this time regarding Anthropic’s “Claude” LLM. But to calibrate our expectations, let me start with an anecdote from college. 

Daniel Dennett’s Book on Consciousness

When I was an undergrad, I was an economics major but I also immersed myself in the philosophy of mind. Either as required reading for a class, or just something I picked up on my own, I read Daniel Dennett’s Consciousness Explained. At the time, I was an atheist materialist, and I found the book very compelling; I don’t know how I would feel about it now (having rediscovered Christianity).  But for our purposes in this post, I want to highlight Dennett’s discussion about John Searle’s so-called Chinese room argument.

In a 1980 article, the philosopher Searle argued that a computer could never truly understand and thus have a mind or be intelligent, because its operations could always be reduced to mechanical procedures. He illustrated with a thought experiment that has since become famous in the literature: Suppose an English speaker who knows no other tongue, is locked inside a room where he has elaborate instructions on how to manipulate Chinese symbols that are passed in to him from the outside, and then he passes the output once he has completed the tasks. From the outside, fluent Chinese speakers would imagine that whatever was happening inside the room, someone (or something?) would have to understand Chinese, because the questions they fed into the room would pop out with perfectly sensible responses.

But Searle’s point is that “clearly” there would be no genuine understanding here, because the guy doing the actual manipulation didn’t understand Chinese, and a bunch of papers with rules written on them (for symbol manipulation) don’t understand Chinese either.

In response to this, Dennett claims that Searle has pulled a fast one on his audience, based purely on misdirection, for the whole thought experiment rests on the assumption that if one component of a system doesn’t understand Chinese, then neither can a bunch of components put together. And yet, Dennett argues, you could likewise argue that a Chinese person doesn’t understand Chinese, since (after all) if you focused on any individual neuron in his brain, it would merely be “blindly” following the laws of biology/chemistry. (I am paraphrasing pages 435-439 of Dennett’s book, but I hope my summary is faithful to both men.)

Now what’s interesting here—and why I went to the trouble of finding the passage again, after all these years—is that I remember Dennett had included a hypothetical sample from a “conversation” with the Chinese room, to show the reader what it would really feel like if the responses coming out of this thing sure seemed to be emanating from a conscious being. Here it is:

And

(Screenshots are from Dennett’s Consciousness Explained, pp. 436-437)

Now at the time (in the late 1990s) when I read this, I was a total computer geek, who was for example fairly adept at programming for a kid. The more advanced computer games of my childhood allowed you to type in commands like “give the troll the amulet” or “look under the bed,” and the programmers had also included a list of naughty words so if you typed those, the game would say to you, “Do you kiss your mother with that mouth?” etc. But it was pretty clear how limited and formulaic the “conversations” with the computer were, at the time. 

In contrast, the hypothetical exchange in Dennett’s book was so clearly light years ahead of anything computers could do at the time, that I remember thinking, “Oh sure, if a computer could talk like that for an extended period, then it would be Data from Star Trek…” 

With that as a benchmark for College Bob, let’s now turn to a conversation (and I use that word advisedly) I recently had with Claude.

Claude Referees Bob’s Twitter Exchange

 The following will be fairly self-explanatory. But I want to be clear, at the outset I just uploaded a single PNG screenshot of my exchange from Twitter (complete with nesting); below you are seeing the entirety of what I gave Claude to work with. Check it out:

As I say, I think the above is fairly self-explanatory. It should go without saying that this conversation blows out of the water the hypothetical example that Dennett had dreamed up in his book (which originally came out in 1991). I urge the reader not to take for granted Claude’s ability to interpret the “conversation” of the nested tweets in the original screenshot I shared; that is simply mind-blowing to me that it could explain the sequence of events. The fact that it “knows” about set theory and Gödel’s incompleteness theorems is just icing on the cake, which you might expect from a computer.

 Notice also how Claude said “Ha!” when I revealed my identity. Understanding humor is literally the thing that tripped up the fictional android (Data) from Star Trek. Claude here isn’t going to be stealing jobs from stand-up comics anytime soon, but that “Ha!” was definitely an appropriate response to the situation.

 Yet the thing that truly startled me is when (after I revealed my identity) Claude came back and said to me, “Though I do have to ask, did you come here specifically to test…” That actually made me sit back in my seat. There is nothing “obvious” about this part of Claude’s response, as if I had typed, “Mary had a little” and Claude knew to finish with “lamb.” It is simply astonishing that, whatever is going on under the hood, this combination of letters popped out.

 Finally, let me conclude by saying that we can still see limitations. At the very end, in Claude’s final two responses to me, there was some redundancy in the references to Twitter’s limitations. A human (at least native English speaker) wouldn’t talk like this, and if I were engaged in a genuine Turing test, this “tell” would make me confident I was talking to a computer.

But the thing is, this is pretty deep into the conversation to spot an obvious (and still quite subtle) flaw. I imagine the next generation of Claude (or competing LLMs) will overcome this. Readers may be interested in my 2024 post where I defined the “Murphy Window” as “the length of time it takes a typical human to be sure he is dealing with an AI system, rather than a genuine human.” In that article, I predicted that the Murphy Window would continue to grow, such that in practice, philosophical questions about whether the machines could “really think” would become moot for many purposes.

Dr. Robert P. Murphy is the Chief Economist at infineo, bridging together Whole Life insurance policies and digital blockchain-based issuance.

Twitter: @infineogroup, @BobMurphyEcon

Linkedin: infineo group, Robert Murphy

Youtube: infineo group

To learn more about infineo, please visit the infineo website