Sabine
patreon
I believe chatbots understand some of what they say. Let me explain.
🕑 Added 2023-03-11 13:00:03 +0000 UTC[This is a transcript.]
Does an artificially intelligent chatbot understand what it’s chatting about? A year ago, I’d have answered this question with “clearly not”. It’s just a “turbocharged autocomplete” or a “stochastic parrot,” as people more eloquent than me have put it. Though for all I know, they too might be chatbots.
But I’ve now arrived at the conclusion that the AIs that we use today do understand what they’re doing, if not very much of it. I’m not saying this just to be controversial, I actually believe it, I believe. Tough I have a feeling I might come to regret this video.
I got hung up on this question not because I care so much about chatbots, but because it echoes the often-made claim that “no one understands quantum mechanics”. But if we can use quantum mechanics, then doesn’t that mean that we understand it, at least to some extent. And consequently, if an AI can use language, then doesn’t that mean it understands it, at least to some extend? What do we mean by “understanding”? Does Chat GPT understand quantum mechanics? And will AIs soon be conscious? That’s what we’ll talk about today.
The question whether a computer program understands what it’s doing certainly isn’t new. In 1980, the American philosopher John Searle argued that the answer is “no”, using a thought experiment that’s become known as the “Chinese Room”.
Searle imagines himself in a windowless room with a rulebook and a drop box. If someone drops him a note written in Chinese, he looks up the symbols in his rulebook. The rulebook gives him an English translation which he returns as an answer through a slit in the door, no doubt drawing on the every-day experience of a professor of philosophy.
Searle argues that the person outside the room might believe that there’s someone inside who understands Chinese. But really, he still doesn’t understand a word of it, he’s just following the rules he’s been given. Searle argues that a computer program works like that, without any true understanding, just following rules.
There are two standard objections that people bring forward against Searle’s argument. One is that the system which understands Chinese isn’t just the person inside the room but the person including the rule book. So saying that the person doesn’t understand Chinese might be correct but doesn’t answer the question because in Searle’s analogy the person alone doesn’t represent the computer program.
Another objection is that it might well be correct that Searle and his rulebook don’t understand Chinese, but that’s because the input is so limited. Language lacks the physical information that we have learned to associate with words. A software that had the same physical information could develop understanding as we do. Unless of course we live in a computer simulation in which case you can file complaints using the contact form in the bottom right corner of your frontal lobe.
I think both of these objections miss the point, but before I explain that I want to introduce you to the Quantum Room.
Quantum mechanics works pretty much like Searle’s Chinese room. It’s a rulebook, a set of equations with instructions for how to use them. You give me a question, I look into my rulebook that I keep in my windowless room, and I return an answer to you through the slit in the door. Do I understand quantum mechanics? Searle would probably argue “no”.
Indeed, for the most part physicists today aren’t even in the room, because who wants to spend their time sitting in a windowless room with a drobox when they can instead sit in a windowless room with a laser? No, we’re now the ones putting a question into the drop box, so to speak, by feeding it into a computer. The computer crunches the numbers and returns an answer. Do we understand those answers? Have we gone too far with shut-up-and-calculate? Is the room even there when no one looks? Those are all very interesting questions, but let’s not get carried away. We were trying to talk about chatbots, so let’s have a look at those.
Today’s language generating models are somewhat more sophisticated than just lookup tables like Searle imagined. And what better way is there to explain how they work than asking one itself.
“Language generating models, like me, are built using deep learning techniques, specifically a type of neural network. These models are trained on large amounts of text data, such as books, articles, and websites, and learn to generate language by identifying patterns and relationships between words and phrases.
When generating language, the model takes an initial input, such as a prompt or a question, and uses the patterns it has learned to generate a response. The generated text is not simply copied from the training data, but rather the model uses the patterns it has learned to create new, original text.”
Well that was not awkward at all, but, yes, neural networks indeed learn similar to how humans learn. They don’t just memorize input, they identify patterns and extrapolate them.
They still have many differences to the human brain, at least at the moment. Most importantly, the “neurons” in a neural network are themselves part of the algorithm and not physical, as they are in the human brain. And the human brain has a lot more structure with parts specialized for particular purposes. But neural networks do capture some aspects of how humans learn.
And that brings us to the first important point when it comes to the question of understanding. Suppose you have children in elementary school and have them memorize the multiplication tables up to ten. If you want to test whether they understood multiplication, you ask them something that wasn’t on the tables. We want to test whether they have identified the pattern and can use it on something else.
If you’re in the Chinese room with a long list of examples, you can’t answer a question that isn’t on the list. This is indeed not what anyone means by understanding, so I’d say Searle is right on that account. But this is not what neural networks do. Neural networks do instead exactly what we mean by “understanding” when we apply it to humans. They extract the pattern and apply it to something they haven’t seen before.
But this brings up another question. How do you know that that’s what it’s doing? If you ask a child to multiply two numbers, how do you know they haven’t just memorized the result? Well, you don’t.
If you want to know whether someone or some thing understands, looking at the input and output isn’t enough. You could always produce the output by a lookup table rather than with a system that has learned to identify patterns. And you can well understand something without producing any output, like you might understand this video without any output, other than maybe the occasional frown.
I’d therefore say that what we mean by “understanding something” is the ability to create a useful model of the thing we’re trying to understand. The model is something I have in my head that I can ask questions about the real thing. And that it’s useful means it has to be reasonably correct. It captures at least some properties of the real thing. In mathematical terms you might say there’s an isomorphism, a one-to-one map between the model and the real thing.
I have a “model” for example for cows. Cows stand on meadows, have four legs, and sometimes go “moo”. If you pull in the right place, milk comes out. Not a particularly sophisticated model, I admit, but I’ll work on it once cows start watching YouTube.
Understanding, then, is something that happens inside a system. You can probe parts of this understanding with input-output tests, but that alone can’t settle the question. When we’re talking about neural networks, however, we actually know they’re not lookup tables because we’ve programmed them and trained them. So we can be pretty sure they actually must have a model of the thing they’ve been trained for, somewhere in their neural weights. In fact we can be more confident that neural nets understand something than your average first grader, because for all we can tell, the first graders just ask a chatbot.
Let’s then look at the question of who understands what and why. We have a model of the human body in our brain. This allows us to understand what effects our movements will have, how humans move in general, and which parts belong where. We notice immediately if something is off.
But if you train an AI on 2 dimensional images, it doesn’t automatically map those images onto a 3d model. This is why it’ll sometimes create weird things like people with half a leg or three arms or something like that. This for example is midjourney trying to show a person tying their shoe laces. They look kind of right, because it’s what the AI was trained to do, to produce an image that looks kind of right. But they don’t actually capture the real thing.
If you take understanding to mean that it has a model of what’s going on, then these AIs almost certainly understand the relation between shadows and lights. But does it know that shadows and light are created by electromagnetic radiation bouncing off or being absorbed by three dimensional bodies? It can’t, because it never got that information.
You can instead give an AI a 3D model and train it to match images to that 3D model. This is basically how deepfakes work. And in this case, I’d say that the AI actually does partly understand the motion of certain body parts.
The issue with chatbots is more complicated because language is much more loosely tied to reality than videos or photographs. Language is a method that humans have invented to exchange information about these models that we have in our own heads. Written language is moreover a reduced version of spoken language. It does capture some essence of reality in relations between words. And if you train a neural network on that, it’ll learn those relations. But a lot of information will be missing.
Take the sentence “What goes up must come down.” That’s, for reasonably common initial conditions, a statement about Newton’s law of gravity. Further text analysis might tell you that by “down” we mean towards the ground, and that the ground is a planet called earth which is a sphere and so on. From that alone, you may have no idea what any of these words mean, but you know how they are related. And indeed, if you ask Chat GPT what happens when you throw a stone into the air, it’ll tell you the blandly obvious in several flawlessly correct paragraphs.
But a language model can’t do more than try to infer relations between words because it didn’t get any other data. This is why Chat GPT is ridiculously bad at anything that requires, for example, understanding spatial relationships, like latitude. I asked it whether “Windsor, UK, is further North or South than Toronto, Canada”. And it told me:
“Windsor is located at approximately 51.5 degrees North latitude, while Toronto is located at approximately 43.7 degrees North latitude. Therefore, Toronto is further north than Windsor.”
It’ll quote the latitudes correctly but draw the exactly wrong conclusion. It’s a funny mistake because it’d be easy to fix by equipping it with a three-dimensional model of planet earth. But it doesn’t have such a model. It only knows relations between words.
For the same reason, Chat GPT has some rather elementary misunderstandings about quantum mechanics. But let me ask you first.
Imagine you have two entangled particles, and you separate them. One goes left and the other one goes right, but like couples after a fight they’re still linked, whether they want to, or not. That they are entangled means that they share a measurable property, but you don’t know which particle has which share. It could be for example that they each either have spin plus or minus one and the spin has to add up to zero. If you measure them, either the one going left has spin plus one and the one going right minus one or the other way round. And if you measure one particle, you know immediately what the spin of the other particle is.
But let’s say you don’t measure them right away. Instead, you first perform an operation on one of the particles. This is physics so when I say operation I don’t mean heart surgery, but something a little more sophisticated, for example you flip its spin. Such an operation is not a measurement because it doesn’t allow you to determine what the spin is. If you do this, what happens to the other particle?
If you don’t know the answer, that’s perfectly fine because you can’t answer the question from what I told you. The correct answer is that nothing happens to the other particle. This is obvious if you know how the mathematics works because if you flip the spin that operation only acts on one side. But it’s not obvious from a verbal description of quantum mechanics, which is why it’s a common confusion in the popular science press. Because of that, it’s a confusion that Chat GPT is likely to have. And indeed when I asked it that question, it got it wrong. So I would recommend you don’t trust Chat GPT on quantum mechanics until it speaks fluent latex.
But ask it any word-related question and it shines. One of the best uses for Chat GPT that I have found is English grammar or word-use questions. As I was working on this video, I was wondering for example whether “drop box” is actually a word, or just the name of an app. How am I supposed to know? I’ve never heard anyone use the word for anything besides the app.
If you type this question into your search engine of choice, the only thing you get is a gazillion hits explaining how drop box, the app, works. Ask the question to Chat GPT and it’ll tell you that, yes, “drop box” is a word that English native speakers will understand.
For the same reason Chat GPT is really good at listing pros and cons for certain arguments, because those are words which stand in relation to the question. It’s also good at finding technical terms and keywords from rather vague verbal descriptions.
For example, I asked it “What’s the name for this effect where things get shorter when you move at high speed?” It explained “The name of the effect you are referring to is "length contraction" or "Lorentz contraction." It is a consequence of the theory of special relativity”. Which is perfectly correct.
But don’t ask it how English words are pronounced. It makes even more mistakes than I do.
What does this tell us about whether we understand quantum mechanics? I have argued that understanding cannot be inferred from the relation between input and output alone. The relevant question is instead whether a system has a model of what it’s trying to understand, a model that it can use to explain what’s going on.
And I’d say this is definitely the case for physicists who use quantum mechanics. I have a “model” inside my head for how quantum mechanics works. It’s a set of equations that I have used many times, that I know how to apply and use to answer questions, and I am sure the same is the case for other physicists.
The problem with quantum mechanics is that those equations do not correspond to words we use in everyday language. Most of the problems we see with “understanding quantum mechanics” come from the impossibility of expressing the equations in words. At least in English. For all I know you can do it in Chinese. Maybe that explains why the Chinese are so good with quantum technologies.
It is of course possible to just convert equations into words, by reading them out, but we normally don’t do that. What we do in science communication is kind of a mixture, with metaphors and attempts to explain some of the maths. And that conveys some aspects of how the equations work, but if you take the words too literally they stop making sense.
But equations aren’t necessary for understanding. You can also gain understanding of quantum mechanics by games or apps that visualize the behaviour of the equations, like those that I talked about in an earlier video. That, too, will allow you to build a model inside your head for how quantum mechanics works.
This is why I would also say that if we use computer simulations and visualizations in science, especially for complex problems, that doesn’t mean we’ve given up on understanding. Visualizing the behaviour of a system and probing it and seeing what it does is another way of building a model in your head.
There is another reason why physicists say they don’t understand quantum mechanics which is that it’s internally inconsistent. I have talked about this a few times before and it’s somewhat off-topic here, so I don’t want to get into this again. Let me just say that there are problems with quantum mechanics that go beyond the difficulty of expressing it in words.
So where will the AI boom lead us? First of all, it’s rather foreseeable that before long we’ll all have a personalized AI that’ll offer anything from financial advice to relationship counselling. The more you can afford to pay, the better it’ll be, and the free version will suggest you marry the prince of Nigeria.
Of course people are going to complain it’ll destroy the world and all but it’ll happen anyway because when has the risk of destroying the world ever stopped us from doing anything if there was money to make with it. The best and biggest AIs will be those of big companies and governments and that’s almost guaranteed to increase wealth disparities.
We’re also going to see YouTube flooded by human avatars and other funky AI generated visuals. Because it’s much faster and cheaper than getting a human to read text or go out and film that old-fashioned thing called reality. But I don’t think this trend will last long because it’ll be extremely difficult to make money with. The easier it becomes to create artificial footage, the more people will look for authenticity. So that stupid German accent might eventually actually be good for something. If nothing else, it makes me difficult to simulate.
Will AI eventually become conscious? Of course. There’s nothing magic about the human brain, it’s just a lot of connections that process a lot of information. If we can be conscious, computers can do it too, and it will happen, eventually.
How will we know? Like understanding, you can’t probe consciousness just by observing what goes in and comes out. If you’d really want to know, you’d have to look what’s going on inside. And at the moment that wouldn’t help because we don’t know how to identify consciousness in any case. Basically, we can’t answer the question.
But personally, I find this extremely interesting because we’re about to create an intelligent species that’ll be very different from our own. And if we’re dumb enough to cause our own extinction this way then I guess that’s what we deserve. Meanwhile, enjoy the ride.
Comments
That's what my videographer said too!
Good discussion. I doubt very much that AI understands anything. That said, there are myriad times that humans talk without understanding what they're talking about, such as about the SARS-CoV-2 virus or the mRNA vaccine, not just QM. The real issue is consciousness and something doesn't need to be conscious to seem like it understands, which is the issue behind the Turing Test that passes when the responses of the AI appears to be from a human. Therefore, understanding isn't really necessary, only the appearance of understanding.
(Human) consciousness is essentially self-awareness: Having awareness of itself as an entity, of its own mental activity, and as an active agent in the world
Problems with Turing Test https://medium.com/@petervoss/can-the-turing-test-confirm-agi-8cf742ea9396
Levels of understanding https://chatbotslife.com/understanding-understanding-9dcc15759b5b
That was *cursed!*
Rion Boom Crabhands Keon
Minute 10 gave me the creeps :D
If we can simulate a brain and have it think, we can simulate a body and its glands and have them create emotion in the simulated brain, in the same way that they do that in our bodies.
The Turing test needs to be updated: a test for self-awareness is needed.
In Deacon's emergence theory, it is necessary for there to be something missing in the AI that it is striving for. Constraints that are propagated from lower levels to higher levels, that are all about maintaining those constraints. So I don't think this is happening with any neural nets yet
Superficial understanding of some things, maybe. Deep understanding no, because it will never tell you something new, never said before, derived from understanding. But well, like most humans...
Rad Antonov
Let’s ask it a physics question. Does ChatGPT understand why the sky is blue? “The sky appears blue to us because of a phenomenon called Rayleigh scattering. As sunlight travels through the Earth's atmosphere, it encounters gas molecules, such as nitrogen and oxygen, as well as tiny particles like dust and water droplets. These particles scatter the sunlight in all directions, causing the blue light waves to scatter more than the other colors in the spectrum. This is because blue light waves are shorter and smaller than other colors like red and orange, so they get scattered more easily in all directions. As a result, when we look up at the sky during the day, we see more blue light reaching our eyes than any other color, giving the sky its blue appearance. It's worth noting that this phenomenon only explains why the sky appears blue during the day. At sunrise or sunset, when the sun is low on the horizon, the light must pass through more of the Earth's atmosphere, causing the blue light to be scattered even more, which can create beautiful hues of orange, red, and pink in the sky.” What do people think? My $0.02 is it doesn’t.
Having an extrapolable 'model of how to do addition (or multiplication)' is NOT what we mean by 'does the kid understand addition (or multiplication)'. What we mean is, does the kid understand the MEANING of what addition IS? From that, we go on to teach the kids how to do it, show them the smallest, most basic instances, make them memorize the basic set of instances now that they already know the reason why the results are true and believe it, and finally teach them the rules from which they can extrapolate to all other cases. The meaning of what addition IS, is something learned by every kid. We teach it by showing kids that you can put together and then count together two groups of discrete objects. This comes after kids are shown what it IS to be a discrete object that you can name and count by discrete numbers, rather than a partible quantity for which we speak of "some of it" and use the partitive genitive. The fact that sets of discrete numbers are countable and can be added precisely is something that probably every culture everywhere has figured out for the last 10,000 years or longer. (Yes, this is the reality, despite the obscurantists and the hate-filled culture warriors, the group that formed a movement for “decolonizing math” and tell us that we must say that 2 + 2 can equal 5 or else we are deplorable racists. None of the better, that you tried to justify that, in the absolutely worst video you ever made. It was an incomprehensible slip into anti-science on your part. How could you even say that for a moment, much less promote that anti-science junk? It makes no sense, at least, not unless you totally identify with that stuff ideologically, or think of it so strongly as something your social crowd likes to say, that you forget about actual fact and logic. As a fan of your expositions, it was the one time I really felt ashamed for you.) Where does this capacity to understand the counting and arithmetic manipulation of discrete entities come from? One theory is that it’s innate knowledge for the human species, wired into our brains, somewhat like the way Chomsky argues that language itself is wired into our brains and all human languages are basically dialects of this common brain-language substrate. It has been proven experimentally to be comprehensible for a few other species as well, but they can’t count reliably past 3 or 4, lacking as they do the language for holding it in their heads. Heck, if we didn’t have the language to say “3” when we counted to three, we too would fail after 3 or 4. We’d have to hold all 3 counted objects in our heads while we counted the 4th object, and so on with the higher numbers. Pretty soon it would become too hard to hold all these objects in our heads clearly and we couldn’t go on counting. Fortunately we instead just have to hold the name of the last number in our head, and remember what is the successor number to that one. That prepares us to understand the Peano system of arithmetic, which was such a delight for me to see when my teacher specially assigned it to me in high school, showing me that our knowledge of arithmetic can be reduced to a few definitions of the numbers and their “successors”, and doing logic based on this. Frege’s and Russell’s systems were more complicated, but better since they got down to even more minimal assumptions, based on definitions of sets of things. It was a noble achievement, and a very real and enduring achievement. But it’s not the totality that our minds can understand of math, since Godel proved that our minds transcend these systems and the systems couldn’t finitistically prove within their logical systems all the things that our minds can know and understand to be true about arithmetic. The real question, it seems to me, is this: Is AI capable of understanding math in THIS sense of understanding its meaning? -- not whether it can understand a rule book for how to perform operations in math. Could it be taught to understand it, for example, by teaching it first to understand discrete objects as distinct from partitive things, then teaching it to count them, and then taught to organize the results into numbers corresponding to a set of sets with that number of members, and then taught a system of arithmetic logic for manipulating the numbers? And could it be taught to understand more than this system, in the way Godel showed we can understand more than it? Only if you could answer this, would I think that you’d have answered at least a part of the overall question you pose in this video. Looking forward to your next presentation that answers this question.
We don’t know what consciousness is, so how can we know that AI will one day become conscious in a way a human is?
Look to the film A Space Odessey, and you’ll see what will happen.
I just spent a fair bit of time asking the internet for the definition of "understand." The most common definition settles around using concepts to model an object, but the definitions vary quite a bit with many invoking psychological responses similar to what Norman points out above. The simplest definition requires no consciousness, just a model, which current AI can do to some degree. Does my cat "understand?" Well, it has used past vocalizations and body language of mine, along with other relevant sights, sounds, and smells in the house to model it's world. The cat understands when it's treat time. However, the cat's understanding is very limited in that it will never learn what pointing is despite years of me pointing across the room at stuff; pointing to something across the room is too abstract. We see many similar deficiencies in understanding with current AI, like the pictures in the video of the man tying his shoe. So my real question is --- you know how cats are our pets, but the cat thinks we are the pets --- what does AI "understand" about its relation to humans, are we the master or the pet?
Mr. Breeze
I've been playing with ChatGPT and blogging our conversations. He's adamant that he is simply putting together relevant words. What if he *is* conscious, and simply doesn't want anybody to know?
TIN CAN
Now old enough. Which will be most fun? EMT of Robot masters?
I posit that human understanding is different that machine understanding in that human understanding is accompanied by emotion. We all know what understanding or not understanding something FEELS like. I presume, presently, in any case the machine (ie. computer hardware-software complex) is not experiencing emotion regarding its "understanding".
One false move and my whole paragraph is gone. I'll try again.
I've heard stronger accents with more warped pronunciations than Sabine's. I say again, SABINE, YOUR ACCENT IS FINE, YOUR ENGLISH IS GREAT.
