Metacognition beyond humans

The core of metacognition is having some kind of self-model, a model of the world that includes ourselves in it — our capacities, our skills, the actions that we are taking. The building blocks of that self-model can actually be quite simple. We think an important component of that is the ability to track uncertainty about different sources of information that are driving our decisions. We know that many other animal species can form a sense of confidence in their actions; they can monitor when they make errors on simple tasks, so we think that is a very widespread capacity. That is something that also seems straightforward for artificial systems to emulate. What is perhaps more distinct about human metacognition is that it also involves a third-person perspective on ourselves. We can know things about ourselves in the same way that we know things about other people. We can represent ourselves as having a certain set of traits or characteristics, and we think that that is unlikely to be shared by other animal species — perhaps our closest primate relatives, but not much further than that. Whether that ability for a third-person perspective on ourselves as agents in the world is also something that artificial systems can emulate is an open question. I think there is evidence that artificial systems, such as large language models, can form sophisticated representations of other minds; they exhibit remarkably sophisticated theory of mind. Whether they are also then doing that to themselves and to their own processes is still very much an open question.

It is quite striking. You can have a child who is two or three years old, who is verbal, walking and talking and doing lots of interesting things, and yet they may not yet have the capacity to introspect and reflect on what they know and what they do not know.

It is very hard to gain experimental evidence for cultural acquisition of metacognition, partly because it is rooted in naturalistic interactions in real-world settings. What we would need to gain that evidence are studies that examine child development longitudinally and ask whether exposure to certain caregiver interactions earlier in life predicted the emergence of metacognition later in life. Some of those studies are being done, but the challenge comes with the measurement of metacognition, which is in itself a complex thing to do, especially in young children. One real challenge in that area is that you can often see spurious differences in metacognition that might be a consequence of the child having better memory or better problem-solving ability, and that has nothing to do with their capacity for metacognition. Whenever we are studying metacognition in the lab, whenever we are trying to measure it, we always need to be mindful of the importance of controlling for confounds that might be introduced by variation in other aspects of cognitive performance.

There have been ingenious tests developed to track metacognition in animals, and these involve developing nonverbal analogs of human metacognition tests. In a human metacognition test, we might ask someone to rate how confident they feel in making a particular judgment, or to recognize when they have made an error on a particular task. We can develop similar analogs of these metacognitive judgments for nonverbal animals by training the animal to bet on whether their own performance is successful or not. This is known as post-decisional wagering.

One of the early experiments in the animal metacognition literature was carried out by John David Smith with a dolphin named Natua, and the experiment was set up so that the dolphin was trained to distinguish between two different tones, two different sounds of different pitch.

Nonhuman primates, macaque monkeys and chimpanzees show similar signs of metacognitive competence. In fact, the ability to pass these tests of post-decisional wagering is a more strict test of metacognition, because there has been controversy about these opt-out tasks (the ones that were used originally in the experiments with Natua), since tasks using the opt-out response are ambiguous as to whether they actually involve metacognition or whether they can be solved using first-order representations of the world. The idea here is that the animal might press that third button not because it lacks confidence in its response, but because it has learnt that pressing that third button is the appropriate thing to do when presented with a stimulus that is halfway between the other two. Tasks involving the opt-out response are therefore ambiguous in how we interpret them, but more recent tasks developed using post-decision wagering are more secure in isolating metacognitive processes.

Some of the building blocks of metacognition, such as the capacity to track confidence in a variety of representations about the world, seem relatively straightforward to implement in artificial systems. Large language models like ChatGPT predict the next word in a sequence, and along with that prediction comes a range of probabilities about which word will be most appropriate. In one sense, baked into the architecture of these models is a sense of confidence in whether their answer is correct or not. A variety of researchers have probed whether those confidence estimates that are internal to the system indeed reflect something about the probability of that answer being correct, and, at least for simple questions, the newer versions of ChatGPT and related systems are well calibrated; they have good metacognition about whether they know or do not know the answer.

AI systems are already participating in meaningful social interactions. People are using them for therapeutic purposes, for friendship, for advice, and I think this is only going to continue to grow. One thing we know about the benefits of metacognition for enabling productive social interaction in humans is that being able to share fine-grained estimates of whether we think we have a full picture of a situation, whether we are not so confident in our knowledge, whether we need to seek advice, this is really the glue that enables effective collaboration in broader teams. Work by my former mentor Chris Frith has shown that when people are interacting with each other, they naturally share information about how uncertain or how confident they are in their views, and this is important for them coming to a consensus opinion about the problem at hand. At the moment, at least, many of these AI systems do not have their metacognitive estimates on public display. As I mentioned earlier, these confidence estimates are hidden inside the system. They might be well calibrated, but they are not often being explicitly communicated to the user. One challenge here is to reduce this gap between the internal workings of the system, which might well have well-developed metacognition, and how that is then communicated to the person interacting with that system.

One danger is that people over-attribute competence or confidence to the views of these systems because their internal probabilities are not on public display. We have shown in our lab in recent work that people have these illusions of confidence in AI systems. They are more willing to trust them for advice about a variety of topics than they are to trust a human who is giving identical advice.

It is possible to study the brains of artificial systems in a very similar way that we can study human or animal brains, and, in fact, they are even more accessible, because we know every element of their workings. Labs in AI research are developing techniques where, just like we can image the brain at work, we can take the individual neurons or individual units that make up these very large neural networks and visualize their activity. We are able to probe the different layers of the neural networks that make up these systems with a very high degree of precision. One success story has been the demonstration that artificial neural networks trained to categorize visual images end up developing similar internal representations to those found in the primate visual system accessed by recordings from neurons in a biological brain. There seems to be a very close mapping between the types of computations that these artificial systems are discovering for themselves that help them solve a particular problem, like classifying images, and the types of computations that the brain itself might be using.

There is a desire in the field of cognitive science to want to think about AI systems in the same way that we think about the mind and vice versa. There is a desire to try and characterize what an AI system does, like ChatGPT, in terms of whether it has language, whether it has memory, whether it has theory of mind and so on.