Thinking fast and slow in AI: The role of metacognition (2021)
arxiv.org169 points by teleforce 19 hours ago
169 points by teleforce 19 hours ago
There was this post a few days ago https://news.ycombinator.com/item?id=49797323
It had this to say in the linked post:
This led to the natural question: can gzip do language modeling? (...). Here’s some real, unedited output after priming it on tiny Shakespeare:
gzipt --corpus data/tinyshakespeare.txt --prompt $'MENENIUS:\n' --length 200
MENENIUS:
'Though all at once canq
MARCIUS:
Pray now, nocamest thou to a morsel.
LARTIUS:
Hence, and
I' the end admire, where G
again; and after it ag .
The other is what thinking does, it tries to predict the missing sequence of tokens from a high entropy source, the prompt, in order to increase the likelihood of correctly decompressing the desired results from its weights.
I don't know how this is related, but it reminds me of how I have always believed compression to be the ultimate sign of intelligence. If you can reduce something while keeping comprehension, you are finding more abstract symbols to represent the information of the original source.
How can things be compressed without losing information or structure?
Like for text, what would that involve? How do you compress a string or multi-line string without losing information and hopefully structure (paragraphs, would it be like replacing periods and the following space with just sticking the starting capitalized letter of the following word to the previous sentence's last letter and when it decompresses theres some kind of note that converts that back into the. First letter of the next sentence
Other than what the others mentioned about finding more efficient representations, you can also compress by pre-agreeing on some common terminology.
In many ways, we are communicating using a compressed channel (words) since we both have pre-agreed on the meaning of these words.
That and also that agreements can evolve with time and also within a discussion, so a basic level of agreement is necessary, but complete consensus about the meaning of all words is unnecessary and often unproductive to communications held in good faith.
You analyze the frequency of combinations of bytes, then replace those with high frequency with pointers to a single instance.
How can things be compressed without losing information or structure?
Like for text, what would that involve?
Like if you had eight boxes of loose lego, simply shuffling around the boxes wouldn't give you much in the way of reducing the space the legos take up. but if you took the legos (bytes) themselves out of the boxes, you end up saving a lot more space.
The initial content is the most efficient representation but not for persistent digital storage and that's an important distinction because what we call sparse or inneficient is actually quite efficient, but only if you consider human consumption as the optimization target.
It reminds me of "At the time we drew boxes labeled 'perception', 'cognition' with arrows between them." An imprecise quote that I can't place.
I guess my box labelled 'subconsciousness' is trying to say that low-level mechanisms that give rise to the observed cognitive phenomena might have nothing to do with neat boxes.
You mean "Artificial Intelligence meets Natural Stupidity" by Drew McDermott
McDermott's A Critique of Pure Reason pretty much captured all of the misgivings I had about "Good Old-Fashioned Artificial Intelligence", which was slightly unfortunate as I was trying to complete a PhD in that very area at the time (around 1990 or so...)
Did you complete it?
No - there were lots of other interesting things going on in the early 90s that I managed to pivot to. Never regretted not completing as I had given up any desire to work in academia by that point.
Donald Broadbent drew a lot of boxes in the 50s/60s (https://en.wikipedia.org/wiki/Broadbent's_filter_model_of_at...). There have been plenty of critics of this tendency, I recall some calling it 'boxology' rather than psychology.
Most humans have weak meta-cognition, a large percentage doesn't have verbal thoughts.
Meta-cognition makes sense in a dynamic and updatable and modular system, for example I can monitor thoughts coming from my amygdala with my prefrontal cortex and then adjust how I process these thoughts.
In LLMs it makes zero sense, even if you feed the output of one model into another, there is no way they can update the heuristics behind how those were computed.
Interestingly that is not what we got, but maybe we should loop at architectures like this again? The JEPA loop is interesting, but might fail for the in-flexibility of the component ordering
How relevant is this fast/slow thinking thing with regards to current frontier models?
I know a large organization who's built their AI framework completely around this concept, and I feel that it's not really meaningful concept with the capabilities of current models.
That's because an LLM thinks in terms of language, while we think in a different way, then convert the ideas to language. It can be said that language is a tool for the serialization (writing) and deserialization (reading) of human ideas. It is also an incredible useful and powerful tool by itself. This last sentence has been proved true by LLMs themselves. However, since it is working on the serialized version of ideas, I agree with you in that's not the optimal way to think and something not serialized (maybe world models) can be invented that's better for thinking. All this in no way diminishes the usefulness of language and of automated language generation.
> That's because an LLM thinks in terms of language, while we think in a different way, then convert the ideas to language
Do we? I just learned from a speaker[1] that we literally need words to recognize emotions. People who have a poor vocabulary have lower emotional intelligence because without being able to attach a word to an emotion, the brain is unable to recognize & process it.
[1] Dude seemed to be knowledgeable about the subject. He's a specialized trainer, should be educated in this exact field. So hopefully I'm not lying to anyone here :)
I can't agree about the "unable to recognize and process it", simply because that idea is totally contrary to my own experience. I have in fact many memories which have emotions in them, without words or other external elements. However, seeing that language serialization seems to enable a vastly extended memory (entire sagas remembered as songs), it is understandable that something is gained by serialization of emotional experiences, just as something more immediate is lost.
Sounds uncannily similar to the pseudofacts you hear a lot in Neurolinguistic Programming training courses for sales reps. Would ask for a scientific publication reference on that one.
It seems to me that idea is rooted in social consensus.
Someone expresses an emotion but doesn't know how to react to it, their inner group all have an opinion about it, and the consensus is selected as the "appropiate" reaction to it. The individuals who react this way will claim this consensus is the same as emotional intelligence.
Just as there are also people who react in one way, and completely disregard any external opinion about it. They simply have firm opinions and don't need the consensus.
I will not comment on who can belong to each group, that's an exercise for the reader.
'fast' means executing a policy, that is, a state-action mapping. A trained RL model does this.
'slow' means making one or several action-dependent forecasts, evaluating the expected value of the outcomes, and making a decision based on that.
Neither map exactly to the situation with LLMs, but very roughly, the first is analogous to trained classifiers and the second to reasoning models.
The analogy breaks down, since each instance of token being produced is an example of a policy execution (system 1), and reasoning is just stringing lots of these together. But there are those who argued, before LLMs, that system 2 is just "policy composition" anyway...
Wouldn't you need a classifier to even decide if it is system 1 or 2? How capable does this classier need to be?
I think of System 1 as a hash map. If you have a map, and see a new state/key whose action/value is not defined in the map, you have to go with System 2.
Very relevant. Modern models use CoT to do “slow thinking” and this enables them to achieve much greater performance. You can also turn off thinking and answer directly which is quite similar to “fast thinking”, good at approximate maths, not capable of algorithms, etc.
Of course the shapes of what an AI can do in fast vs slow are quite different.
No clue what’s the consensus on this but my internal mental model is absolutely that LLM AI is pure fast mode, no slow mode. The “reasoning” loops are an attempt to mimic the slow mode but ultimately it doesn’t really work. I’m curious about the recent maths advances though, they seem to possibly challenge this.
Its just hype talk. Slow mode is conscious in humans, fast is subconscious, so we need to discuss AI consciousness to talk about system 2 thinking.
So the entire debate is fubar.
You can ask a model for output directly and stop, or you can recursively ask it to keep refining the output.
That seems to fit the fast vs slow model of human thought reasonably well.
Not quite. The better analogy for you is the "thinking" setting on your model.
> You can ask a model for output directly and stop
That’s still several orders of magnitudes too slow to fit fast vs slow. Think of 30ms vs 3-4 seconds to get an idea of what we’re talking about here
That’s a function of the amount of processing power involved not the underlying architecture of decision making.
In terms of making an LLM faster but not in terms of meta-cognition. System 1 thinking as defined by Kahneman doesn’t have 100000x more compute than System 2, it is actually the opposite. That completely contradicts your claim
>System 1 thinking as defined by Kahneman doesn’t have 100000x more compute than System 2, it is actually the opposite.
When are you measuring?
Systems 1 thinking is closer to precomputed tables in some ways. That is by evolution or massive amounts of training your neural network has a narrow fast path it can execute with as little compute at execution as needed.
Slower paths means loops here for humans where the output of a neuron gets feed back into itself. The fastest path = a feed forward neural network without loops.
LLM’s operate strictly feed forward neural networks.
The ratio between a single pass and multiple passes is unchanged when you throw more processing power at both.
In a human 30ms vs 3-4 seconds is a 1:100 ratio. Single vs multiple passes with an LLM varies but a 1:100 ratio isn’t unrealistic. So with enough compute and the right workload single vs multiple pass LLM could sit in that exact same 30ms vs 3-4 second timeframe.
Multiple passes doesn't make it system 2. The defining characteristic of system 2 is consciousness which is expensive and slows down the system.
So to talk about system 2 in AI we need to talk about consciousness. As long as AI is not officially conscious there is no System 2 thinking implemented
Consciousness is ill defined hogwash when used in such descriptions.
The process of internal refinement without external action however fits.
Using system 1 and 2 for AI is ill defined hogwash indeed.
The systems apply for humans and describe conscious and subconscious processing. Without that the entire reference to thinking fast and slow is bullshit.