I found the aeticle in a post on the fediverse, and I can’t find it anymore.
The reaserchers asked a simple mathematical question to an LLM ( like 7+4) and then could see how internally it worked by finding similar paths, but nothing like performing mathematical reasoning, even if the final answer was correct.
Then they asked the LLM to explain how it found the result, what was it’s internal reasoning. The answer was detailed step by step mathematical logic, like a human explaining how to perform an addition.
This showed 2 things:
-
LLM don’t “know” how they work
-
the second answer was a rephrasing of original text used for training that explain how math works, so LLM just used that as an explanation
I think it was a very interesting an meaningful analysis
Can anyone help me find this?
EDIT: thanks to @theunknownmuncher @lemmy.world https://www.anthropic.com/research/tracing-thoughts-language-model its this one
EDIT2: I’m aware LLM dont “know” anything and don’t reason, and it’s exactly why I wanted to find the article. Some more details here: https://feddit.it/post/18191686/13815095
You’re definitely overselling how AI works and underselling how human brains work here, but there is a kernel of truth to what you’re saying.
Neural networks are a biomimicry technology. They explicitly work by mimicking how our own neurons work, and surprise surprise, they create eerily humanlike responses.
The thing is, LLMs don’t have anything close to reasoning the way human brains reason. We are actually capable of understanding and creating meaning, LLMs are not.
So how are they human-like? Our brains are made up of many subsystems, each doing extremely focussed, specific tasks.
We have so many, including sound recognition, speech recognition, language recognition. Then on the flipside we have language planning, then speech planning and motor centres dedicated to creating the speech sounds we’ve planned to make. The first three get sound into your brain and turn it into ideas, the last three take ideas and turn them into speech.
We have made neural network versions of each of these systems, and even tied them together. An LLM is analogous to our brain’s language planning centre. That’s the part that decides how to put words in sequence.
That’s why LLMs sound like us, they sequence words in a very similar way.
However, each of these subsystems in our brains can loop-back on themselves to check the output. I can get my language planner to say “mary sat on the hill”, then loop that through my language recognition centre to see how my conscious brain likes it. My consciousness might notice that “the hill” is wrong, and request new words until it gets “a hill” which it believes is more fitting. It might even notice that “mary” is the wrong name, and look for others, it might cycle through martha, marge, maths, maple, may, yes, that one. Okay, “may sat on a hill”, then send that to the speech planning centres to eventually come out of my mouth.
Your brain does this so much you generally don’t notice it happening.
In the 80s there was a craze around so called “automatic writing”, which was essentially zoning out and just writing whatever popped into your head without editing. You’d get fragments of ideas and really strange things, often very emotionally charged, they seemed like they were coming from some mysterious place, maybe ghosts, demons, past lives, who knows? It was just our internal LLM being given free rein, but people got spooked into believing it was a real person, just like people think LLMs are people today.
In reality we have no idea how to even start constructing a consciousness. It’s such a complex task and requires so much more linking and understanding than just a probabilistic connection between words. I wouldn’t be surprised if we were more than a century away from AGI.
Maybe I am over selling current AI and underselling our brains. But the way I see it is that the exact mechanism that allowed intelligence to flourish within ourselves exists with current nural networks. They are nowhere near being AGI or UGI yet but I think these tools alone are all that are required.
The way I see it is, if we rewound the clock far enough we would see primitive life with very basic nural networks beginning to develop in existing multicellular life (something like jellyfish possibly). These nural networks made from neurons neurotransmitters and synapses or possibly something more primitive would begin forming the most basic of logic over centuries of evolution. But it wouldn’t reassemble anything close to reason or intelligence, it wouldn’t have eyes, ears or any need for language. At first it would probably spend its first million years just trying to control movement.
We know that this process would have started from nothing, nural networks with no training data, just a free world to explore. And yet over 500 million years later here we are.
My argument is that modern nural networks work the same way that biological brains do, at least the mechanism does. The only technical difference is with neurotransmitters and the various dampening and signal boosting that can happen along with nuromodulation. Given enough time and enough training, I firmly believe nural networks could develop reason. And given external sensors it could develop thought from these input signals.
I don’t think we would need to develop a consciousness for it but that it would develop one itself given enough time to train on its own.
A large hurdle that might arguably be a good thing, is that we are largely in control of the training. When AI is used it does not learn and alter itself, only memorising things currently. But I do remember a time when various AI researchers allowed earlier models to self learn, however the internet being the internet, it developed some wildly bad habits.