The current approach to train large language models to be able to reason about problems is to add logical break downs to the training data for some input.

This approach is called chain of thought and works very well for clearly defined problems like solving a quadratic equation or other tasks with a set of clear steps in general. The interesting part is the application to new problems that do not exist in the training data. A new prompt is then compared to similar requests, basically superposing possible solution paths to known problems. The generated steps can then be further used as intermediate prompts internally to prune steps that do not lead to answers (with some supervised learning) in the problem representation.

In general this is not different to the previous approach of finding the statistically most likely answer, except it breaks down the request to small steps, where the answer to each step is statistically easier to find and has a larger confidence to be correct.

Flat or Hierarchy Model

We can invert this approach and try to find some conclusions to our own thought process. Is our reasoning capability the same, meaning that it emerges from a superposition of known basic questions and answers? Generally speaking, is it just piling a flat set of data input and output until most input can be mapped to a reasonably close problem set?

At the moment I am skeptical that without some live pruning process, like an additional level of reasoning that uses an abstract model of the world, and that can start the logical deduction from very basic axioms, is able to lead to a model that will be useful on complicated questions. Especially, if the problem is fuzzy, meaning that information is missing or even wrong. Where previous assumptions need to be actively challenged.

Another problem is extrapolating, transferring or abstractions of problems. All these features of reasoning are hard to define and make explicit in the training data. I am not sure extending the chain of thought steps to include very basic assumptions will improve this, as the more abstract and removed you are from current reasoning paths the more difficult it will be generating a correct answer from the training set. It is like making a prediction from current data far into the future, it will get less and less accurate.

Limits

Still, I do not know how far this approach will take us. For well defined problems this will lead to models that are able to handle most prompts. Unfortunately, the challenge is finding the limits of the model. The problem sets it will be able to handle confidently and the problems it will not be able to. Otherwise no model, especially in critical systems will be secure enough to be used.

A Kernel

Going back to the initial discussion point about reasoning being a superposition of flat data. If we compare to how humans learn I think there is a fundamental difference. For example, learning language in humans is in my opinion based on some pre-existing structures in the brain that are able to detect patterns in sounds, correlate them to visual input like facial expressions, gestures, objects and deriving some understanding from this.

It is a continuous process at each step refining previous input and also testing actively if the learned input is correct by making sounds, speaking words etc. with feedback from parents. In the end some abstract understanding of language is built, not just for a specific language but also abstract concepts like grammar and what words are.

Stripping the individual data points, like when a parent pointed to something and made a sound, does not impact this abstract model. I would call this final construct a kernel, something abstract and fundamental representing a concept. I think there are many kernels for different areas but all interconnected leading to the ability to transfer between them on an abstract level.

Going one step further we could think of a fundamental kernel, one that defines how to reason itself. And that would be the actual source of general intelligence. Maybe training by chain of thought and just brute data sizes might lead to this kernel. I think it would emerge in a different way, one that starts with creating a basic feedback loop that is refined over time.

Evolutionary Approach

This is not a new thought, as this evolutionary approach is decades old. But the issue here is that the training necessary to create a fundamental kernel is outside our time frame. For humans this took millions of years and endless generations. So we have two options here, either decoding the brain and finding how our kernel works, or creating an environment that is able to train much quicker.

I think decoding the brain is not feasible with current technology. But what might be possible is creating a simulation that is complex enough to force the evolution of reason.

But maybe the brute force statistical approach will be good enough. It definitely will be useful if the error rate for simple problems is small enough. Let's see.