Reservoir Computing Comparison

Suggestion that biological brains may have more in common with reservoir computing than deep learning, given the differences in learning algorithms

The comparison between deep learning and neuroscience may be fundamentally flawed, as biological brains likely align more closely with the principles of reservoir computing than with the rigid structures of end-to-end loss optimization. While deep learning is immensely powerful, its reliance on global credit assignment and fixed-step processes creates a restrictive framework that differs significantly from how organic intelligence functions. Consequently, there is a growing interest in finding a middle ground that moves beyond these limitations, exploring models that can capture the fluid, variable complexity found in both biological systems and the next generation of generative tools.

View on HN · Topics

Comparing Deep Learning with neuroscience may turn out to be erroneous. They may be orthogonal.

The brain likely has more in common with Reservoir Computing (sans the actual learning algorithm) than Deep Learning.

Deep Learning relies on end to end loss optimization, something which is much more powerful than anything the brain can be doing. But the end-to-end limitation is restricting, credit assignment is a big problem.

Consider how crazy the generative diffusion models are, we generate the output in its entirety with a fixed number of steps - the complexity of the output is irrelevant. If only we could train a model to just use Photoshop directly, but we can't.

Interestingly, there are some attempts at a middle ground where a variable number of continuous variables describe an image: < https://visual-gen.github.io/semanticist/ >

Summarizer