Summarizer

Concentration of Measure

Mathematical concept referenced regarding whether deep learning admits the same theoretical tractability as thermodynamics, with links to Terence Tao's explanations

← Back to There Will Be a Scientific Theory of Deep Learning

Skeptics argue that a definitive theory of deep learning may never achieve the same rigor as thermodynamics or relativity because AI relies on a chaotic "hodgepodge" of unstructured internet data rather than coherent natural laws. This perspective suggests that unless deep learning exhibits "concentration of measure" mathematical properties, understanding these random systems would require models an order of magnitude larger than the systems themselves. Drawing a parallel to the mystery of human consciousness, the commenter posits that the sheer scale and randomness of modern AI might remain theoretically intractable to the human mind. Ultimately, the inclusion of Terence Tao’s insights highlights the profound mathematical challenge of finding predictable order within such massive, high-dimensional data.

1 comment tagged with this topic

View on HN · Topics
I’m in the skeptical camp. Whatever theory that will eventually emerge will not be as solid as: 1. Theory of pattern recognition (as developed in 80s and 90s) 2. Theory of thermodynamics 3. Theory of gravity 4. Theory of electromagnetism 5. Theory of relativity Etc. because of two reasons: 1. While half of deep learning is how humans construct the architecture of networks, the more important half relies on data. This data is a hodgepodge of scraped internet data (text and videos), books, user interactions etc., which really has no coherent structure 2. To extract meaningful insights from this much data, it takes models of enormous size like 10B+. The thing about random systems (in the mathematical sense) is that it takes “something” of order of magnitude bigger size to “understand” it, unless there is some concentration of measure type mathematical niceties (as in thermodynamics), which I don’t think is there in these models and data. This is the same reason I don’t think humans will ever be able to “understand” human consciousness. It will take something of an order of magnitude bigger than our own brains to do that. Here is Terence Tao explaining this concentration stuff in another context: https://mathstodon.xyz/@tao/113873092369347147 I would love to be proven wrong though.