This is actually a very old AI insight, acknowledged at least as early as the 80s, let me see if I can find the quote. Found it: > Rodney Brooks explains that, according to early AI research, intelligence was "best characterized as the things that highly educated male scientists found challenging", such as chess, symbolic integration, proving mathematical theorems and solving complicated word algebra problems. "The things that children of four or five years could do effortlessly, such as visually distinguishing between a coffee cup and a chair, or walking around on two legs, or finding their way from their bedroom to the living room were not thought of as activities requiring intelligence. Nor were any aesthetic judgments included in the repertoire of intelligence-based skills.

I am not a mathematician and did not read the unit distance solution too carefully, but my impression was that it used a variation of a known technique to solve the problem. And that makes perfect sense to me, there are a lot of techniques and lot of less relevant problems, I am not surprised that one can solve some of them with known techniques that just nobody has tried [hard enough] before. I am much more sceptical when it come to the important unsolved problems where every known technique has probably been tried several times over. In those instances it will probably take a true leap in understanding to solve them and I am sceptical that large language models are well suited for that because of the way they work.

We're very fortunate to have had some very eminent mathematicians backfill the OpenAI proof with history, context, and a literature review [1]. Ideas behind the proof seem to have been "in the air". Indeed, looked at certain point of view, the OpenAI construction can be viewed as a high-dimensional generalization of a known low-dimensional one. In this vein see the remarks of Gowers, Sawin and Tsimerman in [1]. Are LLMs capable of "true leap[s] in understanding"? I have absolutely no idea. But LLMs keep surprising me. [1] https://arxiv.org/html/2605.20695v1

We are on tiny 1-5T parameter models with local power stations. We can reach Q models just by throwing resources at it. That’s a million times current B models.

Is this a known or quantifiable thing? I thought that the limit had already been determined i.e. the existing models top out and at some point it doesn't matter how much time or energy you let the model consume, it won't improve the result. And with regards to training parameters, I thought we were equally limited there, e.g. the existing models can't benefit from a larger parameter space. I was under the impression that improvements are arriving via how the models are trained and how model prompting context is constructed, rather than just by how much data or how much energy is spent searching over the model space for a particular prompt. Is there some evidence that we have not reached a pleateau with just resource consumption on existing models?

The existing models "top out" not because they don't get better, but because it is uneconomical. What we do know is that a model "tops out" wrt training data - that is, for a model of a given size, there's only so much training data you can squeeze into the set before you stop seeing gains. But conversely it means that if you already have a model of say 1 Ttok that is "trained to capacity", then a model of 2 TTok needs roughly twice as much training data to fully utilize all those weights. Which means that the cost of training it is not 2x but 4x (twice as many params x twice as many tokens). And then of course serving it is 2x more expensive, but even with optimal training the gains aren't 2x. So it very quickly becomes uneconomical. A good example of that kind of model is (was) GPT-4.5. The prices and the consequent lack of demand show why companies don't really do that sort of thing anymore. But no, there's no evidence of a plateau as such. I'm not sure what "evidence that we have not reached a plateau" would even look like.

what is a B model vs. a Q model? what do these letters mean?

B Billion parameter, T trillion, Q Quadrillion.

You cannot think fast enough when your wires are kilometers long. The only way up is in, and silicon transistors just cannot compete with density with biologic brains, ergo, super intelligence is a pipe dream

Baseless assertions. Fab tech continues to improve. There's no reason ML model internals have to be strictly serial - in fact we're already seeing some shifts away from that.

For every interesting problem AI solves there are a long tail of really dumb things that AI performs that humans would never do. Some days I am in awe of one-shot magic eight-ball output and other days I'm so frustrated by the sheer stupidity of what it produces. It remains to be seen whether that long tail of stupidity can ever be resolved in the current form of LLMs.

At some point I was thinking that maybe I am too hard on the AI and that humans routinely produce exceptionally stupid code, however after a while I've realized that this is only partially true. AI produces a class of mistakes that humans would almost certainly not make because creating even the context of the mistake would require a level of skill that would preclude such mistakes. It's like if you took a mid/senior engineer and randomly lobotomized them mid-task.

It’s in the training data!

You are describing the Jagged Frontier. https://mitsloan.mit.edu/sites/default/files/2023-10/SSRN-id...

A million monkeys at typewriters, but with momentary runs of extreme luck/brilliance.

On the other hand, it could stall out at: good enough to take the easy problems, not good enough to take over the field, but damaging enough to erode the quality of new entrants. (Which incidentally is the scenario I think plays out for software)

I think the OpenAI model that resolved the Unit Distance Problem would be capable of solving a significant proportion of mathematics PhD thesis problems.

In general, most humans top out at 3D visualization, and instead rely on crude mathematical tools to work with higher dimensions. Every so often, people like Euler or Leibnitz pops up to give people new methods for blind men with a cane to explore the unseen yet knowable world(s). Scientific work is not normally naturally statistically salient for LLM observational data inferences. =3

AI (in this form) will never be able to solve things we truly cannot solve yet. It might catch things that we didn't project properly or brute force things no human can , but it will never unify general relativity with quantum mechanics. It's amazing at finding hidden truths in large datasets, but won't win a Nobel unassisted.

> AI (in this form) will never be able to solve things we truly cannot solve yet. Argument?

The strongest argument for this is structural: what LLMs are. In a brutal simplistic way: each token is represented in a high dimensional vector. LLMs operate on them. They are the true, underlying meaning of the token for the LLM. Think of it as 1000+ ways to think of that word/token. Those meanings are baked in at training time. So, LLMs might be able to cross-reference them and solve a class of problems that flew under our radar, but can't come up with revolutionary theories that were never in the training set. Of course, they will help winning a Nobel in the years to come, no doubt, but can't speak mathematics we can't understand (beyond simple obfuscation) and won't discover anything substantial on their own.

> but can't come up with revolutionary theories that were never in the training set. Can you elaborate? I don't think the solution to the unit distance problem was in the training set, but I'm guessing you mean there's some higher bar for revolutionary theories LLMs cant reach? If so where do you expect the limit will be?

Instead of going into a long technical argument of why your description of LLMs is flawed, I'll go straight to the point, because people keep moving the goal posts. What exact problem would need to be solved by LLMs to convince you that they DO discover novel solutions?

I'm more interested why you think my understanding is flawed honestly. I thought I distilled it decently well in two sentences. The bottom line is, in this hyperdimensional space you can find relationships that are not easily distinguished by human minds, but the corpus is still fixed, a llm can't truly know anything beyond its training data.

> Think of it as 1000+ ways to think of that word/token I assume you used 1000 because that's in the ballpark of the vector size. But these are not independent scalars, like each might store a certain property. Just like in 2D you can have 4 quadrants (or subdivide further), with a vector of size 1000 you can encode an insane amount of meaning. > Those meanings are baked in at training time. So, LLMs might be able to cross-reference them and solve a class of problems that flew under our radar, but can't come up with revolutionary theories that were never in the training set. There's a lot of jumping to conclusions here, but I'll try to answer more generally. This idea of how LLMs work is mostly to build an intuition, like with a CNN you'd say imagine a layer does edge detection, and so on. And to some degree you can detect those kinds of behavior, but a NN is a VERY general architecture. It needn't work like you say, it can calculate any function and running under a loop and a scratchpad (basically an agent) is turing complete. Even ignoring that, this part is misleading > Those meanings are baked in at training time. Being baked in at training time does not mean it didn't build novel meanings at training time. This is even more significant when you take into account post training RL. A simple proof that transformers can generate novel, superhuman solutions, is that you can build a transformer based chess bot, feed it 0 human games, and train it with RL until it can beat any human, completely novel and unconstrained by human gameplay (because it would've never seen it). You can do that with any task that's verifiable, like coding or math. (Also as a separate fact, as long as a task is easier to verify than solve (basically always), you have somewhat of a million monkeys with a typewriter, and with temperature sampling the model might eventually stumble it's way onto a solution.)

unify general relativity with quantum mechanics. The continuum hypothesis. The traveling salesman problem in polynomial time.

I think it's cool how in a decade we went from "Neural networks will never be able to understand this sentence that's obvious to humans" to "LLMs must be able to solve problems that humanity hasn't been able to after almost a century, and that might even be unsolvable"

it can operate at the level of a mere mathematics professor, who everyone knows are barely conscious, basically automatons. wake me up when it's Einstein

The continuum hypothesis was proven independent of ZFC over sixty years ago, I think even GPT2 could have told you that much.

I don't see how any of this follow. Yes, the LLMs will learn the "meaning" (here narrowly defined as relative configuration in the embedding space) of vectors that correspond to tokens in whatever tokenizer is used to feed into them. But that vector space is not discrete, and nothing precludes the model from internally operating on other vectors that it never saw in training, based on how they relate to those vectors which it did see.

We have yet to see evidence of proper generalization AFAIK. Examples such as this proof are the closest I'm aware of. I haven't read this one in detail yet but the other examples I've seen have been (upon examination) much closer to an (absurdly) deep literature search than to novel thought. Obviously that doesn't mean we won't eventually achieve novel thought, or even that the current form is fundamentally incapable of it, merely that we've yet to see evidence of it and thus the default assumption is that we aren't there yet.

The burden of proof is the other way

In general, most researchers already incorporate LLM into their workflows, as it is quite good at context search. However, the relevant training data is based on the collective works of the field of experts. Collecting current data on that work is what makes the LLM sound relevant, and any improvement of the LLM model requires frequent new data from both researchers and the chat bot users themselves. LLM are not real "AI", and anyone that says otherwise is selling people something. To phrase this differently, LLM companies conduct unauthorized targeted intelligence gathering on peoples work, codify that act of plagiarism or theft as MoE documentation, and sell unaccountable token output to other users. There is a reason output becomes more nonsensical as "AI" companies try to use dynamic weight granularity and conceptual compaction. It is not necessarily "AI" hallucinations, but rather people fooling themselves into believing smart people are no longer needed if they willingly become a hapless exploited data source caste. This simply isn't true, as people will leave the field for awhile. The LLM business model regularly requires copyright theft and plagiarism to persist. It will not magically become sentient/AGI/less-stupid, as these algorithms have been operating for over 40 years. What has changed is the scale of the deployment, data pool size, and the energy consumed. Scientists are still necessary, as they create the world models LLM try to guess at by statistical inference. Hype and FUD ahead of an IPO for a highly dubious revenue company is expected. We look forward to the low cost liquidated GPU hardware in the near future. =3

The Ouroboros in western mythology is a cautionary tale about the uselessness of the first perfect immortal being, and why humans should suffer our imperfections with insightful grace. The concept also made a great Red Dwarf episode. LLM are more like the Mechanical Turk trick, but the persons inside the machine running the con is unaware of how their actions affect the confounded observers. Have a wonderful day =3

Your model of what AI is good at is wrong. Generative AI is not good at wandering off into novel esoteric abstract corners while maintaining correctness, it is good at things that are close to its training data. I suspect that humans will long outperform AI in the domain of "novel esoteric abstract useless math" whereas AI will outperform humans in the domains of (1) making connections between already-well-understood concepts, things that seem obvious in retrospect but which no human figured out just because of the accidents of what people happened to focus on, and (2) proving things that require long, tedious, intellectually unsatisfying calculations, which would cause a human mathematician to give up for boredom.

Consider that since an LLM is really just an large encoding of data, the "proof" is in there already. All further work on it is effectively only rearranging words. Then all math an LLM is capable of is "done" and we have the "proof" in the LLM which by your definition is now "MUCH easier to understand" and this work is somehow sufficient. Do you see the problem with your reasoning?

You're confusing "contains information" with "has produced a result." A proof being latent in an LLM is no more significant than a proof being latent in a book, a theorem prover, or the axioms themselves. Einstein's papers were latent in the genetic code of his parents and the environment of his time. That doesn't mean general relativity was "already done" before Einstein was born. By your logic, no computation has ever accomplished anything because the output was always implicit in the inputs. The entire purpose of computation is extracting information from representations where it's difficult to see into representations where it's easy to see. So no, this isn't a problem with the original reasoning. It's a problem with yours.

The way humans beat AI, at least until AGI, is to innovate conceptually. If what you do is simply rearrange existing concepts then your work will likely get replaced by AI. Coding is a good example of this, you put documented commands, with known behaviours, in sequences that produce defined outcomes. That is the perfect job for AI, if we can solve their irrational desire to please and hallucinate along the way. AI is simply not able to innovate, only combine.

Understanding is and always has been the "hard" bottleneck. In programming work, if one drops understanding and eg let's an agent write code with only superficial human review or none at all, I believe that they can easily get 100x fast or more, the main question being whether the process collapses some point due to sloppy code. In research fields like mathematics, skipping understanding is not something that can be done without a radical reconstruction of what mathematics (as a process/activity/field) is. It sounds plausible that LLMs help generate insights that humans have missed. But there are many open questions, eg the rate of generating insightful vs uninsightful but plausible statements, which can affect how useful they will be, and of course "open"ai has no incentive to share how much effort/cost (tokens and/or human-review) had been put into investigating erdos problems before coming up with this solution.

Well, if progress in LLMs will steadily continue over next 5 years, then models will be so powerful that there will be no longer place for (most of) human researchers in math (remember that 5 years ago there was no chatgpt!). But I think it's more likely that progress will stall and then open models will catch up to frontier models and almost everyone will be able to afford them.

Seems way too binary a statement. I am guessing you mean "frontier LLMs". Small models keep getting better and better and if you make domain specific ones, it will likely be even smaller. Companies renting smaller LLMs or using enterprise models might very well remain in the future. Consumers getting LLMs whose performance dont improve (think gpt 6 forever on premium or gpt 4.x on a cheap tier) might well become a thing.

I'm curious about whether we will start discovering new maths in the next few years that provide insight into unsolved CS or Physics problems!

Mathematician Ken Ono ( https://en.wikipedia.org/wiki/Ken_Ono ) gives a well nuanced viewpoint on AI in Mathematics (and more) - https://www.youtube.com/watch?v=jGZOi-7haCw He states that he struggled to come up with problems which would be challenging for AI to solve (at the below site) and thus forced to accept that mathematicians have to rethink their profession. FrontierMath: Benchmarking AI against advanced mathematical research by Epoch AI - https://epoch.ai/frontiermath As a follow up to the above, see "First Proof: Mathematicians Putting AI to the Test" featuring eminent mathematicians - https://www.youtube.com/watch?v=AaICCTpkI7Q

""" However, the declaration argues math is more than a machine for producing correct answers. The discipline, its authors believe, is a deeply human endeavor built on creativity, understanding, collaboration, and the pursuit of knowledge for its own sake. Those values often clash with the incentives driving AI development. “The tech industry proceeds in accordance with commercial logic, which is antithetical to the values of mathematics,” declaration co-author Michael Harris of Columbia University told The New York Times. """ I mean, what field doesn't? Everyone works to make money. Slightly unrelated, but, their website " https://leidendeclaration.ai/ " itself gives an eerie feeling of being built by Sonnet. That color scheme and the layout is what Sonnet chooses by default most of the times.

> For years, AI researchers have used math as a proving ground for their models. For years?

AI is the interpolation of the human corpus. Is it suprising AI recombines sucessfully where human attention has not explored all plausible solutions? N,o not especially. The key fallacy is that AI is other than human. This is really no different from computer proofs, e.g. 4 color theorem. The fact the prompt is not linked to the solution by individual human intention alone does not make the solution less human in origin.

I still don't understand how "AI" is ready for serious use beyond entertainment purposes Every time I ask ChatGPT to make a table for a subject I know well, I will find an error in one of the results and it is very confident about it until I question it in detail Every time I ask ChatGPT for nutritional breakdown of some dense food source and give it a quantity like 8 ounces and ask for the weight of each ingredient, the weights will be wrong and add up to more than the original weight of 8 ounces These are variations of the old "how many Rs in strawberry" problem, it's still not solved, "AI" cannot reassemble a complex problem properly A lot of what it tells me in detail about some subjects sounds suspiciously like Reddit posts reassembled out of order

Two things that I would recommend trying out if you're interested in exploring this further: 1. If you're not paying for a model, the results will be worse. That sucks but the free access models are just not very good for anything where you need to trust the output, even for basic queries. 2. More important than #1 is access to tool use. If the LLM is just producing a nutritional breakdown from its weights, it's almost always going to be wrong. If the LLM is allowed to break the problem down into deterministic steps, it will do a lot better. In the nutritional breakdown case, an LLM with search + tool access can pretty easily break the problem down: - Searching the web for a recipe or ingredient breakdown for the food - Searching the web for nutritional qualities of each ingredient per some volume of the ingredient - Writing and running a script with e.g. Python that takes in the recipe's projected serving output, the desired serving size, the amount of each ingredient etc, and scales the ingredients to match the desired serving size, and sums the nutritional qualities of the scaled ingredients. I've tried this specific case with Claude + Gemini for my own purposes and they both handle it very well. The challenge currently is that the models will not always arrive at this approach when provided with an ambiguous prompt; sometimes they will, but sometimes they'll just vomit up a fully autocompleted response from their weights. Being more specific in the prompt or defining a skill that details the intended approach lets you get more useful + deterministic results while still taking advantage of the fuzzy glue that LLMs can provide here between steps. Same with the classic strawberry r-counting case. IIUC LLMs have trouble with this because of how training data is tokenized, but any LLM will have no trouble farming out to e.g. > echo -n "strawberry" | grep -o "r" | wc -l > 3

There are basically two kinds of applications. One is where you want to correctly solve the problem at least 99 out of 100 times. LLMs generally don't (and not everybody realizes that) so there are a lot of debates and research around how useful and reliable they are or how to make them so. The other kind of application is where you can try 100 times and you only need to be right once. Solving a mathematical research problem is like that.

I will argue that AI and flood of low quality slop makes genuine human work more valuable, not less. The ability to clearly outmatch trillion dollar machines is a very unique satisfaction. I even write ordinary internet comments with an intention to make them clearly better and more fun to read than boring Claude output.