For every interesting problem AI solves there are a long tail of really dumb things that AI performs that humans would never do. Some days I am in awe of one-shot magic eight-ball output and other days I'm so frustrated by the sheer stupidity of what it produces. It remains to be seen whether that long tail of stupidity can ever be resolved in the current form of LLMs.

At some point I was thinking that maybe I am too hard on the AI and that humans routinely produce exceptionally stupid code, however after a while I've realized that this is only partially true. AI produces a class of mistakes that humans would almost certainly not make because creating even the context of the mistake would require a level of skill that would preclude such mistakes. It's like if you took a mid/senior engineer and randomly lobotomized them mid-task.

We might be undervaluing consistency. You can plan for reliably stupid. Harder to rely on Often Smart, but Unpredictably Stupid.

I supervise quite a few Masters students. In my particular setting, believe me, LLM stupid for the top three chatbots is easier to work with than real human stupid now. We passed that threshold earlier this year.

It’s like a constant game of two truths and a lie.

Very much like gambling, you can hit the jackpot, or just have continuous near (and far) misses.

A million monkeys at typewriters, but with momentary runs of extreme luck/brilliance.

I’m curious, do writers and authors still really care about AI? I think by now most people are completely put off by AI slop, the value of AI writing or image generation is basically zero So I suspect that the cloud will pass on math too, initial demos get extrapolated and people get worried but in the end slop is slop and serious people aren’t getting replaced or even threatened.

This is quite a way to admit that you don't have any writers or artists in your social group. It has absolutely gutted jobs in these industries, and will continue to do so. If you think 'most people are completely put off by AI slop', you're living in a blessed bubble because: most people cannot even tell that the slop is slop, and are happy to engorge themselves on it.

There have been numerous studies by now showing that most people cannot reliably distinguish "slop" from the real thing, and that many genuinely prefer the slop even.

It's not that it's incomprehensible, it's that it appears to be wrong.

I see any kind automation as a good thing as long as it is reliable enough. We stopped copying books manually a long time ago, and the craft was lost, most of us can't do complex calculations manually etc. but it does not matter as long as we can rely on calculators and computers to do it. At this stage, the current wave of AI is not reliable enough that it would be safe to lose the abilities it can replace. The failures modes are often turned into memes and jokes, but they are the thing we should really pay attention to, IMO.

I still don't understand how "AI" is ready for serious use beyond entertainment purposes Every time I ask ChatGPT to make a table for a subject I know well, I will find an error in one of the results and it is very confident about it until I question it in detail Every time I ask ChatGPT for nutritional breakdown of some dense food source and give it a quantity like 8 ounces and ask for the weight of each ingredient, the weights will be wrong and add up to more than the original weight of 8 ounces These are variations of the old "how many Rs in strawberry" problem, it's still not solved, "AI" cannot reassemble a complex problem properly A lot of what it tells me in detail about some subjects sounds suspiciously like Reddit posts reassembled out of order

Two things that I would recommend trying out if you're interested in exploring this further: 1. If you're not paying for a model, the results will be worse. That sucks but the free access models are just not very good for anything where you need to trust the output, even for basic queries. 2. More important than #1 is access to tool use. If the LLM is just producing a nutritional breakdown from its weights, it's almost always going to be wrong. If the LLM is allowed to break the problem down into deterministic steps, it will do a lot better. In the nutritional breakdown case, an LLM with search + tool access can pretty easily break the problem down: - Searching the web for a recipe or ingredient breakdown for the food - Searching the web for nutritional qualities of each ingredient per some volume of the ingredient - Writing and running a script with e.g. Python that takes in the recipe's projected serving output, the desired serving size, the amount of each ingredient etc, and scales the ingredients to match the desired serving size, and sums the nutritional qualities of the scaled ingredients. I've tried this specific case with Claude + Gemini for my own purposes and they both handle it very well. The challenge currently is that the models will not always arrive at this approach when provided with an ambiguous prompt; sometimes they will, but sometimes they'll just vomit up a fully autocompleted response from their weights. Being more specific in the prompt or defining a skill that details the intended approach lets you get more useful + deterministic results while still taking advantage of the fuzzy glue that LLMs can provide here between steps. Same with the classic strawberry r-counting case. IIUC LLMs have trouble with this because of how training data is tokenized, but any LLM will have no trouble farming out to e.g. > echo -n "strawberry" | grep -o "r" | wc -l > 3

There are basically two kinds of applications. One is where you want to correctly solve the problem at least 99 out of 100 times. LLMs generally don't (and not everybody realizes that) so there are a lot of debates and research around how useful and reliable they are or how to make them so. The other kind of application is where you can try 100 times and you only need to be right once. Solving a mathematical research problem is like that.