A few things, I think you’re missing the point here - most tasks do not require the latest frontier models, even if they are a magnitude more intelligent (we don’t actually know if that will be the case). Current Gemini flash is cheap, fast, and pretty capable with good guidance for most tasks - now that companies pay API costs instead of a subscription they will be setting restrictions on token use to not have their budget explode (like Uber in this submission), that’s a strong incentive to NOT use expensive models, and limit their thinking budget - there is competitive pressure from China and others who can offer very decent performances at a fraction of the token price - the price of tokens for the frontier models is likely to go up, but the price to access older models is what depreciates! The overall price per token is going down now that we are in a new world where companies understand that token maxing is one of the stupidest concept ever created by humankind.

As long as the demand for GPUs keeps increasing, there are more data centers being built to house them. When you have waitlists for many many months for Blackwell GPUs, keeping the old ones around as long as customers are willing to pay for them is great. If I as a customer have a use case for a machine learning model I developed awhile ago, so an insect identification model, I had an ML researcher/eng develop it back in 2019, and it runs fine on a 2018-era T4 GPU (NVidia 2080 era), why mess with it?

We aren't talking about insect identification models from 2019.

What do you think are running on the T4 GPUs in AWS? A lot of the use cases I know of for them are mid-level computer vision models that don't need to be frontier level.

I can no longer edit this, but want to expand on my comment. I've seen those vision researchers want to train on H100s at the time and being told know, wait for the T4s. I've seen T4s running BERT models for document classification. When there are enough Blackwells in data centers that H100s are useless for inference by your standards (I don't know if we've arrived there or not yet), there will be people who, say, want to run the Taco Bell ordering chatbot on them. There will be people who have applications that are just fine with Qwen 2.5 who will be happy renting them. There seems to be this crazy consensus that hyperscalers are going to go into their datacenters and throw away their old GPUs. The reality is they have a ton of paying customers for them. And there may be insect identification apps from 2019 that say "you know what? H100s have gotten cheap enough I can use a VLLM so the user can describe where they saw the insect too", or the McDonald's website support chatbot developers say "Hey, the bigger cheapers have gotten cheap enough we can upgrade our models to Qwen 2.5". The frontier level GPUs in e.g. AWS have a huge premium. When the newer generations come out, they will be able to cut prices to a bit of a premium over the operational costs and still make a profit, and there are a ton of down-market customers who will be interested, who aren't willing to try to outbid Anthropic for Blackwells.

Using a shittier model is just more work for the user, I’m not sure why anyone does it, unless they’re playing with it like a toy.

> more work for the user Model routers allow this to happen automatically without any more work by the user. > a shittier model A ton of tasks don't require the most expensive frontier models, etc. > I’m not sure why anyone does it 1. Faster solutions from the LLM - also reduces employee costs of having the employee waiting on the LLM 2. Avoiding things like the half-billion dollar per month bill for a single company’s LLM use recently reported in Axios

I sometimes let Claude Opus create plans, DeepSeek v4 pro implements and writes tests. Claude reviews and corrects. Saves like $2-3 per session. Same quality code.

What you call a shittier model is what was considered frontier and fantastic one generation ago…

Today's frontier models will be tomorrows low-end option. I think whatever model you are using today will be less expensive to use a year or two from now.

Last year's o3 was more expensive than 5.5 is. Whatever model we are using now is probably be more expensive than next year's leading models will be.

I wonder if I could start a US-based company with good data regulation and just serve open-weight models at a competitive price. I feel like the real barrier is just that most companies willing to adopt AI usage enough to make it worth it at this point don't want to be using inferior models.

Deepseek has some models in Bedrock. There is definitely a huge market for a "good enough" model running within the country of the company

An inference only platform selling good open weight model inference without the research overhead could capture a-lot of market for lower size model uses (haiky, gemeni flash). Diffusion-transformers and clever cashing can drop inference even lower, which is improving at a high rate. The biggest reason large models are un-attainable for local applications is the lack hardware with large amount of unified/graphics memory (and the cost of the platforms that do). Once the memory slog goes back to normal and hardware manufacturers adapt to demand, we may see consumer hardware with large memory capacity effectively opening the door for slow but usable frontier model inference (assuming improvements in model efficiency and compute capacity) At that point, inference becomes a race to the bottom. The large labs hope they can attain a leap in capability (which is increasingly looking bleak, with a average catch-up of just a few months) or market dominance through integration (integration in platforms and OS, exclusive deals with companies or governments). For coding agents, i suspect no player will manage lock in enough market to enforce pricing much higher than the true inference cost, and catering to programmers becomes an unsustainable proposition. We will instead be further hit with a lot of AI integrated into our other tooling costs, such as GitHub, Microsoft suite, G-suite, forcing in AI functions as a value-ad into the total cost without giving the option to exclude them. (using their market position)

How many more months do we need to wait, until big companies realize that flash models work just fine if you: 1) Don't ask LLMs for big changes 2) Review everything and point them in the right direction Large models still suck at big changes, they produce questionable architecture and you still have to review the code, if your project is serious enough. The codebase quickly become a mess, if you don't pay enough attention. Does not matter which model. So why bother with big models, when flash models are 10x cheaper and much faster to iterate under guidance? Large models can be used for security and bug audits. Flash models work almost the same for changes under 300 LOC when you dictate how you want your code to look.

It also presupposes that open models will bridge that gap towards opus4.5, which was really when I drank the AI coding koolaid

I wonder to what extent models should figure out which model to forward a query to. Or perhaps the big models could learn the difference between an easy and a hard question and charge accordingly? Perhaps, if it can measure complexity, even generate a quote? Small models are fine for small coding tasks but I don't see why big ones can't be broken down most of the time.

This a thousand times. The bigger models also have a habit of overcomplicating things.

They were proposing a solution.. To use flash models and use them in a way that best amplifies your work.

I'm legit annoyed at opus 4.8 at any setting above 4.8. I believe it can be great for vibe coding, but mundane day work? Hell no, I'd rather work with Haiku. It's too slow, checks too many things, it's annoying as hell.

Every employee doesn't need $1k in token spend per month, either. That kind of spend makes sense for technical workers in r+d. Most other workers are served fine by $20-30 worth of tokens on a budget model. You don't need Opus to help support write emails.

No, but you do want Opus-tier models to do desktop and office software automation (think about people who intensely use Excel and the like). Actually those might take even more tokens that coding in a lot of cases. Why do you think Claude Cowork is successful, and why do you think Codex is leaning so hard into Computer use?

Decent vs best-money-can-buy. Further, a self-hosted LLM will be much slower.

I think we're all past the "bet-money-can-buy" stage. The most expensive models are an order of magnitude more expensive than the middle ground ones, so you need to be selective about what you run where. And with a bit of careful routing - there isn't a lot stopping you sending the hard stuff to a cloud model and the average stuff to an on prem model.

Is interactive use for coding something that actually works today? With unsafe mode, even frontier hosted models are slow enough I end up just tabbing out to work on other tasks. It would need to be much faster if I am to sit and stare at it while it churns. Local models might be a lot slower but workflow-wise it doesn't change much for me.

I would expect the overwhelming majority of output tokens would not be the actual code but used for analysis, reasoning, testing and iteration. If you only use the agent for autocomplete then yes, the calculation is probably different.

There's models for every price point. What was SOTA and stupid expensive to run a year ago is a cheap flash model today.

If you believe a 128gb machine that is essentially DGX Spark in a laptop chassis can run models comparable to SOTA you either never ran open models on hard tasks, or you aren't scratching the surface of SOTA closed LLM capability in how you're using them.

Can you show me an example of a hard task that can't be achieved using light models? When we don't want the model to work on autopilot without reviewing the code at all. Even SOTA models will produce garbage code, if you don't guide them all the time. Hard tasks require a lot of guidance and code reviewing, unless you are creating another throw away project where correctness, maintainability and code understanding does not matter.

I am wondering more and more if this becomes true as these smaller models take off. I might be old fashioned but I have yet to crack the workflows some of the hype people spout like Claude codes Boris where he and others talk about running hundreds of agents overnight. I have still found the sweet spot for me is using LLMs but I am still in the drivers seat.

$1.5kpm for SOTA. 128gb you run DSV4 Flash.

You can't get an edge using local models, these guys may have competitors that will spend on SOTA models. They won't likely ever consider local machines even for some offloading scenarios, the complexity and costs will be even higher.

Why isn't self hosting (even just renting a GPU server, not necessarily on premise) at large companies or hosting via something like together AI to run the open weight models not more common? I've tried the open weight models and the premium models like Opus and Gemini Pro, and I find that the latter are a little better, but not nearly to the degree to justify the extreme price difference, since the differences largely don't matter for what I've tried them for, and I expect that many other users likely have similar use cases.

A lot of talk about cheaper models here. Just curios, is there any non-Anthropic model that can do UI well? GPT-5.5 is laughably bad, and I'm never restarting my Anthropic subscription after their 6-month sprint of gaslighting, even if opus was really good at UI.

Aren’t the Chinese labs quickly turning them into a commodity? The open-weight models will have a steady race to the bottom on inference costs just by dint of competition between providers. They aren’t at the frontier yet, but they are rapidly eating the flash market.

Yeah, that's not going to work if you can get e.g. 80% of value by using 10-20x or more cheaper open models. At some point it would just make sense for large companies to rent compute and deploy their version of DeepSeek or whatever (if they don't trust Chinese providers)