Model Capability Skepticism

Doubts that current models can truly handle novel problems or large codebases, concerns about reconstituting training data only

← Back to Harness engineering: Leveraging Codex in an agent-first world

Skepticism toward AI coding often centers on the fear that models merely reconstitute training data rather than solving novel problems, potentially trapping long-lived codebases in a "perpetual MVP" state. Critics argue that while AI excels at routine tasks like scrapers, it frequently struggles with the rigorous demands of reliability, legibility, and the specialized complexity found in large-scale engineering projects. However, this view is challenged by instances of emergent generalization where models transfer knowledge to unfamiliar domains, suggesting capabilities that go beyond simple mimicry. Ultimately, the discourse reflects a sharp tension between those dismissing the technology as marketing hype and those navigating a shifting landscape where AI-led engineering may eventually replace traditional human-AI collaboration.

9 comments tagged with this topic

View on HN · Topics
AI coders are great for making scrapers, possibly because AI companies use their own tools to make an awful lot of scrapers.
View on HN · Topics
I worry most about blindspots with this kind of approach. Let's say that this repository goes on for years, at which point the docs folder is several MB in size. Would Codex be able to think outside of the box? Or would the aggregate of the Markdown content fundamentally cover enough ground to prevent it from thinking of novel new approaches to existing problems?
View on HN · Topics
I personally don't view coding agents making software as "software gotten better" you are comparing a tool and the end result, these are two different things. Agent you use going down and your product going down mean two different things to you customers. I will not deny that we made incredible progress in coding and hell, even design over the past 3.5 years, this technology is here to stay. That being said while I agree that measuring better quality of software is vague (part of the reason it is hard for models as well), there are universal things I believe every engineer will agree on. Reliability, uptime, customer feedback, legibility of your engineering, performance, these are things we often optimized for. Google Maps is a bit of a strawman because neither of us (unless you work on it), knows how much agent code there is, I think it is likely that it's little since it was working fine prior to 2023. I could bring up github reliability as an example, given how much copilot usage they promote at MS, but once again only folks there know for certain. I do, however, see scores of various AI powered SAAS that looks like it is in a perpetual MVP state. I think you are right in that even if agents give us "good enough" results and we can swallow failure rates and our increasingly lesser understanding of what we, or more so model, created, then it is still progress overall, but this is progress not to human-AI collaboration but to AI-only engineering IMO, this is good or bad depending on how you view the future. I'm a scientist and most of code I currently write is somewhere on the intersection of critical software and machine learning, squaring these two is not easy and I guess the way I was taught to reason about engineering informs my opinions on this. Maybe it's just a matter of time before codex can help here in an unconstrained manner as well, but I am skeptical at the moment.
View on HN · Topics
Well, to be fair, the amount of goalpost shifting that is going on is quite intense. AI not being able to work in a "serious" project, and being limited to "toy projects" has been a long standing critique. But also, bigger projects need some amount of loc written and it's a bit silly to pretend that this is not the case or a bad thing. So the answer to the question is roughly: Establishing that an agent can work in a large-ish code base is valuable, because 1) them not being able to do so has been a critique and 2) it's something that is required for a lot of software projects.
View on HN · Topics
I don’t think it’s solvable. And I think Anthropic etc know it. LLMs can only reconstitute things in its training data and they are so hungry they can’t do a good job in long lived codebase full of complexity and novelty. There’s never going to be enough similar code on the open internet.
View on HN · Topics
> LLMs can only reconstitute things in its training data Such as a 4D raytracing engine in Metal? Or integrating APIs for features first released months after their knowledge cut-off date? LLMs have shown an ability to transfer "knowledge" and capabilities across domains, languages, and use-cases outside their training data. Case in point: GPT-2 "learning" to translate English to French and vice versa despite non-English examples having been voluntarily (and almost entirely) removed from the dataset.
View on HN · Topics
Was this in the GPT2 paper?
View on HN · Topics
In "Language Models are Unsupervised Multitask Learners"[0]. Not sure whether it’s "the" GPT-2 paper. 3.7 Translation > Performance on this task was surprising to us, since we deliberately removed non-English webpages from WebText as a filtering step. In order to con- firm this, we ran a byte-level language detector2 on WebText which detected only 10MB of data in the French language […] [0]: https://cdn.openai.com/better-language-models/language_model...
View on HN · Topics
I for one am not protesting because I know that this is bullshit marketing nonsense. Look at reliability metrics of OpenAI, they’re terrible. Everyone knew a long way ahead that it’s a scam, now they’re cranking up pricing and trying to rug pull. There will be a lot of developers who will come out very well once the stock tanks. That’s my two cents