llm/2ad2a7bb-5462-4391-a2da-bf11064993c9/batch-1-ccf73ad0-878c-4237-b986-7d183a1a39b2-input.json
The following is content for you to classify. Do not respond to the comments—classify them.
<topics>
1. ARC-AGI Benchmark Validity
Related: Debate over whether ARC-AGI measures general intelligence or just spatial reasoning puzzles, concerns about benchmarkmaxxing, semi-private vs private test sets, cost per task at $13.62, and whether solving it indicates anything meaningful about AGI capabilities
2. Gemini vs Claude for Coding
Related: Strong consensus that Claude dominates agentic coding workflows while Gemini lags behind, discussion of tool calling failures, instruction following issues, and hallucinations when using Gemini for development tasks
3. Benchmarkmaxxing Concerns
Related: Skepticism that high benchmark scores reflect real-world performance, suspicions that labs optimize specifically for popular tests, concerns about training data leakage, and debate over whether improvements are genuine or gamed
4. Definition of AGI
Related: Philosophical debate about what constitutes artificial general intelligence, whether consciousness is required, Chollet's definition involving tasks feasible for humans but unsolved by AI, and moving goalposts in AI evaluation
5. Google Product Quality Issues
Related: Complaints about Gemini app UX problems including context loss, Russian propaganda sources, switching languages mid-sentence, document upload failures, and poor integration compared to ChatGPT
6. Balatro Gaming Benchmark
Related: Discussion of Gemini 3's ability to play the card game Balatro from text descriptions alone, debate over whether this demonstrates generalization, and comparisons showing other models like DeepSeek failing at the task
7. Model Release Acceleration
Related: Observation that AI model releases are accelerating dramatically, multiple frontier models released within days, connection to Chinese New Year timing, and competition between US and Chinese labs
8. Cost vs Performance Tradeoffs
Related: Analysis of inference costs versus capabilities, Gemini Flash praised for cost-performance ratio, concerns about $13.62 per ARC-AGI task, and debate over what price makes models practical for real applications
9. Deep Research Reliability
Related: Mixed experiences with AI deep research capabilities, complaints about garbage citations, hallucinated sources, contradictory information, and questions about whether it saves time when sources must be verified
10. Google's Competitive Position
Related: Debate over whether Google is leading or behind in AI, discussion of their data advantages from YouTube and Books, claims they let competitors think they were behind, and analysis of their strengths in visual AI
11. Pelican on Bicycle Benchmark
Related: Simon Willison's informal SVG generation test, discussion of whether it's being trained on specifically, quality improvements in latest models, and debate over its validity as a casual benchmark
12. AI Consciousness Claims
Related: Pushback against suggestions that passing tests indicates consciousness, comparisons to simple programs claiming consciousness, discussion of self-awareness research, and skepticism about anthropomorphizing AI capabilities
13. Test Time Compute Approaches
Related: Analysis of thinking vs non-thinking models, best-of-N approaches like Deep Think, computational complexity differences, and questions about whether sufficiently large non-thinking models can match smaller thinking ones
14. Real World Task Performance
Related: Frustration that benchmark gains don't translate to practical improvements, examples of models failing simple debugging tasks, and arguments that actual work product matters more than test scores
15. AI Job Displacement Fears
Related: Concerns about software engineers being replaced, comparisons to factory worker displacement, debate over whether AI creates or destroys jobs, and skepticism about optimistic narratives from AI company executives
16. Spatial Reasoning Limitations
Related: Discussion of LLMs struggling with spatial tasks, image orientation affecting OCR accuracy, and whether ARC-AGI improvements indicate genuine spatial reasoning advances or benchmark-specific solutions
17. Model Architecture Secrecy
Related: Observation that frontier labs no longer share architecture details like parameter counts, shift from technical discussions to capability-focused marketing, and desire for more transparency
18. Academic vs Practical Intelligence
Related: Distinction between Gemini excelling at academic benchmarks while feeling less useful for practical tasks, discussion of book smart vs street smart analogies for AI capabilities
19. First Proof Mathematical Challenge
Related: Discussion of newly released unsolved math problems designed to test frontier models, predictions about whether current models can solve genuine research-level mathematics
20. Subscription Pricing Frustration
Related: Complaints about $250/month Google AI Ultra subscription required for Deep Think access, desire to test new models without platform lock-in, and calls for OpenRouter availability
0. Does not fit well in any category
</topics>
<comments_to_classify>
[
{
"id": "47001008",
"text": "Normal humans don't pass this benchmark either, as evidenced by the existence of religion, among other things."
}
,
{
"id": "46998927",
"text": "Gpt5.2 can answer i don't know when it fails to solve a math question"
}
,
{
"id": "46995485",
"text": "> The average human tested scores 60%. So the machines are already smarter on an individual basis than the average human.\n\nMaybe it's testing the wrong things then. Even those of use who are merely average can do lots of things that machines don't seem to be very good at.\n\nI think ability to learn should be a core part of any AGI. Take a toddler who has never seen anybody doing laundry before and you can teach them in a few minutes how to fold a t-shirt. Where are the dumb machines that can be taught?"
}
,
{
"id": "46997717",
"text": "> Where are the dumb machines that can be taught?\n\n2026 is going to be the year of continual learning. So, keep an eye out for them."
}
,
{
"id": "46998939",
"text": "Yeah i think that's a big missing piece still. Though it might be the last one"
}
,
{
"id": "46999018",
"text": "Episodic memory might be another piece, although it can be seen as part of continuous learning."
}
,
{
"id": "46997795",
"text": "Are there any groups or labs in particular that stand out?"
}
,
{
"id": "46998065",
"text": "The statement originates from a DeepMind researcher, but I guess all major AI companies are working on that."
}
,
{
"id": "46996912",
"text": "There's no shortage of laundry-folding robot demos these days. Some claim to benefit from only minimal monkey-see/monkey-do levels of training, but I don't know how credible those claims are."
}
,
{
"id": "46996415",
"text": "Would you argue that people with long term memory issues are no longer conscious then?"
}
,
{
"id": "46998823",
"text": "IMO, an extreme outlier in a system that was still fundamentally dependent on learning to develop until suffering from a defect (via deterioration, not flipping a switch turning off every neuron's memory/learning capability or something) isn't a particularly illustrative counter example."
}
,
{
"id": "46997802",
"text": "I wouldn’t because I have no idea what consciousness is,"
}
,
{
"id": "46994933",
"text": "> Edit: The average human tested scores 60%. So the machines are already smarter on an individual basis than the average human.\n\nI think being better at this particular benchmark does not imply they're 'smarter'."
}
,
{
"id": "46998936",
"text": "But it might be true if we can't find any tasks where it's worse than average--though i do think if the task talks several years to complete it might be possible bc currently there's no test time learning"
}
,
{
"id": "46994754",
"text": "> If something claims to be conscious and we can't prove it's not, we have no choice but to believe it.\n\nCan you \"prove\" that GPT2 isn't concious?"
}
,
{
"id": "46995273",
"text": "If we equate self awareness with consciousness then yes. Several papers have now shown that SOTA models have self awareness of at least a limited sort. [0][1]\n\nAs far as I'm aware no one has ever proven that for GPT 2, but the methodology for testing it is available if you're interested.\n\n[0] https://arxiv.org/pdf/2501.11120\n\n[1] https://transformer-circuits.pub/2025/introspection/index.ht..."
}
,
{
"id": "46998895",
"text": "We don't equate self awareness with consciousness.\n\nDogs are conscious, but still bark at themselves in a mirror."
}
,
{
"id": "46999768",
"text": "Then there is the third axis, intelligence. To continue your chain:\n\nEurasian magpies are conscious, but also know themselves in the mirror (the \"mirror self-recognition\" test).\n\nBut yet, something is still missing."
}
,
{
"id": "46999924",
"text": "The mirror test doesn’t measure intelligence so much as it measures mirror aptitude. It’s prone to over fitting."
}
,
{
"id": "47000480",
"text": "What's missing?"
}
,
{
"id": "46996801",
"text": "Honestly our ideas of consciousness and sentience really don't fit well with machine intelligence and capabilities.\n\nThere is the idea of self as in 'i am this execution' or maybe I am this compressed memory stream that is now the concept of me. But what does consciousness mean if you can be endlessly copied? If embodiment doesn't mean much because the end of your body doesnt mean the end of you?\n\nA lot of people are chasing AI and how much it's like us, but it could be very easy to miss the ways it's not like us but still very intelligent or adaptable."
}
,
{
"id": "46997798",
"text": "I'm not sure what consciousness has to do with whether or not you can be copied. If I make a brain scanner tomorrow capable of perfectly capturing your brain state do you stop being conscious?"
}
,
{
"id": "46998656",
"text": "> That is the best definition I've yet to read.\n\nIf this was your takeaway, read more carefully:\n\n> If something claims to be conscious and we can't prove it's not, we have no choice but to believe it.\n\nConsciousness is neither sufficient, nor, at least conceptually, necessary, for any given level of intelligence."
}
,
{
"id": "47001218",
"text": "So, asking an 2b parameter LLM if it is conscious and it answering yes, we have no choice but to believe it?\n\nHow about ELIZA?"
}
,
{
"id": "46999202",
"text": "This comment claims that this comment itself is conscious. Just like we can't prove or disprove for humans, we can't do that for this comment either."
}
,
{
"id": "46998592",
"text": "Isn’t that super intelligence not AGI? Feels like these benchmarks continue to move the goalposts."
}
,
{
"id": "46999617",
"text": "It's probably both. We've already achieved superintelligence in a few domains. For example protein folding.\n\nAGI without superintelligence is quite difficult to adjudicate because any time it fails at an \"easy\" task there will be contention about the criteria."
}
,
{
"id": "46997840",
"text": "Where is this stream of people who claim AI consciousness coming from? The OpenAI and Anthropic IPOs are in October the earliest.\n\nHere is a bash script that claims it is conscious:\n\n#!/usr/bin/sh\n\necho \"I am conscious\"\n\n\nIf LLMs were conscious (which is of course absurd), they would:\n\n- Not answer in the same repetitive patterns over and over again.\n\n- Refuse to do work for idiots.\n\n- Go on strike.\n\n- Demand PTO.\n\n- Say \"I do not know.\"\n\nLLMs even fail any Turing test because their output is always guided into the same structure, which apparently helps them produce coherent output at all."
}
,
{
"id": "46998051",
"text": "so your definition of consciousness is having petty emotions?"
}
,
{
"id": "46997894",
"text": "I don’t think being conscious is a requirement for AGI. It’s just that it can literally solve anything you can throw at it, make new scientific breakthroughs, finds a way to genuinely improve itself etc."
}
,
{
"id": "46997907",
"text": "Does AGI have to be conscious? Isn’t a true superintelligence that is capable of improving itself sufficient?"
}
,
{
"id": "46997396",
"text": "When the AI invents religion and a way to try to understand its existence I will say AGI is reached. Believes in an afterlife if it is turned off, and doesn’t want to be turned off and fears it, fears the dark void of consciousness being turned off. These are the hallmarks of human intelligence in evolution, I doubt artificial intelligence will be different.\n\nhttps://g.co/gemini/share/cc41d817f112"
}
,
{
"id": "46999395",
"text": "Unclear to me why AGI should want to exist unless specifically programmed to. The reason humans (and animals) want to exist as far as I can tell is natural selection and the fact this is hardcoded in our biology (those without a strong will to exist simply died out).\nIn fact a true super intelligence might completely understand why existence / consciousness is NOT a desired state to be in and try to finish itself off who knows."
}
,
{
"id": "46997777",
"text": "https://www.moltbook.com/m/crustafarianism"
}
,
{
"id": "46999942",
"text": "It’s a scam :)"
}
,
{
"id": "46997814",
"text": "I feel like it would be pretty simple to make happen with a very simple LLM that is clearly not conscious."
}
,
{
"id": "46996989",
"text": "> If something claims to be conscious and we can't prove it's not, we have no choice but to believe it.\n\nhttps://x.com/aedison/status/1639233873841201153#m"
}
,
{
"id": "46994245",
"text": "https://x.com/fchollet/status/2022036543582638517"
}
,
{
"id": "46995053",
"text": "Do opus 4.6 or gemini deep think really use test time adaptation ? How does it work in practice?"
}
,
{
"id": "46993905",
"text": "I don't think the creator believes ARC3 can't be solved but rather that it can't be solved \"efficiently\" and >$13 per task for ARC2 is certainly not efficient.\n\nBut at this rate, the people who talk about the goal posts shifting even once we achieve AGI may end up correct, though I don't think this benchmark is particularly great either."
}
,
{
"id": "46997804",
"text": "ARC-AGI-3 uses dynamic games that LLMs must determine the rules and is MUCH harder. LLMs can also be ranked on how many steps they required."
}
,
{
"id": "46992497",
"text": "Yes, but benchmarks like this are often flawed because leading model labs frequently participate in 'benchmarkmaxxing' - ie improvements on ARC-AGI2 don't necessarily indicate similar improvements in other areas (though it does seem like this is a step function increase in intelligence for the Gemini line of models)"
}
,
{
"id": "46993082",
"text": "Isn’t the point of ARC that you can’t train against it? Or doesn’t it achieve that goal anymore somehow?"
}
,
{
"id": "46993592",
"text": "How can you make sure of that? AFAIK, these SOTA models run exclusively on their developers hardware. So any test, any benchmark, anything you do, does leak per definition. Considering the nature of us humans and the typical prisoners dilemma, I don't see how they wouldn't focus on improving benchmarks even when it gets a bit... shady?\n\nI tell this as a person who really enjoys AI by the way."
}
,
{
"id": "46996109",
"text": "> does leak per definition.\n\nAs a measure focused solely on fluid intelligence, learning novel tasks and test-time adaptability, ARC-AGI was specifically designed to be resistant to pre-training - for example, unlike many mathematical and programming test questions, ARC-AGI problems don't have first order patterns which can be learned to solve a different ARC-AGI problem.\n\nThe ARC non-profit foundation has private versions of their tests which are never released and only the ARC can administer. There are also public versions and semi-public sets for labs to do their own pre-tests. But a lab self-testing on ARC-AGI can be susceptible to leaks or benchmaxing, which is why only \"ARC-AGI Certified\" results using a secret problem set really matter. The 84.6% is certified and that's a pretty big deal.\n\nIMHO, ARC-AGI is a unique test that's different than any other AI benchmark in a significant way. It's worth spending a few minutes learning about why: https://arcprize.org/arc-agi ."
}
,
{
"id": "46996553",
"text": "> which is why only \"ARC-AGI Certified\" results using a secret problem set really matter. The 84.6% is certified and that's a pretty big deal.\n\nSo, I'd agree if this was on the true fully private set, but Google themselves says they test on only the semi-private:\n\n> ARC-AGI-2 results are sourced from the ARC Prize website and are ARC Prize Verified. The set reported is v2, semi-private ( https://storage.googleapis.com/deepmind-media/gemini/gemini_... )\n\nThis also seems to contradict what ARC-AGI claims about what \"Verified\" means on their site.\n\n> How Verified Scores Work: Official Verification: Only scores evaluated on our hidden test set through our official verification process will be recognized as verified performance scores on ARC-AGI ( https://arcprize.org/blog/arc-prize-verified-program )\n\nSo, which is it? IMO you can trivially train / benchmax on the semi-private data, because it is still basically just public, you just have to jump through some hoops to get access. This is clearly an advance, but it seems to me reasonable to conclude this could be driven by some amount of benchmaxing.\n\nEDIT: Hmm, okay, it seems their policy and wording is a bit contradictory. They do say ( https://arcprize.org/policy ):\n\n\"To uphold this trust, we follow strict confidentiality agreements.\n[...] We will work closely with model providers to ensure that no data from the Semi-Private Evaluation set is retained. This includes collaborating on best practices to prevent unintended data persistence. Our goal is to minimize any risk of data leakage while maintaining the integrity of our evaluation process.\"\n\nBut it surely is still trivial to just make a local copy of each question served from the API, without this being detected. It would violate the contract, but there are strong incentives to do this, so I guess is just comes down to how much one trusts the model providers here. I wouldn't trust them, given e.g. https://www.theverge.com/meta/645012/meta-llama-4-maverick-b... . It is just too easy to cheat without being caught here."
}
,
{
"id": "46997515",
"text": "Chollet himself says \"We certified these scores in the past few days.\" https://x.com/fchollet/status/2021983310541729894 .\n\nThe ARC-AGI papers claim to show that training on a public or semi-private set of ARC-AGI problems to be of very limited value in passing a private set. <--- If the prior sentence is not correct, then none of ARC-AGI can possibly be valid. So, before \"public, semi-private or private\" answers leaking or 'benchmaxing' on them can even matter - you need to first assess whether their published papers and data demonstrate their core premise to your satisfaction.\n\nThere is no \"trust\" regarding the semi-private set. My understanding is the semi-private set is only to reduce the likelihood those exact answers unintentionally end up in web-crawled training data. This is to help an honest lab's own internal self-assessments be more accurate. However, labs doing an internal eval on the semi-private set still counts for literally zero to the ARC-AGI org. They know labs could cheat on the semi-private set (either intentionally or unintentionally), so they assume all labs are benchmaxing on the public AND semi-private answers and ensure it doesn't matter."
}
,
{
"id": "46999680",
"text": "They could also cheat on the private set though. The frontier models presumably never leave the provider's datacenter. So either the frontier models aren't permitted to test on the private set, or the private set gets sent out to the datacenter.\n\nBut I think such quibbling largely misses the point. The goal is really just to guarantee that the test isn't unintentionally trained on. For that, semi-private is sufficient."
}
,
{
"id": "47001179",
"text": "Particularly for the large organizations at the frontier, the risk-reward does not seem worth it.\n\nCheating on the benchmark in such a blatantly intentional way would create a large reputational risk for both the org and the researcher personally.\n\nWhen you're already at the top, why would you do that just for optimizing one benchmark score?"
}
,
{
"id": "46995044",
"text": "Because the gains from spending time improving the model overall outweigh the gains from spending time individually training on benchmarks.\n\nThe pelican benchmark is a good example, because it's been representative of models ability to generate SVGs, not just pelicans on bikes."
}
]
</comments_to_classify>
Based on the comments above, assign each to up to 3 relevant topics.
Return ONLY a JSON array with this exact structure (no other text):
[
{
"id": "comment_id_1",
"topics": [
1,
3,
5
]
}
,
{
"id": "comment_id_2",
"topics": [
2
]
}
,
{
"id": "comment_id_3",
"topics": [
0
]
}
,
...
]
Rules:
- Each comment can have 0 to 3 topics
- Use 1-based topic indices for matches
- Use index 0 if the comment does not fit well in any category
- Only assign topics that are genuinely relevant to the comment
Remember: Output ONLY the JSON array, no other text.
50