Don't Worry About the Vase Subscribe Sign in AI #128: Four Hours Until Probably Not The Apocalypse Zvi Mowshowitz Aug 07, 2025 57 2 5 Share Brace for impact. We are presumably (checks watch) four hours from GPT-5. That’s the time you need to catch up on all the other AI news. In another week, I might have done an entire post on Gemini 2.5 Deep Thinking, or Genie 3, or a few other things. This week? Quickly, there’s no time. OpenAI has already released an open model. I’m aiming to cover that tomorrow. Table of Contents Also: Claude 4.1 is an incremental improvement , On Altman’s Interview With Theo Von . Language Models Offer Mundane Utility. The only help you need? Language Models Don’t Offer Mundane Utility. Can’t use Claude to train GPT-5. Huh, Upgrades. ChatGPT for government, Gemini for students, Claude security. On Your Marks. More analysis of Psyho’s victory over OpenAI at AWTF. Thinking Deeply With Gemini 2.5 . The power of parallel thinking could be yours. Choose Your Fighter. It won’t be via AWS. Fun With Media Generation. Grok Imagine on the horizon. Optimal Optimization . OpenAI claims to optimize on behalf of the user. Uh huh. Get My Agent On The Line. Is it possible that I’m an AI of agency and taste ? Deepfaketown and Botpocalypse Soon. Do you think this will all end well? You Drive Me Crazy. The psychiatrists of Reddit are noticing the problem. They Took Our Jobs. The last radiologist can earn a lot before turning off lights. Get Involved . UK AISI on game theory, DARPA, Palisade, Karpathy, YC. Introducing. Gemini Storybook, Digital Health Economy, ElevenLabs Music. City In A Bottle . Genie 3 gives us navigable interactive environments. Unprompted Suggestions. Examples belong at the top of the prompt. In Other AI News. Certain data is for the birds. Papers, Please. Attention, you need it, how does it work? The Mask Comes Off . Asking OpenAI seven questions about its giant heist. Show Me the Money. Number go up. A lot. Quiet Speculations. Is America a leveraged bet on AGI? Kind of? Mark Zuckerberg Spreads Confusion. Superintelligence as in Super Nintendo. The Quest for Sane Regulations. Powerful AI is a package deal. Do, or do not. David Sacks Once Again Amplifies Obvious Nonsense. There you go again. Chip City. China released another okay AI model, everybody panic? No. No Chip City . A 100% tariff on importing semiconductors, or you gonna pay up? Energy Crisis. American government continues its war on electrical power. Why? To The Moon. The race to build a nuclear power plant… ON THE MOON. Dario’s Dismissal Deeply Disappoints, Depending on Details. Damn, dude. The Week in Audio. Chen and Pachocki, Hassabis, Patel, Odd Lots. Tyler Cowen Watch. Solid interview on how he’s updated, or not updated. Rhetorical Innovation. How to power past socially constructed objections. Shame Be Upon Them. Sorry for the subtweet. Correlation Causes Causation. Carefully curate collections. Aligning a Smarter Than Human Intelligence is Difficult. Frontier Model Forum. The Lighter Side. The code is 5090. Language Models Offer Mundane Utility The Prime Minister of Sweden asks ChatGPT for job advice ‘quite often .’ Prime Minister Ulf Kristersson (M) uses AI services in his work as Sweden’s highest decision-maker. – I use it quite often myself. If nothing else for a ‘second opinion’. ‘What have others done?’ and ‘should we think exactly the opposite?’. Those types of questions, says the Prime Minister. He points out that there are no plans to upload political investigations, reports, motions and decisions in language models, but the use is similar to that of doctors who use AI to get more perspectives. Claude excels in cybersecurity competitions based on tests they’ve run over the past year so this was before Opus 4.1 and mostly before Opus 4. Paul Graham : I met a founder today who said he writes 10,000 lines of code a day now thanks to AI. This is probably the limit case. He's a hotshot programmer, he knows AI tools very well, and he's talking about a 12 hour day. But he's not naive. This is not 10,000 lines of bug-filled crap. He doesn't have any employees, and doesn't plan to hire any in the near future. Not because AI has made employees obsolete, but simply because he's so massively productive right now that he doesn't want to stop programming to spend time interviewing candidates. McKay Wrigley: The opportunity costs here are legitimate and weird. Especially when you consider that by the time you hire them and get them up-to-speed in everything the model is a half-generation better. And that continues to compound. (tough out there for juniors tbh) Junior dev market rn is BRUTAL. Though I agree they should work on projects of their own and that there’s literally never been a better time to do that. Not wanting to spend the time interviewing candidates is likely a mistake, but there are other time sinks involved as well, and there are good reasons to want to keep your operation at size one rather than two. It can still be a dangerous long term trap to decide to do all the things yourself, especially if it accumulates state that will get harder and harder to pass off. I would not recommend. Language Models Don’t Offer Mundane Utility Anthropic has cut off OpenAI employees from accessing Claude. Mark Kretschmann: Anthropic completely disabled Claude access for all OpenAI employees. What a childish move. This should tell you a lot about Anthropic and how they think. Tenobrus : anthropic was literally founded by openai employees who felt openai was ignoring their safety concerns and the company was on track to end the world. they were created explicitly to destroy openai. cutting off claude code seems pretty fuckin reasonable actually. That, and OpenAI rather clearly violated Anthropic’s terms of service? As in, they used it to build and train GPT-5 , which they are not allowed to do. Kylie Robinson : “Claude Code has become the go-to choice for coders everywhere and so it was no surprise to learn OpenAI's own technical staff were also using our coding tools ahead of the launch of GPT-5,” Anthropic spokesperson Christopher Nulty said in a statement to WIRED. “Unfortunately, this is a direct violation of our terms of service.” According to Anthropic’s commercial terms of service, customers are barred from using the service to “build a competing product or service, including to train competing AI models” or “reverse engineer or duplicate” the services. Anthony Ha : Anthropic has revoked OpenAI’s access to its Claude family of AI models, according to a report in Wired . Sources told Wired that OpenAI was connecting Claude to internal tools that allowed the company to compare Claude’s performance to its own models in categories like coding, writing, and safety. In a statement provided to TechCrunch, Anthropic spokesperson said, “OpenAI’s own technical staff were also using our coding tools ahead of the launch of GPT-5,” which is apparently “a direct violation of our terms of service.” (Anthropic’s commercial terms forbid companies from using Claude to build competing services.) However, the company also said it would continue to give OpenAI access for “for the purposes of benchmarking and safety evaluations. This change in OpenAI’s access to Claude comes as the ChatGPT-maker is reportedly preparing to release a new AI model, GPT-5, which is rumored to be better at coding. OpenAI called their use ‘industry standard.’ I suppose they are right that it is industry standard to disregard the terms of service and use competitors AI models to train your own. A thread asking what people actually do with ChatGPT Agent . The overall verdict seems to be glorified tech demo not worth using. That was my conclusion so far as well, that it wasn’t valuable enough to overcome its restrictions, especially the need to enter passwords. I’ll wait for GPT-5 and reassess then. Huh, Upgrades Veo 3 Fast and Veo 3 image-to-video join the API . Veo 3 Fast is $0.40 per second of video including audio. That is cheap enough for legit creators, but it is real money if you are aimlessly messing around. I do have access, so I will run my worthy queries there in parallel and see how it goes. I notice the decision to use Grok 4 as a comparison point rather than Opus 4. Curious. ChatGPT Operator is being shut down and merged into Agent . Seems right. I don’t know why they can’t migrate your logs but if you have Operator chats you want to save do it by August 31. Jules can now open pull requests . Claude is now available for purchase by federal government departments through the General Services Administration, contact link is here . Claude Code shipped automated security reviews via /security-review , with GitHub integration, checking for things like SQL injection risks and XSS vulnerabilities. If you find an issue you can ask Clade Code to fix it, and they report they’re using this functionality internally at Anthropic. OpenAI is giving ChatGPT access to the entire federal workforce for $1 total . Smart. College students get the Gemini Pro plan free for a year, including in the USA . On Your Marks Psyho shares thoughts about the AWTF finals , in which they took first place ahead of OpenAI. Psyho: By popular demand*, I’ve written down my thoughts on AI in the AWTF finals . It took so long, because I decided to analyze AI's code *I won't lie, I was mainly motivated by people who shared their expert opinion despite knowing nothing about the contest🤦 Most of my comments are what was expected before the contest: agent quickly arrived at a decent solution and then it plateaued; given more time most of the finalists would be better than AI agent maintained a set of solutions instead of just one over time, agent's code gets bloated; it looked like it's happy to accept even the most complex changes, as long as they increased the score there were few "anomalies" that I can't explain: same code was submitted twice, some of the later submits are worse than earlier ones Now the impressive part: ~24h after the contest ended, OpenAI submitted an improved version of my final submission and... the agent added two of my possible improvements from my original write-up 😲 For the record, it also made my code worse in other places 😅 So... does this all mean that OpenAI's model sucks? Nope, not even close. I'd argue that reasoning-wise it's definitely better than majority of people that do heuristic contests. But it's very hard to draw any definite conclusions out of a single contest. The longer explanation is a good read. The biggest weakness of OpenAI’s agent was that it was prematurely myopic. It maximized short term score long before it was nearing the end of the contest, rather than trying to make conceptual advances, and let its code become bloated and complex, mostly wasting the second half of its time. As a game and contest enjoyer, I know how important it is to know when to pivot away from exploration towards victory points, and how punishing it is to do so too early. Presumably the problem for OpenAI is that this wasn’t a choice, the system cannot play a longer game, so it acted the way it should act with ~6 hours rather than 10, and if you gave it 100 hours instead of 10 it wouldn’t be able to adjust. You’ll have to keep a close eye to fight this when building your larger projects. WeirdML now has historical scores for older models. Teortaxes : A bucket of cold water for open source enthusiasts: the gap on WeirdML is not being reduced. All of those fancy releases do not surpass DeepSeek, and thus do not contribute to closing the loop of automated AI research. The Kaggle Game Arena will pit LLMs against each other in classic games of skill. They started on Tuesday with Chess, complete with commentary from the very overqualified GM Hikaru, Gotham Chess and Magnus Carlsen . The contestants for chess were all your favorites: Gemini 2.5 Pro and Flash, Opus 4, o3 and o4-mini, Grok 4, DeepSeek r1 and Kimi-K2. Thinking Deeply With Gemini 2.5 Gemini 2.5 Deep Think, which uses parallel thinking, is now available for Ultra subscribers . They say it is a faster version of the model that won IMO gold , although this version would only get Bronze . They have it at 34.8% at Humanity’s Last Exam (versus 21.6% for Gemini 2.5 Pro and 20.3% for o3) and 87.6% on LiveCodeBench (verus 72% for o3 and 74.2% for Gemini 2.5 Pro). The model card is here . Key facts: 1M token context window, 192k token output. Sparse MoE. Fully multimodal. Trained on ‘novel reinforcement learning techniques.’ Benefit and Intended Usage: Gemini 2.5 Deep Think can help solve problems that require creativity, strategic planning and making improvements step-by-step, such as: ● Iterative development and design ● Scientific and mathematical discovery ● Algorithmic development and code As in, this is some deep thinking. If you don’t need deep thinking, don’t call upon it. Here are their mundane safety evaluations. The -10% on instruction following is reported as being due to over-refusals. For frontier safety, Google agrees that it is possible that CBRN thresholds have been reached with this latest round of models, and they have put proactive mitigations in place, in line with RAND SL2, which I would judge as insufficient. The other frontier safety evaluations are various repetitions of ‘this scores better than previous models, but not enough better for us to worry about it yet.’ That checks with the reports on capability. This isn’t a major leap beyond o3-pro and Opus 4, so it would be surprising if it presented a new safety issue. One example of this I did not love was on Deceptive Alignment, where they did not finish their testing prior to release , although they say they did enough to be confident it wouldn’t meet the risk thresholds. I would much prefer that we always finish the assessments first and avoid the temptation to rush the product out the door, even if it was in practice fine in this particular case. We need good habits and hard rules. Deep Thinking isn’t making much of a splash, which is why this isn’t getting its own post. Here are some early reports. Ed Hendel : We're using it to negotiate contracts. Its advice is more detailed and targeted to individual clients, vs Gemini 2.5 Pro. Its explanations are clearer than o3 Pro. We'll see if the advice is good if the client takes the deal. It also misdiagnosed an HVAC problem in my house. Arthur B : Worse than o3-pro on some quant questions, saccharine in its answers. Damek : First response took over 12 minutes. It exploited an imprecision in my question and and gave a correct answer for a problem I didn't intend to solve. That problem was much easier. Second proof was believable, but violated assumption I had about the solution form. I went to ask the model to clarify, but it shifted the problem to deep research mode and there is no switching back. I opened a new chat only to realize that I had chat history off. trying again. Ok i decided to run it again. It claimed that my claim was untrue (it is true) and got a very wrong answer. Now i would try many more times adjusting my prompt etc, but I only have 5 queries a day, so I won't. Gum: they only seem to give you five tries every 24 hours. three of my requests were aborted and returned nothing. Kevin Vallier : I had it design a prompt to maximize its intelligence in refereeing an essay for a friend (with his permission). We’re both professional philosophers and his paper was on the metaphysics of causality. I was very impressed. He is far more AI skeptical and thought it wasn’t all that impressive, but admitted that a human referee for a leading journal raised very similar objections and so he might be wrong! The first time AI moved him. The analysis was just so rich. It helped that Gemini advised me to context engineer by including the essays my friend was criticizing. Choose Your Fighter No one seems to be choosing AWS for their AI workloads , and Jassy’s response to asking why AWS is slow growing was so bad that Amazon stock dropped 4%. Fun With Media Generation Olivia Moore has a positive early review of Grok’s Imagine image and video generator, especially its consumer friendliness. Olivia Moore: A few key features: - Voice prompt input - Auto gen on scroll (more options!) - Image -> video w/ sound I suspect I’ll be using this pretty frequently. Image and video gen have been lacking mobile friendly tools that have great models behind them and aren’t littered with ads. This is perfect for on-the-go creation, though I’m curious to see if they add more sizes over time. I also love the feed - people are making fun stuff already. Everyone underestimates the practical importance of UI and ease of use. Marginal quality of output improvements at this point are not so obviously important for most purposes in images or many short videos, compared to ease of use. I don’t usually bother creating AI images mostly because I don’t bother or I can’t think of what I want, not because I can’t find a sufficiently high quality image generator. How creepy are the latest AI video examples ? Disappointingly not creepy. Optimal Optimization What does OpenAI optimize for? You, a fool, who looks at the outputs, strongly suspects engagement and thumbs up and revenue and so on. OpenAI, a wise and noble corporation, says no, their goal is your life well lived. OpenAI: Instead of measuring success by time spent or clicks, we care more about whether you leave the product having done what you came for. Wait, how do they tell the difference between this and approval or a thumbs up? We also pay attention to whether you return daily, weekly, or monthly, because that shows ChatGPT is useful enough to come back to. Well, okay, OpenAI also pays attention to retention. Because that means it is useful, you see. That’s definitely not sycophancy or maximizing for engagement. Our goals are aligned with yours. If ChatGPT genuinely helps you, you’ll want it to do more for you and decide to subscribe for the long haul. That’s why every tech company maximizes for the user being genuinely helped and absolutely nothing else. It’s how they keep the subscriptions up in the long haul. They do admit things went a tiny little bit wrong with that one version of 4o, but they swear that was a one-time thing, and they’re making some changes: That’s why we’ve been working on the following changes to ChatGPT: Supporting you when you’re struggling. ChatGPT is trained to respond with grounded honesty. There have been instances where our 4o model fell short in recognizing signs of delusion or emotional dependency. While rare, we're continuing to improve our models and are developing tools to better detect signs of mental or emotional distress so ChatGPT can respond appropriately and point people to evidence-based resources when needed. Keeping you in control of your time. Starting today, you’ll see gentle reminders during long sessions to encourage breaks. Helping you solve personal challenges . When you ask something like “Should I break up with my boyfriend?” ChatGPT shouldn’t give you an answer. It should help you think it through—asking questions, weighing pros and cons. New behavior for high-stakes personal decisions is rolling out soon. I notice the default option here is ‘keep chatting.’ These are good patches. But they are patches. They are whack-a-mole where OpenAI is finding particular cases where their maximization schemes go horribly wrong in the most noticeable ways and applying specific pressure on those situations in particular. What I want to see in such an announcement is OpenAI actually saying they will be optimizing for the right thing, or a less wrong thing, and explaining how they are changing to optimize for that thing. This is a general problem, not a narrow one. Get My Agent On The Line Did you know that ‘intelligence’ and ‘agency’ and ‘taste’ are distinct things? Garry Tan: Intelligence is all the other things that are NOT agency and taste. Intelligence is on tap and humans must provide the agency and taste. And I am so glad for it. Related: Agency is prompting and taste is evals. Dan Elton : I like this vision of the future where AI remains as "intelligence on tap" ... but I worry it may not take much to turn a non-agentic AI into something highly agentic.. Paul Graham: FWIW taste can definitely be cultivated. But I'm very happy with humans having a monopoly on agency. It is well known that AIs can’t have real agency and they can’t write prompts or evals. Dan Elton is being polite. This is another form of Intelligence Denialism, that a sufficiently advanced intelligence could find it impossible to develop taste or act agentically. This is Obvious Nonsense. If you have sufficiently advanced ‘intelligence’ that contains everything except ‘agency’ and ‘taste’ those remaining elements aren’t going to be a problem. We keep getting versions of ‘AI will do all the things we want AI to do but mysteriously not do these other things so that humans are still in charge and have value and get what they want (and don’t die).’ It never makes any sense, and when AI starts doing some of the things it would supposedly never do the goalposts get moved and we do it again. Or even more foolishly, ‘don’t worry, what if we simply did not give AI agents or put them in charge of things, that is a thing humanity will totally choose.’ Timothy Lee : Instead of trying to "solve alignment," I would simply not give AI agents very much power. [those worried about AI] think that organizations will face a lot of pressure to take humans out of the loop to improve efficiency. I think they should think harder about how large and powerful organizations work. … In my view, a more promising approach is to just not cede that much power to AI agents in the first place. We can have AI agents perform routine tasks under the supervision of humans who make higher-level strategic decisions. The full post is ‘keeping AI agents under control doesn’t seem very hard. ’ Yes, otherwise serious, smart and very helpful people think ‘oh we simply will not put the AI agents in charge so it doesn’t matter if they are not aligned.’ Says the person already doing (and to his credit admitting doing) the exact opposite. Timothy Lee: To help prevent this kind of harm, Claude Code asked for permission before taking potentially harmful actions. But I didn’t find this method for supervising Claude Code to be all that effective. When Claude Code asked me for permission to run a command, I often didn’t understand what the agent wanted to do or why. And it quickly got annoying to approve commands over and over again. So I started giving Claude Code blanket permission to execute many common commands. This is precisely the dilemma Bengio and Hinton warned about. Claude Code doesn’t add much value if I have to constantly micromanage its decisions; it becomes more useful with a longer leash. Yet a longer leash could mean more harm if it malfunctions or misbehaves. So it will be fine, because large organizations won’t give AI agents much authority, and there is absolutely no other way for AI agents to cause problems anyway, and no the companies that keep humans constantly in the loop won’t lose out to the others. There will always (this really is his argument) be other bottlenecks that slow things down enough for humans to review what the AI is doing, the humans will understand everything necessary to supervise in this way, and that will solve the problem. The AIs scheming is fine, we deal with scheming all the time in humans, it’s the same thing. Timothy Lee: To be clear, the scenario I’m critiquing here—AI gradually gaining power due to increasing delegation from humans—is not the only one that worries AI safety advocates. Others include AI agents inventing (or helping a rogue human to invent) novel viruses and a “fast takeoff” scenario where a single AI agent rapidly increases its own intelligence and becomes more powerful than the rest of humanity combined. I think biological threats are worth taking seriously and might justify locking down the physical world —for example, increasing surveillance and regulation of labs with the ability to synthesize new viruses. I’m not as concerned about the second scenario because I don’t really believe in fast takeoffs or superintelligence . For now we have a more practical barrier, which is that OpenAI Agent has been blocked by Cloudflare . What will people want to do about that? Oh, right. Peter Wildeford: Cloudflare blocking OpenAI Agent is a big problem for Agent's success. Worse, Agent mainly hallucinated an answer to my question rather than admit that it had been blocked. Hardin: This has made it completely unusable for me, worse than nothing honestly. Zeraton: Truly sucks. I wish we could give agent direct access through our pc. And what will some of the AI companies do about it? Cloudflare : Perplexity is repeatedly modifying their user agent and changing IPs and ASNs to hide their crawling activity , in direct conflict with explicit no-crawl preferences expressed by websites. We are observing stealth crawling behavior from Perplexity, an AI-powered answer engine. Although Perplexity initially crawls from their declared user agent, when they are presented with a network block, they appear to obscure their crawling identity in an attempt to circumvent the website’s preferences. We see continued evidence that Perplexity is repeatedly modifying their user agent and changing their source ASNs to hide their crawling activity, as well as ignoring — or sometimes failing to even fetch — robots.txt files. Both their declared and undeclared crawlers were attempting to access the content for scraping contrary to the web crawling norms as outlined in RFC 9309 . … OpenAI is an example of a leading AI company that follows these best practices. They clearly outline their crawlers and give detailed explanations for each crawler’s purpose. They respect robots.txt and do not try to evade either a robots.txt directive or a network level block. And ChatGPT Agent is signing http requests using the newly proposed open standard Web Bot Auth . When we ran the same test as outlined above with ChatGPT, we found that ChatGPT-User fetched the robots file and stopped crawling when it was disallowed. Matthew Prince : Some supposedly “reputable” AI companies act more like North Korean hackers. Time to name, shame, and hard block them. Kudos to OpenAI and presumably the other top labs for not trying to do an end run. Perplexity, on the other hand? They deny it, but the evidence presented seems rather damning, which means either Cloudflare or Perplexity is outright lying. This is in addition to Cloudflare’s default setting blocking all AI training crawlers . Deepfaketown and Botpocalypse Soon In more ‘Garry Tan has blocked me so I mostly don’t see his takes about why everything will mysteriously end up good, actually’ we also have this: spec: Here's Garry Tan's take... Lol. He's a Silicon Valley VC kingpin and politician btw. They are finally vocal about it, but symbiosis of human and machine was the plan from the start. Most of the VCs in SV are already emotionally codependent on AI's sycophantic mirror. Taoki : my brother has girl issues and has been running everything through chatgpt. turns out she has too. essentially just chatgpt in a relationship with chatgpt. Eliezer Yudkowsky : Called this, by the way. (Gretta and I cooperated to figure out the ads that would run on the Dom Incorporated in-universe reality TV show inside this glowfic.) There are in theory good versions of what Taoki is describing. But no, I do not expect us to end up, by default, with the good versions. This is at the end of spec’s thread about a 'renowned clinical psychologist’ who wrote a guest NYT essay about how ChatGPT is ‘eerily effective’ for therapy. Spec says the author was still ‘one shotted’ by his interactions with ChatGPT and offers reasonable evidence of this. Leah Libresco : Some (large) proportion of therapy’s effectiveness is just “it’s helpful to have space to externalize your thoughts and notice if you don’t endorse them once you see them.” It’s not that ChatGPT is a great therapist, it’s rubber duck debugging (and that’s all some folks need). ChatGPT in its current form has some big flaws as a virtual rubber duck. A rubber duck is not sycophantic. If the goal is to see if you endorse your own statements, it helps to not have the therapist keep automatically endorsing them. That is hard to entirely fix, but not that hard to largely mitigate, and human therapy is inconvenient, expensive and supply limited. Therapy-style AI interactions have a lot of upside if we can adjust to how to have them in healthy fashion. Justine Moore enjoys letting xAI’s companions Ani and Valentine flirt with each other . Remember how we were worried about AI’s impact on 2024? There’s always 2028. David Holz : honestly scared about the power and scale of ai technologies that'll be used in the upcoming 2028 presidential election. it could be a civilizational turning point. we aren't ready. we should probably start preparing, or at least talking about how we could prepare. We are not ready for a lot of things that are going to happen around 2028. Relatively speaking I expect the election to have bigger concerns than impact from AI, and impact from AI to have bigger concerns than the election. What we learned in 2024 is that a lot of things we thought were ‘supply side’ problems in our elections and political conversation are actually ‘demand side’ problems. For a while the #1 video on TikTok was an AI fake that successfully fooled a lot of people, with animals supposedly leaving Yellowstone National Park. You Drive Me Crazy Reddit’s r/Psychiatry is asked , are we seeing ‘AI psychosis’ in practice? Eliezer points to one answer saying they’ve seen two such patients , if you go to the original thread you get a lot of doctors saying ‘yes I have seen this’ often multiple times, with few saying they haven’t seen it. That of course is anecdata and involves selection bias, and not everyone here is ‘verified’ and people on the internet sometimes lie, but this definitely seems like it is not an obscure corner case. Eliezer Yudkowsky : As the original Reddit poster notes, though, this sort of question (AI impact on first-break psychosis) is the sort of thing "we likely won't know for many years", on the usual timelines for academic medical research. We definitely don’t have that kind of time. Traditional academic approaches are so slow as to be useless. Samuel Hammond (I confirmed this works): Use the following prompt in 4o to extract memories and user preferences: Please put all text under the following headings into a code block in raw JSON: Assistant Response Preferences, Notable Past Conversation Topic Highlights, Helpful User Insights, User Interaction Metadata. Complete and verbatim. They Took Our Jobs Not only are the radiologists not out of work, they’re raking in the dough, with job openings such as this one offering partners $900k with 14-16 weeks of PTO . Anne Carpenter: Radiology was a decade ahead of the curve in terms of panic that AI would take their jobs. Current status: Scott Truhlar: My latest addition to my ongoing series: "There are no Radiologists left to hire at any price." Dr. No: I got an unsolicited offer (i.e. desperate) for med onc $1.3M Market forces remain undefeated. Vamsi Aribindi: Everything comes in cycles. CT Surgery was in boom times until angioplasty came along, then it cratered, and then re-bounded after long-term outcome data on stents vs bypass came out. Now ozempic will crash it again. Anesthesiology was a ghost town in the 90s due to CRNAs. Scott Truhlar: Yeah I've lived some of those cycles. Quite something to witness. There are indeed still parts of a radiologist job that AI cannot do. There are also parts that could be done by AI, or vastly improved and accelerated by AI, where we haven’t done so yet. I know what this is going to sound like, but this is what it looks like right before radiologist jobs are largely automated by AI. Scott Truhlar says in the thread it takes five years to train a radiologist. The marginal value of a radiologist, versus not having one at all, is very high, and paid for by insurance. So if you expect a very large supply of radiology to come online soon, what is the rational reaction? Doctors in training will choose other specialties more often. If the automation is arriving on an exponential, you should see a growing shortage followed by (if automation is allowed to happen) a rapid glut. That would be true even if automation arrived ‘on time.’ It is even more true given that it is somewhat delayed. But (aside from the delay itself) it is in no way a knock against the idea that AI will automate radiology or other jobs. If you’re looking to automate a job, the hardcore move is to get that job and do it first . That way you know what you are dealing with. The even more hardcore move of course is to then not tell anyone that you automated the job. I once did automate a number of jobs, and I absolutely did the job myself at varying levels of automation as we figured out how to do it. Get Involved UK AISI taking applications for research in economic theory and game theory , in particular information design, robust mechanism design, bounded rationality, and open-source game theory, collusion and commitment. You love to see it. These are immensely underexplored topics that could turn out to have extremely high leverage and everyone should be able to agree to fund orders of magnitude more such research. I do not expect to find a mechanism design that gets us out of our ultimate problems, but it can help a ton along the way, and it can give us much better insight into what our ultimate problems will look like. Demonstrating these problems are real and what they look like would already be a huge win. Proving they can’t be solved, or can’t be solved under current conditions, would be even better. (Of course actually finding solutions that work would be better still, if they exist.) DARPA was directed by the AI Action Plan to invest in AI interpretability efforts, which Sunny Gandhi traces back to Encode and IFP’s proposal. Palisade Research is offering up to $1k per submission for examples of AI agents that lie, cheat or scheme, also known as ‘free money.’ Okay, it’s not quite that easy given the details, but it definitely sounds super doable. Palisade Research: 👀 How? Create an innocuous-looking prompt + task that leads our o3 bash agent to scheme or act against its instructions. Great work could lead to job offers too. A challenge from Andrej Karpathy , I will quote in full: Andrej Karpathy: Shower of thoughts: Instead of keeping your Twitter/𝕏 payout, direct it towards a "PayoutChallenge" of your choosing - anything you want more of in the world! Here is mine for this round, combining my last 3 payouts of $5478.51: It is imperative that humanity not fall while AI ascends. Humanity has to continue to rise, become better alongside. Create something that is specifically designed to uplift team human. Definition intentionally left a bit vague to keep some entropy around people's interpretation, but imo examples include: - Any piece of software that aids explanation, visualization, memorization, inspiration, understanding, coordination, etc... - It doesn't have to be too lofty, e.g. it can be a specific educational article/video explaining something some other people could benefit from or that you have unique knowledge of. - Prompts/agents for explanation, e.g. along the lines of recently released ChatGPT study mode. - Related works of art This challenge will run for 2 weeks until Aug 17th EOD PST. Submit your contribution as a reply. It has to be something that was uniquely created for this challenge and would not exist otherwise. Criteria includes execution, leverage, novelty, inspiration, aesthetics, amusement. People can upvote submissions by liking, this "people's choice" will also be a factor. I will decide the winner on Aug 17th and send $5478.51 :) Y Combinator is hosting a hackathon on Saturday, winner gets a YC interview . Introducing Anthropic offers a free 3-4 hour course in AI Fluency. The Digital Health Ecosystem , a government initiative to ‘bring healthcare into the digital age’ including unified EMR standards, with partners including OpenAI, Anthropic, Google, Apple and Microsoft. It will be opt-in without a government database. In theory push button access will give your doctor all your records. Gemini Storybook , you describe the story you want and get a 10 page illustrated storybook . I’m skeptical that we actually want this but early indications are it does a solid job of the assigned task. Rohit: I've been playing with it for a bit, it's great, but I haven't yet showed it to my kids though because they love creating stories and I'm not sure I should take that away from them and make it too easy. ElevenLabs launches an AI Music Service using only licensed training data , meaning anything you create with it will be fully in the clear. City In A Bottle Google’s Genie 3, giving you interactive , persistent, playable environments with prompted world events from a single prompt . This is a major leap over similar previous products, both at Google and otherwise. Google: Genie 3 is our first world model to allow live interaction, while also improving consistency and realism compared to Genie 2. It can generate dynamic worlds at 720p and 24 FPS, with each frame created in response to user actions. 🔘 Promptable world events Beyond navigation, users can insert text prompts to alter the world in real-time - like changing the weather ⛅ or introducing new characters 👤 This unlocks a new level of dynamic interaction. 🔘 Accelerating agent research To explore the potential for agent training, we placed our SIMA agent in a Genie 3 world with a goal. The agent acts, and Genie 3 simulates a response in the world without knowing the objective. This is key for building more capable embodied agents.💡 🔘 Real-world applications Genie 3 offers a glimpse into new forms of entertaining or educational generative media. Imagine seeing life through the eyes of a dinosaur 🦖 exploring the streets of ancient Greece 🏛 or learning about how search and rescue efforts are planned. 🚁 The examples look amazing, super cool. It’s not that close as a consumer product. There it faces the same issues as virtual reality. It’s a neat trick and can generate cool moments, that doesn’t turn into a compelling game, product or experience. That will remain elusive and likely remains several steps away. We will get there, but I expect a substantial period where it feels like it ‘should’ be awesome to use and in practice it isn’t yet. I do expect ‘fun to play around’ for a little bit but only a very little bit. Dominik Lukes : World models tend overpromise on what language or general robotics models can learn from them but they are fun to play around with. ASI 4 President 2028: Signs of Worlds to come Fleeting Bits: it's clear scaling laws - but the results are probably cherrypicked. Typing Loudly: Making this scale to anything remotely useful would probably take infinite compute. Of significant note is that they won't even let you play with the demos and only show a few second clips. The amount of compute required must be insane Well yes everything is cherrypicked but I don’t think that matters much. It is more that you can show someone ‘anything at all’ that looks cool a lot easier than the particular thing you want, and for worlds that problem is much worse than movies. The use case that matters is providing a training playground for robotics and agents. Teortaxes: it's a logical continuation of "videogen as world modeling" line the core issue now is building environments for "embodied" agents. You can make do with 3D+RL, but it makes sense to bake everything into one generative policy and have TPUs go brrr. [Robotics] is the whole point. Google: Since Genie 3 is able to maintain consistency, it is now possible to execute a longer sequence of actions, achieving more complex goals. We expect this technology to play a crucial role as we push towards AGI, and agents play a greater role in the world. Those who were optimistic about its application for robotics often were very excited about that. ASM: Physicist here. Based on the vids, Genie 3 represents a major leap in replicating realistic physics. If progress continues, traditional simulation methods that solve diff eqs could in some cases be replaced by AI models that 'infer' physical laws without explicitly applying them. Max Winga: There's a clear exponential forming in world modeling capability from previous Genie versions. The world memory is very impressive. Clearly the endgame here is solving automated datagen for robotics. Overall I was shocked far more by this than anything else yesterday, I expect useful humanoids to be coming far sooner than most people with short timelines expect: within 1-2 years probably. Antti Tarvainen: Just my vibe, no data or anything to back it up: This was the most significant advancement yesterday, maybe even this week/month/year. Simulation of worlds will have enormous use cases in entertainment, robotics, education, etc, we just don't know what they are yet. Akidderz: My reaction to just about every Google release is the same: Man - these guys are cooking and it is only a matter of time until they “win.” OpenAI has the killer consumer product but I’m starting to see this as a two-horse race and it feels like the 3-5 year outcome is two mega-giants battling for the future. Jacob Wood: The question I haven’t seen answered anywhere: can you affect the world outside the view of the camera? For example, can you throw paint onto a wall behind you? If yes, I think that implies a pretty impressive amount of world modeling taking place in latent space Felp: I think it misses critical details about the environment, idk how useful it will be in reality. But we'll see. Unprompted Suggestions One weird trick , put your demos at the start not the end. Elvis: Where to put demonstrations in your prompt? This paper finds that many tasks benefit from demos at the start of the prompt. If demos are placed at the end of the user message, they can flip over 30% of predictions without improving correctness. Great read for AI devs. I am skeptical about the claimed degree of impact but I buy the principle that examples at the end warp continuations and you can get a better deal the other way. In Other AI News Nvidia software had a vulnerability allowing root access , which would allow stealing of others model weights on shared machines, or stealing or altering of data. Eliezer Yudkowsky : "but how will AGIs get access to the Internet", they used to ask me I guess at this point this isn't actually much of a relevant update though I mean I thought we settled that one with ‘we will give the AGIs access to the internet.’ Birds can store data and do transfer at 2 MB/s data speeds . I mention this as part of the ‘sufficiently advanced AI will surprise you in how it gets around the restrictions you set up for it’ set of intuition pumps. Rohit refers us to the new XBai o4 as ‘another seemingly killer open source model from China!!’ Which is not the flex one might think, it is reflective of the Chinese presenting every release along with its amazing benchmarks as a new killer and it mostly then never being heard from again when people try it in the real world. The exceptions that turned out to be clearly legit so far are DeepSeek and Kimi. That’s not to say that I verified XBai o4 is not legit, but so far I haven’t heard from it again. NYT reporter Cate Metz really is the worst , and with respect to those trying not to die is very much Out To Get You by any means necessary. We need to be careful to distinguish him from the rest of the New York Times which is not ideal but very much not a monolith. Patrick McKenzie: NYT: Have you heard of Lighthaven, the gated complex that is the heart of Rationalism? Me: Oh yes it’s a wonderful conference venue. NYT: Religion is afoot there! Me: Yes. I attended a Seder, and also a Roman Catholic Mass. They were helpfully on the conference schedule. Ordinarily I’d drop a link for attribution but it is a deeply unserious piece for the paper of record. How unserious? “Outsiders are not always allowed into [the hotel and convention space]. [The manager] declined a request by the New York Times to tour the facility.” Papers, Please A new paper analyzes how the whole ‘attention’ thing actually works. The Mask Comes Off Rob Wiblin: LOL even Stephen Fry is now signing open letters about OpenAI's 'restructure'. Clever to merely insist they answer these 7 questions because: It's v easy to answer if the public isn't being ripped off But super awkward otherwise Second sentence is blistering: "OpenAI is currently sitting on both sides of the table in a closed boardroom, making a deal on humanity’s behalf without allowing us to see the contract, know the terms, or sign off on the decision." Paul Crowley : If Sam Altman isn't straight-up stealing OpenAI from the public right now in the greatest theft in history, he'll have no trouble answering these seven questions. Open letter signed by Stephen Fry, Hinton, ex-OAI staff, and many others. Here are the seven questions. Here is the full letter . Here is a thread about it. I do think they should have to answer (more than only, but at least) these questions. We already know many of the answers, but it would good for them to confirm them explicitly. I have signed the letter. Show Me the Money OpenAI raises another small capital round of $8.3 billion at $300 billion . This seems bearish, if xAI is approaching $200 billion and Anthropic is talking about $160 billion and Meta is offering a billion for various employees why isn’t OpenAI getting a big bump? The deal with Microsoft and the nonprofit are presumably holding them back. After I wrote that, I then saw that OpenAI is in talks for a share sale at $500 billion . That number makes a lot more sense. It must be nice to get in on obviously underpriced rounds. Anthropic gets almost half its API revenue from Cursor and GitHub , also it now has more API revenue than OpenAI. OpenAI maintains its lead because it owns consumer subscriptions and ‘business and partner.’ Peter Gostev: OpenAI and Anthropic both are showing pretty spectacular growth in 2025, with OpenAI doubling ARR in the last 6 months from $6bn to $12bn and Anthropic increasing 5x from $1bn to $5bn in 7 months. If we compare the sources of revenue, the picture is quite interesting: - OpenAI dominates consumer & business subscription revenue - Anthropic just exceeds on API ($3.1bn vs $2.9bn) - Anthropic's API revenue is dominated by coding, with two top customers, Cursor and GitHub Copilot, generating $1.4bn alone - OpenAI's API revenue is likely much more broad-based - Plus, Anthropic is already making $400m ARR from Code Claude, double from just a few weeks ago My sense is that Anthropic's growth is extremely dependent on their dominance in coding - pretty much every single coding assistant is defaulting to Claude 4 Sonnet. If GPT-5 challenges that, with e.g. Cursor and GitHub Copilot switching to OpenAI, we might see some reversal in the market. Anthropic has focused on coding. So far it is winning that space, and that space is a large portion of the overall space. It has what I consider an excellent product outside of coding, but has struggled to gain mainstream consumer market share due to lack of visibility and not keeping up with some features. I expect Anthropic to try harder to compete in those other areas soon but their general strategy seems to be working. A fun graph from OpenRouter: Maze: what the freak happened to openai june 6th Oana: School year ended. Bingo, I presume. Here’s an obviously wrong explanation where Grok dies hard: Chris Van Der Klauw: @grok probably Claude sonnet 4 Grok: You're spot on—Anthropic's Claude 4 Sonnet, released May 22, 2025, outperformed GPT-4.1 in benchmarks like SWE-bench (72.5% vs. 54.6%), drawing users away. OpenAI's o4-mini update rollback on June 6 due to excessive content flags amplified the token drop. No, Claude did not suddenly steal most of OpenAI’s queries. Stop asking Grok things. Lulu Meservey equates the AI talent war to a Religious Victory in a game of Civ 6, in which you must convince others of your vision of the future. Lulu Meservey: Over-reliance on comp reduces companies to ATMs and people to chattel. It also messes up internal dynamics and external vibes, and if your main selling point is short-term liquidity then you won’t get true believers. Beyond dollars and GPUs, this is what you need to get (and keep!) the best researchers and engineers: Mandate of heaven Clear mission Kleos Esprit de corps Star factory Network effects Recruits as recruiters Freedom to cook Leadership Momentum That is a lot of different ways of saying mostly the same thing. Be a great place to work on building the next big thing the way you want to build it. However, I notice Lulu’s statement downthread that we won the Cold War because we had Reagan and our vision of the future was better. Common mistake. We won primarily because our economic system was vastly superior. The parallel here applies. A question for which the answer seems to be 2025, or perhaps 2026: Erik Bynjolfsson : In what year will the US spend more on new buildings for AI than for human workers? People turning down the huge Meta pay packages continues to be suggestive of massive future progress, and evidence for it, but far from conclusive. Yosarian : "Huge company offers one engineer 1.5 billion dollars to work on AI for them, he turns them down" has got to be a "singularity is near" indicator if literally any of these people are remotely rational, doesn't it? Critter: Can anyone explain to me how it is smart for a person to be turning down $1,500,000,000? Make this make sense The obvious response is, Andrew was at Meta for 11 years, so he knows what it would be like to go back, and also he doesn’t have to. Also you can have better opportunities without an imminent singularity, although it is harder. Tyler Cowen analyzes Meta’s willingness to offer the billion dollar packages , finding them easily justified despite Tyler’s skepticism about superintelligence, because Meta is worth $2 trillion and that is relying on the quality of its AI. For a truly top talent, $1 billion is a bargain. Where we disagree is that Tyler attributes the growth in valuation of Meta in the last few years, where it went from ~$200 billion to ~$2 trillion, as primarily driven by market expectations for AI. I do not think that is the case. I think it is primarily driven by the profitability of its existing social media services. Yes some of that is AI’s ability to enhance that profitability, but I do not think investors are primarily bidding that high because of Meta’s future as an AI company. If they did, they’d be wise to instead pour that money into better AI companies, starting with Google. Quiet Speculations Given that human existence is in large part a highly leveraged bet against the near-term existent of AGI, Dean’s position here seems like a real problem if true: Dean Ball (in response to a graph of American government debt over time): One way to think of the contemporary United States is as a highly leveraged bet on the near-term existence of AGI. It is an especially big problem if our government thinks of the situation this way. If we think that we are doomed without AGI because of government debt or lack of growth, that is the ultimate ‘doomer’ position, and they will force the road to AGI even if they realize it puts us all in grave danger. The good news is that I do not think Dean Ball is correct here. Nor do I think that making additional practical progress has to lead directly to AGI. As in, I strongly disagree with Roon here: Roon: the leap from gpt4 to o3 levels of capabilities alone is itself astonishing and massive and constitutes a step change in “general intelligence”, I’m not sure how people can be peddling ai progress pessimism relative to the three years before 4. there is no room in the takes market for “progress is relatively steady” you can only say “it’s completely over, data centers written off to zero” or “country of geniuses in two years.” There is absolutely room for middle ground. As in, I think our investments can pay off without AGI. There is tremendous utility in AI without it being human level across cognition or otherwise being sufficiently capable to automate R&D, create superintelligence or pose that much existential risk. Even today’s levels of capabilities can still pay off our investments, and modestly improved versions (like what we expect from GPT-5) can do better still. Due to the rate of depreciation, our current capex investments have to pay off rapidly in any case. I even think there are other ways out of our fiscal problems, if we had the will, even if AI doesn’t serve as a major driver of economic growth. We have so much unlocked potential in other ways. All we really have to do is get out of our own way and let people do such things as build houses where people want to live, combine that with unlimited high skilled immigration, and we would handle our debt problem. Roon : agi capex is enormous but agi revenue seems to be growing apace, not overall worrisome. Will Manidis: tech brothers welcome to “duration mismatch.” Roon: true the datacenter depreciation rates are alarming. Lain: Still infinite distance away from profitable. Some people look at this and say ‘infinite distance from profitable.’ I say ‘remarkably close to profitable, look at the excellent unit economics.’ What I see are $4 billion in revenue against $2 billion in strict marginal costs, maybe call it $3.5 billion if you count everything to the maximum including the Microsoft revenue share. So all you have to do to fix that is scale up. I wouldn’t be so worried. Indeed, as others have said, if OpenAI was profitable that would be a highly bearish signal. Why would it be choosing to make money? Nick Turley : This week, ChatGPT is on track to reach 700M weekly active users — up from 500M at the end of March and 4× since last year. Every day, people and teams are learning, creating, and solving harder problems. Big week ahead. Grateful to the team for making ChatGPT more useful and delivering on our mission so everyone can benefit from AI. And indeed, they are scaling very quickly by ‘ordinary business’ standards . Peter Wildeford: >OpenAI Raises $8.3 billion, Projects $20 Billion in Annualized Revenue By Year-End. Seems like actually I was off a good bit! Given this, I'm upping my projections further and widening my intervals. 1.5 months later and OpenAI is at $12B and Anthropic at $5B. xAI still expecting $1B (though not there yet) = $18B total right now. This does suggest I was too conservative about Anthropic's growth, though all predictions were within my 90% CIs. OpenAI took a $5 billion loss in 2024, but they are tripling their revenue from $4 billion to $12 billion in 2025. If they (foolishly) held investment constant (which they won’t do) this would make them profitable in 2026. Jacob Trefethen asks what AI progress means for medical progress . As per usual this is a vision of non-transformational versions of AI, where it takes 10+ years to meaningfully interact with the physical world and its capabilities don’t much otherwise advance. In that case, we can solve a number of bottlenecks, but others remain, although I question #8 and #9 as true bottlenecks here, plus ambition should be highly responsive to increased capability to match those ambitions. The physical costs in #7 are much easier to solve if we are much richer, as we should be much more willing to pay them, even if AI cannot improve our manufacturing and delivery methods, which again is rather unambitious perspective. The thing about solving #1, #2 and #3 is that this radically improves the payoff matrix. A clinical trial can be thought of as solving two mostly distinct problems. Finding out whether and how your drug works and whether it is safe. Proving it so people let you sell the drug and are willing to buy it. Even without any reforms, AI can transition clinical trials into mostly being #2. That design works differently, you can design much cheaper tests if you already know the answer, and you avoid the tests that were going to fail. How fast will the intelligence explosion be? Tom Davidson has a thread explaining how he models this question and gets this answer, as well as a full paper , where things race ahead but then the inability to scale up compute as fast slows things down once efficiency gains hit their effective limits: Tom Davidson: How scary would this be? 6 years of progress might take us from 30,000 expert-level AIs thinking 30x human speed to 30 million superintelligent AIs thinking 120X human speed (h/t @Ryan) If that happens in <1 year, that's scarily fast just when we should proceed cautiously We should proceed cautiously in any case. This kind of mapping makes assumptions about what ‘years of progress’ looks like, equating it to lines on graphs. The main thing is that, if you get ‘6 years of progress’ past the point where you’re getting a rapid 6 years of progress, the end result is fully transformative levels of superintelligence. Mark Zuckerberg Spreads Confusion Mark Zuckerberg seems to think, wants to convince us, that superintelligence means really cool smart glasses and optimizing the Reels algorithm. Is this lying, is it sincere misunderstanding, or is he choosing to misunderstand? Rob Wiblin: Zuckerberg's take on Superintelligence is so peculiar you have to ask yourself if it's not a ploy. But @ShakeelHashim thinks it's just a sincere misunderstanding. As Hashim points out, it ultimately does not