llm/fc441fc4-f49b-4628-aa88-18400c0ae3ef/a7b4c48c-3ed7-4bc1-8ce0-4926e13ebac0-output.json
# Summary: Notes for Early Indicators Project
## Project Overview
This Notion page documents an ongoing research initiative focused on developing early indicators to track AI progress and its potential transformative impacts. The project aims to identify measurable signals that could inform debates about AI timelines, adoption rates, and societal effects—recognizing that predicting a fundamentally different future requires models and data rather than simple extrapolation.
## Collaborative Methodology
The project leader has found that serious analytical work on complex topics requires multiple rounds of deep thinking, which proves difficult in group settings. Success has come primarily through intensive 1:1 engagement via email and calls, with persistent follow-up. The plan involves multiple phases: initial deep analysis followed by group brainstorming sessions (potentially via Zoom or in-person at locations like Constellation) to aggregate ideas. Key participants being recruited include researchers from METR, Epoch AI, the Forecasting Research Institute, and policy-focused organizations like FAS and various DC-based groups.
## Key Research Questions and Cruxes
The project identifies several critical uncertainties requiring measurement:
- **Capability Milestones**: When will "superhuman coders" and "superhuman AI researchers" emerge? How is the task horizon (measured by METR's methodology showing AI completing ~50-minute human tasks at 50% success) progressing?
- **Benchmark vs. Reality Gap**: How fundamental is the disconnect between benchmark performance and real-world task completion, and is it growing or shrinking?
- **Automation Bottlenecks**: Are certain skills within coding or AI research proving harder to automate? Can algorithmic improvements compensate for compute limitations?
- **Economic Indicators**: Is Epoch AI's tracking of NVIDIA revenue and AI company financials bearing out? Could rapid algorithmic gains decouple impact from hardware revenue?
- **Adoption Patterns**: How rapidly will AI provide significant uplift across diverse everyday tasks, and what distinguishes tasks experiencing uplift from those that aren't?
## Evidence on AI Adoption and Progress
Several linked resources provide context on current AI diffusion:
**Adoption Rates**: According to Epoch AI's analysis, AI is being adopted faster than almost any technology in history. ChatGPT reached 10% US weekly usage within two years—far exceeding historical technology diffusion trends. Business adoption via platforms like Ramp suggests ~40% of businesses now pay for AI products.
**Productivity Claims vs. Reality**: A critical Substack post ("Where's the Shovelware?") argues that despite widespread claims of 10x productivity gains from AI coding tools, observable outputs (new apps, websites, games, GitHub projects) show no corresponding surge. The author's self-study found AI potentially slowed him by 21%, aligning with METR research showing developers overestimate AI productivity benefits.
**Time Horizon Progress**: METR's research shows frontier AI models have ~50-minute time horizons (the task length humans take that AI completes with 50% success), doubling roughly every 4-7 months. This trend appears consistent across coding, math, and reasoning domains, though agentic GUI tasks lag significantly behind.
## Bear Case Perspectives
A linked LessWrong post presents a skeptical view arguing that:
- Current scaling approaches won't achieve AGI; models are asymptoting at something short of true general intelligence
- Improvements since GPT-3.5 represent "window dressing" rather than fundamental capability advances
- LLMs excel at in-distribution problems and "eisegesis-friendly" tasks where humans do interpretive work
- True agency requiring sustained coherence across long inferential distances remains fundamentally limited
- A qualitatively different architectural approach may be needed, potentially arriving in the 2030s
## Economic and Bubble Considerations
Extensive notes examine whether AI investment represents a bubble. Key observations include:
- Current AI capex-to-revenue ratios (~6x) exceed historical technology buildouts
- OpenAI projects unprecedented revenue growth from $10B to $100B over three years
- Circular deals between AI companies may mask true market fundamentals
- However, investment levels as percentage of GDP remain below peaks of past technology buildouts like railroads
## Measurement Challenges
The project emphasizes that different sectors will experience AI impacts unevenly. Nick Allardice suggests focusing on specific industries as leading indicators—software engineering may be fast-moving while other sectors remain slow due to diffusion challenges, regulatory barriers, and institutional inertia. Global south adoption faces additional hardware, connectivity, language, and economic opportunity barriers that may delay impact by years or decades.
## Next Steps
Immediate plans include organizing Zoom brainstorming sessions, potentially an in-person session, and engaging additional participants from policy and economics backgrounds. The project may eventually produce draft legislation around AI measurement and monitoring requirements.