llm/168b1d60-4680-4a3c-8c55-edcc91462b51/f50ccd29-6d43-4e82-8fed-8a912a4ef12f-output.json
# Summary: Phase 2 Brainstorming on AI Forecasting and Measurement ## Purpose and Goals This document captures notes from a brainstorming session focused on measuring and forecasting AI's development trajectory. The primary objectives include: - Gaining visibility into AI trends for policy planning - Obtaining advance notice of transformational changes - Estimating the steepness and extent of AI-driven transformation ## Measurement Approaches ### Data Collection Strategies Participants identified several measurement approaches, with a cross-cutting theme that high-quality surveys of companies would be valuable. Key methods discussed include: - **Surveys and controlled access**: Many ideas involve sensitive data requiring controlled researcher access (similar to social media transparency measures) - **AI-assisted analysis**: Using AI to qualitatively grade screen recordings and other high-volume data - **Benchmark calibration**: Correlating benchmark scores with real-world measurements (e.g., METR time horizons) - **Case studies**: Deep examination of how firms across sectors use AI, including revenue, spending, and profit analysis across the AI value chain ### Measuring Utility The group discussed measuring AI's practical utility through: - Benchmarks for specific domains (cyber, law, professional labor markets) and skills (reliability, long-context reasoning) - Uplift randomized controlled trials in real-world settings measuring actual outcomes - Power user case studies examining individuals and organizations achieving significant AI-assisted productivity gains - Work tests from organizations to assess AI performance on realistic tasks ### Measuring Adoption and Impact Proposed metrics include tracking: - AI-assisted versus AI-infeasible sector comparisons (employment, productivity, profit) - Revenue of AI service providers - Specific indicators like AI-discovered pharmaceutical products and AI-engaged academic publications - K-12 student AI tutoring usage A September 2024 Federal Reserve Bank of St. Louis study estimated that 0.5%–3.5% of all U.S. work hours were assisted by generative AI. ### Measuring Freedom of Action The discussion addressed tracking AI autonomy through: - Worker surveys on AI decision approval frequency - Monitoring agent infrastructure and online constraints - Discrete indicators (e.g., AI spending limits, internet access permissions) - Bug bounty reports on prompt injection vulnerabilities ## Curve-Bending Mechanisms Participants identified factors that could accelerate or slow AI progress: - **Intelligence explosion potential**: Measuring AI's role in accelerating its own development - **Algorithmic breakthroughs**: Tracking discontinuous capability jumps and new approaches - **Resource constraints**: Monitoring exhaustion of data, compute, or electricity - **Economic factors**: Investment slowdowns or AI winters from depleted ideas - **External events**: Market downturns, geopolitical conflicts, regulatory responses ## Supporting Resources The document references several external databases and research: - **MIT AI Risk Repository**: A comprehensive database of over 1,700 AI risks categorized by cause and domain - **AI Incident Database**: A living database tracking real-world AI harms and near-harms - **Our World in Data**: Tracking private investment in AI companies (2013-2024) - **Stanford AI Index Report**: Source data on AI investment trends ## Key Insights on AI Adoption According to linked research, generative AI adoption has been remarkably rapid: - By August 2024, nearly 40% of U.S. adults ages 18-64 used generative AI - Adoption has outpaced both personal computers and the internet at comparable points after mass market introduction - Usage spans a wide range of tasks and demographics, suggesting AI is becoming a general-purpose technology - Current productivity impact is estimated at 0.1%–0.9% growth in labor productivity