CAIS Izzy Barrass asked about this on 6/5/24:

Does anyone have any good papers/citations/data points on why we're at a tipping point for AI agents? Need to find data points for NeurIPS workshop proposal.

→ https://on.ft.com/3yWP1kV

→ https://www.insightpartners.com/ideas/ai-agents-disrupting-automation/

[Steven Basart] @Zifan Wang is working on AI Agents so he's a good (primary) source

Steven Basart also pointed to https://www.swebench.com/

Dan Hendrycks, 6/2/24 on #random:

Some timelines analysis:

Around ~15 trillion tokens are available through scraping and OCRing PDFs. This is enough for GPT-4.5 (10x compute GPT-4). It seems plausible GPT-4.5 will unlock adequate agentic capabilities, but other capabilities are unclear. There will be multiple GPT-4.5-level compute models at the end of the year.

GPT-5 (100x compute GPT-4) would need more tokens. The main path is synthetic data, which may end up just being multiple translations of existing tokens. It is not clear how much models will learn from these tokens. A GPT-5 could definitely be trained just by continuing to train the GPT-4.5 for another ~9 months, so 100x compute of GPT-4 is feasible for fall of 2025. If synthetic data isn’t enough, then a better pretraining algorithm could make all the difference. Consequently, the ML community needs to find a better pretraining algorithm fairly quickly for scaling to continue if synthetic data does not help that much.