Home New Top Charts Search 10/12/2024 46 MIN It's Not About Scale, It's About Abstraction - Francois Chollet Machine Learning Street Talk (MLST) Play François Chollet discusses the limitations of Large Language Models (LLMs) and proposes a new approach to advancing artificial intelligence. He argues that current AI systems excel at pattern recognition but struggle with logical reasoning and true generalization. This was Chollet's keynote talk at AGI-24, filmed in high-quality. We will be releasing a full interview with him shortly. A teaser clip from that is played in the intro! Chollet introduces the Abstraction and Reasoning Corpus (ARC) as a benchmark for measuring AI progress towards human-like intelligence. He explains the concept of abstraction in AI systems and proposes combining deep learning with program synthesis to overcome current limitations. Chollet suggests that breakthroughs in AI might come from outside major tech labs and encourages researchers to explore new ideas in the pursuit of artificial general intelligence. TOC 1. LLM Limitations and Intelligence Concepts [00:00:00] 1.1 LLM Limitations and Composition [00:12:05] 1.2 Intelligence as Process vs. Skill [00:17:15] 1.3 Generalization as Key to AI Progress 2. ARC-AGI Benchmark and LLM Performance [00:19:59] 2.1 Introduction to ARC-AGI Benchmark [00:20:05] 2.2 Introduction to ARC-AGI and the ARC Prize [00:23:35] 2.3 Performance of LLMs and Humans on ARC-AGI 3. Abstraction in AI Systems [00:26:10] 3.1 The Kaleidoscope Hypothesis and Abstraction Spectrum [00:30:05] 3.2 LLM Capabilities and Limitations in Abstraction [00:32:10] 3.3 Value-Centric vs Program-Centric Abstraction [00:33:25] 3.4 Types of Abstraction in AI Systems 4. Advancing AI: Combining Deep Learning and Program Synthesis [00:34:05] 4.1 Limitations of Transformers and Need for Program Synthesis [00:36:45] 4.2 Combining Deep Learning and Program Synthesis [00:39:59] 4.3 Applying Combined Approaches to ARC Tasks [00:44:20] 4.4 State-of-the-Art Solutions for ARC Shownotes (new!): https://www.dropbox.com/scl/fi/i7nsyoahuei6np95lbjxw/CholletKeynote.pdf?rlkey=t3502kbov5exsdxhderq70b9i&st=1ca91ewz&dl=0 [0:01:15] Abstraction and Reasoning Corpus (ARC): AI benchmark (François Chollet) https://arxiv.org/abs/1911.01547 [0:05:30] Monty Hall problem: Probability puzzle (Steve Selvin) https://www.tandfonline.com/doi/abs/10.1080/00031305.1975.10479121 [0:06:20] LLM training dynamics analysis (Tirumala et al.) https://arxiv.org/abs/2205.10770 [0:10:20] Transformer limitations on compositionality (Dziri et al.) https://arxiv.org/abs/2305.18654 [0:10:25] Reversal Curse in LLMs (Berglund et al.) https://arxiv.org/abs/2309.12288 [0:19:25] Measure of intelligence using algorithmic information theory (François Chollet) https://arxiv.org/abs/1911.01547 [0:20:10] ARC-AGI: GitHub repository (François Chollet) https://github.com/fchollet/ARC-AGI [0:22:15] ARC Prize: $1,000,000+ competition (François Chollet) https://arcprize.org/ [0:33:30] System 1 and System 2 thinking (Daniel Kahneman) https://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555 [0:34:00] Core knowledge in infants (Elizabeth Spelke) https://www.harvardlds.org/wp-content/uploads/2017/01/SpelkeKinzler07-1.pdf [0:34:30] Embedding interpretive spaces in ML (Tennenholtz et al.) https://arxiv.org/abs/2310.04475 [0:44:20] Hypothesis Search with LLMs for ARC (Wang et al.) https://arxiv.org/abs/2309.05660 [0:44:50] Ryan Greenblatt's high score on ARC public leaderboard https://arcprize.org/ Episode Webpage Information Show Machine Learning Street Talk (MLST) Frequency Updated Semiweekly Published October 12, 2024 at 9:21 PM UTC Length 46 min Rating Clean United States Español (México) العربية Русский 简体中文 Français (France) 한국어 Português (Brazil) Tiếng Việt 繁體中文 (台灣) Select a country or region Africa, Middle East, and India See All Algeria Angola Armenia Azerbaijan Bahrain Benin Botswana Brunei Darussalam Burkina Faso Cameroun Cape Verde Chad Côte d’Ivoire Congo, The Democratic Republic Of The Egypt Eswatini Gabon Gambia Ghana Guinea-Bissau India Iraq Israel Jordan Kenya Kuwait Lebanon Liberia Libya Madagascar Malawi Mali Mauritania Mauritius Morocco Mozambique Namibia Niger (English) Nigeria Oman Qatar Congo, Republic of Rwanda São Tomé and Príncipe Saudi Arabia Senegal Seychelles Sierra Leone South Africa Sri Lanka Tajikistan Tanzania, United Republic Of Tunisia Turkmenistan United Arab Emirates Uganda Yemen Zambia Zimbabwe Asia Pacific See All Afghanistan Australia Bhutan Cambodia 中国大陆 Fiji 香港 Indonesia (English) 日本 Kazakhstan 대한민국 Kyrgyzstan Lao People's Democratic Republic 澳門 Malaysia (English) Maldives Micronesia, Federated States of Mongolia Myanmar Nauru Nepal New Zealand Pakistan Palau Papua New Guinea Philippines Singapore Solomon Islands 台灣 Thailand Tonga Turkmenistan Uzbekistan Vanuatu Vietnam Europe See All Albania Armenia Österreich Belarus Belgium Bosnia and Herzegovina Bulgaria Croatia Cyprus Czechia Denmark Estonia Finland France (Français) Georgia Deutschland Greece Hungary Iceland Ireland Italia Kosovo Latvia Lithuania Luxembourg (English) Malta Moldova, Republic Of Montenegro Nederland North Macedonia Norway Poland Portugal (Português) Romania Россия Serbia Slovakia Slovenia España Sverige Schweiz Türkiye (English) Ukraine United Kingdom Latin America and the Caribbean See All Anguilla Antigua and Barbuda Argentina (Español) Bahamas Barbados Belize Bermuda Bolivia (Español) Brasil Virgin Islands, British Cayman Islands Chile (Español) Colombia (Español) Costa Rica (Español) Dominica República Dominicana Ecuador (Español) El Salvador (Español) Grenada Guatemala (Español) Guyana Honduras (Español) Jamaica México Montserrat Nicaragua (Español) Panamá Paraguay (Español) Perú St. Kitts and Nevis Saint Lucia St. Vincent and The Grenadines Suriname Trinidad and Tobago Turks and Caicos Uruguay (English) Venezuela (Español) The United States and Canada See All Canada (English) Canada (Français) United States Estados Unidos (Español México) الولايات المتحدة США 美国 (简体中文) États-Unis (Français France) 미국 Estados Unidos (Português Brasil) Hoa Kỳ 美國 (繁體中文台灣) Copyright © 2026 Apple Inc. All rights reserved. Internet Service Terms Apple Podcasts web player & Privacy Cookie Warning Support Feedback To listen to explicit episodes, sign in. Sign In Stay up to date with this show Sign in or sign up to follow shows, save episodes, and get the latest updates. Sign In Select a country or region Africa, Middle East, and India See All Algeria Angola Armenia Azerbaijan Bahrain Benin Botswana Brunei Darussalam Burkina Faso Cameroun Cape Verde Chad Côte d’Ivoire Congo, The Democratic Republic Of The Egypt Eswatini Gabon Gambia Ghana Guinea-Bissau India Iraq Israel Jordan Kenya Kuwait Lebanon Liberia Libya Madagascar Malawi Mali Mauritania Mauritius Morocco Mozambique Namibia Niger (English) Nigeria Oman Qatar Congo, Republic of Rwanda São Tomé and Príncipe Saudi Arabia Senegal Seychelles Sierra Leone South Africa Sri Lanka Tajikistan Tanzania, United Republic Of Tunisia Turkmenistan United Arab Emirates Uganda Yemen Zambia Zimbabwe Asia Pacific See All Afghanistan Australia Bhutan Cambodia 中国大陆 Fiji 香港 Indonesia (English) 日本 Kazakhstan 대한민국 Kyrgyzstan Lao People's Democratic Republic 澳門 Malaysia (English) Maldives Micronesia, Federated States of Mongolia Myanmar Nauru Nepal New Zealand Pakistan Palau Papua New Guinea Philippines Singapore Solomon Islands 台灣 Thailand Tonga Turkmenistan Uzbekistan Vanuatu Vietnam Europe See All Albania Armenia Österreich Belarus Belgium Bosnia and Herzegovina Bulgaria Croatia Cyprus Czechia Denmark Estonia Finland France (Français) Georgia Deutschland Greece Hungary Iceland Ireland Italia Kosovo Latvia Lithuania Luxembourg (English) Malta Moldova, Republic Of Montenegro Nederland North Macedonia Norway Poland Portugal (Português) Romania Россия Serbia Slovakia Slovenia España Sverige Schweiz Türkiye (English) Ukraine United Kingdom Latin America and the Caribbean See All Anguilla Antigua and Barbuda Argentina (Español) Bahamas Barbados Belize Bermuda Bolivia (Español) Brasil Virgin Islands, British Cayman Islands Chile (Español) Colombia (Español) Costa Rica (Español) Dominica República Dominicana Ecuador (Español) El Salvador (Español) Grenada Guatemala (Español) Guyana Honduras (Español) Jamaica México Montserrat Nicaragua (Español) Panamá Paraguay (Español) Perú St. Kitts and Nevis Saint Lucia St. Vincent and The Grenadines Suriname Trinidad and Tobago Turks and Caicos Uruguay (English) Venezuela (Español) The United States and Canada See All Canada (English) Canada (Français) United States Estados Unidos (Español México) الولايات المتحدة США 美国 (简体中文) États-Unis (Français France) 미국 Estados Unidos (Português Brasil) Hoa Kỳ 美國 (繁體中文台灣)