This article explores a theoretical perspective on the progress of large language models (LLMs), arguing that while their scaling has led to remarkably general performance improvements, they still lack key cognitive functions needed for true human-level intelligence or AGI. The author, a theoretical computer scientist, contends that LLMs excel at memorization and pattern-matching due to next-token prediction but have yet to demonstrate the ability to innovate, solve novel problems, or plan effectively over long timescales. He highlights the historical progression of AI—from perceptrons through CNNs to reinforcement learning and now transformers—and emphasizes that raw scaling alone isn’t enough; breakthroughs have historically required new conceptual insights that address underlying cognitive limitations. The piece serves as a caution against overly optimistic timelines for AGI, pointing out that despite some impressive feats, LLMs have not yet performed any groundbreaking creative work, and further advancements may require decades of research and innovation.

Key Points:

• General Performance vs. Human-Level Intelligence: LLMs currently show impressive generalization ability but lack crucial cognitive functions like efficient, continual learning and long-term planning.
• Historical Context: The development of AI has been marked by periodic breakthroughs that combined scaling with novel theoretical insights—from perceptrons and CNNs to reinforcement learning.
• Limitations of Next-Token Prediction: The training paradigm of next-token prediction restricts LLMs’ ability to deliberate and plan, which are key components of genuine intelligence.
• The Need for Conceptual Breakthroughs: The author suggests that overcoming current limitations will likely require entirely new theoretical approaches rather than just more compute or tweaks to existing architectures.
• AGI Timelines Are Likely Overestimated: Despite some hype, the pathway to achieving true AGI is expected to be a slow process, potentially taking decades rather than a few imminent years.

And here’s a fun one: Why did the LLM go to therapy? Because it couldn’t plan a coherent response to its own existential crisis—guess it needed more than just next-token prediction!
Link to Article