Why the Future of AI Will Go Beyond Transformers

Over the past decade, AI has experienced unprecedented growth. But as enterprises move from experimentation to large-scale deployment, a new question is emerging: Are current AI architectures reaching their limits?

At the centre of this conversation is the growing realisation that while AI models have been scaling, they no longer deliver transformative leaps.

“Scaling laws still work, but what they deliver has changed. Larger models fed with more tokens continue to improve on benchmarks, although the gains have narrowed to the point where enterprises no longer see the dramatic leaps of earlier generations,” Waqas Ahmed, VP of AI Engineering at enterprise information management company OpenText, tells AIM.

A major constraint lies in enterprise use. According to the McKinsey & Company State of AI 2025 report, 51% of organisations using AI have already experienced negative outcomes due to inaccurate outputs.

Beyond this, domain specificity presents another challenge. Models trained on general internet data often struggle with enterprise-grade complexity.

The third and perhaps most fundamental limitation is architectural. “Transformer self-attention becomes exponentially more expensive as context grows,” Ahmed notes. 

The self-attention mechanism enables transformer models—neural network architectures that process sequential data in parallel—to weigh the importance of words in a sequence, allowing them to capture context. Expanding context windows significantly increases compute costs, making efficiency a central concern.

The Real Bottleneck

While much attention has been given to expanding context windows, the real issue inside enterprises lies elsewhere. “The storage problem has largely been solved… Simply being able to load very large documents into a model is no longer impressive on its own,” Ahmed explains.

Instead, the challenge is retrieval. With 90% of enterprise data locked across fragmented systems, according to International Data Corporation, models often fail not because they lack intelligence, but because they cannot access the right information at the right time.

Even when retrieval works, reasoning remains difficult. Connecting insights across multiple documents, interpreting regulations, or understanding complex relationships requires structured reasoning—something current models struggle with.

This is why enterprises are increasingly investing in knowledge graphs, retrieval systems, and governed data pipelines, recognising that context quality matters more than context size.

The transition away from current architectures is unlikely to be dramatic or ideological. “Enterprises will not announce that they are abandoning transformers. They will simply ask for better outcomes and adopt whichever architectures deliver them,” Ahmed says.

This pragmatic shift is already visible. Hybrid approaches—combining transformers with alternative architectures such as convolutional neural networks for localised spatial feature extraction and state space models for faster inferencing—are gaining traction because they offer better efficiency and performance for specific use cases.

Transformer Models are Cracking

From a technical perspective, some leaders believe the limitations of transformers are becoming apparent.

AI company Pathway is building a post-transformer frontier model that solves continual learning. “Our CTO worked at Google Brain and was there when transformers were invented. Because of that, we’ve been able to anticipate their limitations and identify a way to fundamentally rethink AI at a deeper level—building systems that are not only more efficient, but can also get smarter over time and reason over longer horizons,” Zuzanna Stamirowska, Co-founder and CEO, Pathway, tells AIM.

“Transformers as the underlying math for AI were beautiful and fantastic… but they are definitely showing cracks right now,” she adds. ​​ 

These limitations can be mapped across three key areas: scale and cost, which make personalisation both difficult and expensive; energy consumption, which is now constrained more by power availability than compute capacity; and the lack of memory and continuous learning, forcing models to rely on repeated training rather than improving through experience.

“Transformers don't have memory and cannot learn with experience… This is not what LLMs are doing,” she adds.

This limitation has led to the rise of agent-based systems and context engineering, which attempt to compensate for these gaps—but often add complexity rather than solving the root problem. Sub-agents tend to operate on incomplete or conflicting assumptions. Engineers often hardcode complex, brittle logic into system prompts to force specific behaviours. And then there are cost and latency spirals.

To address these challenges, new architectures are drawing inspiration from how the human brain processes information.

New Architectures

At Pathway, the team compared how the human brain works with transformer models to identify what was missing. The goal was to move closer to a more brain-like approach to intelligence. This led them to develop a new architecture designed to unlock several of these capabilities.

“Just like your entire brain… you only call on the things that are important. You don’t light up 100% of your brain all the time,” Stamirowska explains.

The implications are significant, with early results suggesting up to 10x lower compute requirements for reasoning and reduced reliance on external retrieval systems.

The new architecture, as Stamirowska mentions, enables an effectively infinite context window by activating only the most relevant information, making it far more efficient. It also allows knowledge to accumulate without conflict, reducing the need for fine-tuning and eliminating the need for fragile context engineering.

“Instead of becoming Anthropic’s breakfast by relying on generic models, companies can build systems that are deeply personalised and data-driven,” she states.

On launching the architecture, Stamirowska mentions that the company has partnerships with AWS and NVIDIA that span both compute and go-to-market. “Once the models are ready for broader deployment, they will be made available directly through AWS.” 

Paradigm Shift?

Despite these challenges, not everyone believes transformers are nearing the end. “Hard to say. Honestly, it could go either way,” remarks Aravind Jayendran, Founder and CEO of AI-native startup LatentForce.

History suggests that AI evolves in cycles—some breakthroughs are paradigm shifts, while others are incremental improvements.

“For example, in 2012, the AlexNet moment was such a paradigm shift… the LLM era starting in 2022 was the result of large-scale scaling of transformers,” Jayendran states.

Transformers may not disappear, but evolve. What’s clear is that the future of AI will not be defined by a single model or architecture. Strong, well-governed data foundations will take precedence over sheer model size, while hybrid architectures will replace one-size-fits-all approaches to better suit diverse enterprise needs. 

As Stamirowska put it, “This is the time of post-transformers.”