For AMD, AI Success Now Rides on ROCm

A chipmaker can pack more transistors onto silicon, increase memory bandwidth, refine packaging technologies, and push manufacturing processes to their limits. But what ultimately determines success is how much of that capability developers can actually access through compilers, runtimes, kernels, libraries, and frameworks. Having a strong software stack has become as important as the hardware itself in the AI era.

For AMD, that has long been the challenge. The company's hardware looks favourable on paper. The software stack connecting developers to that hardware has been a different story.

That software layer is ROCm, AMD's open-source GPU computing platform. 

For nearly a decade, it has been tasked with competing against CUDA, NVIDIA's software ecosystem, whose advantage extends far beyond hardware into frameworks, tooling, libraries, documentation, and years of accumulated developer familiarity.

ROCm 7 Simplifying Deployment

AMD has increasingly shifted toward upstream contributions and official support in major AI frameworks such as PyTorch, TensorFlow, Triton, and OpenXLA over several ROCm releases. ROCm 7.0 continues that strategy.

The company has also expanded automated testing across major AI frameworks and increasingly ships pre-packaged ROCm-enabled builds for tools such as PyTorch, vLLM, and SGLang, reducing the software development effort.

With ROCm 7.0, AMD has widened framework integration, added FP8 support to more workloads, expanded Radeon and Ryzen AI compute support and pushed more of the stack into pre-packaged builds and upstream codebases.

In an interaction with AIM, Ramine Roane, AMD’s Corporate Vice President of AI, asserts that switching to AMD hardware no longer requires rewriting code. “There is no migration path. It just works out of the box, plug out a GPU, plug in a GPU, and it still works,” he says.

Roane argues that much of the perception around ROCm being difficult to deploy comes from experiences with earlier versions of the stack, when developers often had to build components themselves and manage a long chain of dependencies.

A key part of simplifying the software stack is TheROCK, a newer, streamlined implementation that AMD has positioned as a lighter and more modular foundation for ROCm. Roane says TheROCK builds faster, carries fewer dependencies, is easier to install, and allows developers to create significantly smaller containers. “It’s much more lightweight, nimble, modular, and better documented than the previous ROCm.”

AMD has also pushed support into the frameworks developers already use rather than expecting them to interact directly with ROCm. “What it means is that you actually use PyTorch in all cases, irrespective of the vendor,” says Roane.

According to him, that has become one of the most misunderstood aspects of ROCm. AMD's goal is not to convince developers to learn a new framework or rewrite existing applications, but to ensure that familiar tools such as PyTorch, vLLM, and SGLang work across different hardware platforms.

However, the company acknowledges that low-level developers working on custom kernels, compiler tooling, or performance optimisation still face a different experience. They continue to rely on tools such as HIPCC and ROCm profilers and may need to port or tune code when targeting different hardware architectures. 

The goal now is to make the underlying platform more modular and easier to work with rather than to eliminate the optimisation process itself. Those changes aim to address long-running complaints about installation complexity, dependency management, container size, and source builds.

Overall, the improvements for ROCm are beginning to show up in independent assessments as well. 

Last month, SemiAnalysis reported that AMD's MI355X had become roughly 40% cheaper than NVIDIA’s B200 for single-node FP8 serving on GLM-5 LLM workloads using SGLang, largely due to software-driven performance gains achieved in the weeks following launch. The analytics firm nevertheless noted that NVIDIA retains advantages in areas such as FP4 inference and some distributed inference deployments, highlighting that ROCm's progress has narrowed the gap rather than eliminated it.

AMD ALERT 🚀 MI355 is now 40% cheaper than B200 on GLM5 architecture for Single Node serving FP8 14 weeks after the initial launch of GLM5 on both non-MTP & MTP with spec decode for SGLang v0.12 for both CUDA & ROCm. SPEED IS THE MOAT!! Great work to @AnushElangovan, @roaner,… pic.twitter.com/o0iPWy5hH0

— SemiAnalysis (@SemiAnalysis_) May 19, 2026

Why AMD is Betting on Radeon and Local AI

Discussions on Reddit, Hacker News, and AMD’s own repositories have repeatedly called for broader ROCm support across Radeon hardware, arguing that consumer adoption is essential if AMD wants developers to build AI applications on its platform.

Roane says AMD increasingly sees local AI development as an important part of the ecosystem rather than a niche use case.

While AMD’s recent AI push initially centred on the MI300X data centre GPU accelerator and large-scale enterprise deployments, the company has spent the past year extending many of those software capabilities to Radeon graphics cards as well.

Roane notes that Radeon was not officially covered by ROCm until relatively recently. Once support arrived, the wider community began extending and maintaining support across a broader range of hardware than AMD officially validates.

AMD’s AI software strategy was originally shaped by its high-performance computing heritage. GPUs used in systems such as the Frontier supercomputer were designed primarily around HPC workloads, where FP64 performance was a key requirement. The move into generative AI required support for lower-precision formats such as FP16, FP8, and 4-bit workloads, prompting AMD to prioritise software development around Instinct accelerators first.

“We really didn't focus on Radeon at all, and Radeon wasn't enabled with all the AI goodness we had done, all the kernels, the optimised attention kernels and FP8 support and all that,” Roane adds.

That approach has since changed. According to Roane, AMD has spent the past year bringing many of those AI-focused software improvements to Radeon hardware, including memory optimisations and inference capabilities that were initially developed for data-centre products. The decision was partly driven by demand from developers looking to run AI workloads locally.

Roane has been pushing internally for broader Radeon support because he saw firsthand how difficult it was for newcomers to find accessible hardware for experimentation.

“I have nephews who were in high school and were trying to find a GPU on which they could experiment with a few things, like training a tiny little model, and Radeon was not an option at the time,” he shares. “Now they all switched to AMD. They're all training using ROCm and Radeon and doing inference as well.”

Community discussions suggest that local inference on consumer Radeon hardware has become substantially easier than it was only a few years ago, largely because of broader hardware support and improved framework integration.

While questions around training workflows have not disappeared entirely, recent debates are more likely to focus on performance, optimisation, and hardware support than on whether ROCm can run AI workloads at all.

ROCm, Vulkan, and Software Race with CUDA

That has also led many AMD users to local inference tools such as llama.cpp, Ollama, and LM Studio, often with Vulkan underneath.

Community benchmarks continue to compare Vulkan and ROCm on Radeon hardware, with some reporting stronger token generation from Vulkan and better prompt processing from ROCm.

AMD does not view those projects as competitors. Roane describes them as complementary parts of the broader open-source AI ecosystem and noted that some AMD engineers use Vulkan-based tooling themselves.

He also points to Lemonade, an open-source project developed within AMD that combines technologies such as llama.cpp and Vulkan while targeting AMD CPUs, GPUs, and NPUs simultaneously. According to AMD, the software can distribute different stages of inference across Ryzen hardware, including using the NPU for prompt processing and the integrated GPU for token generation.

While ROCm remains AMD’s primary software platform, developers increasingly choose between multiple inference layers depending on their workload. Production deployments often favour frameworks such as vLLM and SGLang, while local deployments frequently rely on llama.cpp, Ollama, or LM Studio.

AMD's focus, according to Roane, is less about steering developers towards a single stack and more about ensuring its hardware performs well regardless of which open-source tools they choose.

“We totally embrace LLVM, llama.cpp, Vulkan. It’s part of what we sometimes use ourselves. We love to use open-source projects where they give good results,” says Roane.

Some of the concerns that defined ROCm’s reputation have not yet disappeared, but the debate has shifted from basic functionality to performance tuning and optimisation.

Even now, ROCm is still competing with CUDA's far deeper library base, documentation and developer familiarity, and SemiAnalysis continues to argue that NVIDIA retains an edge in software maturity and ecosystem depth.

For AMD, it is no longer an argument that the solution lies in software. Pointing to recent SemiAnalysis findings, Roane says, “Just getting a 40% boost overnight tells you the importance of software.”

“You can’t really get much more than 40% nowadays when you move from one technology node to another. In software, you can get it in a week.”