NVIDIA  recently announced its latest innovation, the next-generation NVIDIA GH200 Grace Hopper platform. This platform centers around an innovative Grace Hopper Superchip, featuring the world’s pioneering HBM3e processor. 

The new setup is a game-changer, offering 3.5 times more memory capacity and 3 times more bandwidth than the current version. This setup includes a single server with 144 Arm Neoverse cores, delivering eight petaflops of AI performance and featuring 282GB of the latest HBM3e memory technology.

This technology has been carefully created to meet the needs of fast computing and the exciting world of generative AI. HBM3e memory, which is 50% faster than current HBM3, delivers a total of 10TB/sec of combined bandwidth, allowing the new platform to run models 3.5x larger than the previous version, while improving performance with 3x faster memory bandwidth. NVIDIA said that it expects to deliver this new AI chip by the second quarter of 2024. 

The new platform uses the Grace Hopper Superchip, which can be connected with additional Superchips by NVIDIA NVLink, allowing them to work together to deploy the giant models used for generative AI. This high-speed, coherent technology gives the GPU full access to the CPU memory, providing a combined 1.2TB of fast memory when in dual configuration.

Races Ahead of AMD and Intel

Nvidia’s new gen GH200 is better than AMD M1300X which is supposed to come later this year.  Recently,  to challenge Nvidia, AMD integrated an additional 64 gigabytes of HBM3 memory into the M1300X. 

The M1300X combines CNA3 with an industry-leading 192 gigabytes of HBM3 memory, delivering memory bandwidth of 5.2 terabytes per second which is lesser than 10TB/sec provided by GH200. Not only AMD, NVIDIA has also left behind Intel which lately is trying to catch up in the race to create GPUs that can be used to train LLMs. 

GH200 is way ahead of Intel’s Gaudi2. The Gaudi2 memory subsystem includes 96 GB of HBM2E memories delivering 2.45 TB/sec bandwidth, in addition to a 48 MB of local SRAM with sufficient bandwidth to allow MME, TPC, DMAs and RDMA NICs to operate in parallel.

Continuing in the same vein, Intel has declared its entry into the AI chip domain this year, gearing up to compete with both NVIDIA and AMD. The forthcoming “Falcon Shores” chip by Intel is anticipated to showcase an impressive 288 gigabytes of memory capacity while supporting 8-bit floating point computation.

Falcon Shores will use regular ethernet switching, similar to Intel’s Gaudi architecture designed for AI tasks. It will have various amounts of HBM3 memory and adaptable I/O, probably indicating different memory options. Intel mentions it will offer up to 288GB of HBM3 memory and a total memory speed of 9.8 TB/s.

Intel’s Falcon Shores was delayed and is set to debut in 2025 with only GPU cores and it is uncertain about when they’ll introduce CPU cores to it. In their latest earnings call Intel CEO Pat Gelsinger said that Gaudi3 will be the volume product for next year and they are already working on Falcon Shores 2 for 2026. 

Considering that NVIDIA’s GH200 is slated for an earlier release than Falcon Shore and Falcon Shore 2, the prospect of Intel surpassing NVIDIA seems quite improbable. Intel has outlined an array of ambitious strategies that they must successfully implement in order to outpace NVIDIA.

What about the software side?

One of the important aspects of NVIDIA’s hardware success is CUDA which enables parallel computing. One can say CUDA is the moat for NVIDIA when it comes to AI 

To compete with CUDA ,AMD  recently released an update to RocM. This is an important step forward for AMD, as CUDA has been one of NVIDIA’s biggest moats, especially in AI-accelerated workloads. 

The large amount of memory bandwidth available on the chip  through RocM will allow companies to buy lesser GPUs, making AMD an interesting value proposition for smaller companies with light to medium AI workloads. Similarly Intel also announced improvements on their CUDA alternative called oneAPI. 

While NVIDIA has an upmarket strategy, AMD and Intel can continue to focus on opensource solutions for smaller companies to compete with NVIDIA.