NVIDIA Sells GPUs, Not Shovels

By now, most of you are familiar with the infamous meme doing the rounds on the internet – “When everyone digs for gold, sell shovels.” This very aptly captures the gold rush created by generative AI, depicting NVIDIA selling shovels, while Microsoft, Google, and Meta are busy buying them. Ironically, we wish it was as easy as selling shovels instead of GPUs. 

Surprisingly, NVIDIA seems to be optimistic amid all the rumours about GPU crunch, and how GPUs are the new currency.

OpenAI’s Andrej Karpathy said, “who gets how many GPUs’ is the top gossip of the Silicon Valley right now”. Even Elon Musk had once said – “GPUs are at this point considerably harder to get than drugs.”

On multiple occasions, Sam Altman as well had shared similar views, where he said the shortage is delaying OpenAI’s short term plans, including multimodality, fine-tuning, 32K context windows, and others. “We are so short on GPUs, the less people use our products the better,” said Altman, saying that they’d love it users use it less because they don’t have enough GPUs.

Many cloud provider execs have also said that the capability of large scale H100 clusters at small and large cloud providers is running out. 

NVIDIA is all Sorted for the Next One Year 

Recently, NVIDIA has confidently projected a revenue of $16 billion for the upcoming third quarter of FY23. Looks like it has an unlimited supply of shovels (GPUs, obviously) to cater to the world’s AI needs. The same sentiment was resonated in the recent record-breaking earnings call where its chief financial officer Colette Kress said that it has everything under control. “NVIDIA has developed and qualified additional capacity and suppliers for key steps in the manufacturing process such as co-op packaging,” she added, implying that they expect supply to increase in the coming quarters. 

Meanwhile, NVIDIA is set to significantly increase the production of its leading H100 AI processor, aiming to triple it at the very least, as reported by Financial Times. According to sources familiar with NVIDIA's plans, the company intends to ship between 1.5 million and 2 million H100s in 2024, a substantial surge from the anticipated 500,000 units for this year. 

NVIDIA’s DGX Systems’ VP Charlie Boyle believes that GPU shortage is a supply chain problem, and not NVIDIA’s GPUs. “So when people use the word GPU shortage, they’re really talking about a shortage of, or a backlog of, some component on the board, not the GPU itself,” he explained. 

Who is Selling Shovels to NVIDIA? 

The main suppliers of NVIDIA are South Korea’s SK Hynix, TSMC, Samsung and others. 

TSMC plans to double its capacity for CoWoS, an advanced packaging technology needed to make NVIDIA’s H100 processor, but warned the bottleneck would still not be resolved until at least the end of 2024. 

Samsung Electronics is currently working with Nvidia on technical verification tasks for the HBM3 for GPUs and advanced packaging services. As soon as the technical verification procedures are completed, Samsung will supply HBM3 to Nvidia and is expected to take charge of the advanced packaging that processes individual GPU chips and HBM3 into a high-performance GPU, the H100. 

SK Hynix has also started the verification process for its high-performance HBM3E memory chip with NVIDIA, the world’s top AI chip designer. Earlier this week, the Korean memory chipmaker said it started shipping out samples of its HBM3E, an extended version of the HBM3, to its clients including NVIDIA. 
This time, NVIDIA is well-equipped to address any obstacle, thanks to the strong supply chain ecosystem of suppliers to sell its GPUs.

It seems to be ahead of its time already. Recently, NVIDIA introduced a GPU named L40S, which doesn’t require as much packaging used in H100 chips. This difference makes the new chips less susceptible to manufacturing bottlenecks. Huang explained this new chip is purposely built to fine-tune LLMs and not to train them. In other words, the L40S chips might be beneficial for smaller AI startups that are fine-tuning models like Meta's open-source Llama 2, and not building them from scratch.