GPU has an Energy Problem

While GPUs are fuelling a massive growth for NVIDIA thanks to the burgeoning demand from AI companies, the costs incurred by these companies don't end with just GPUs. The energy cost for running these GPUs in data centres is enormous. Recently, a study showed that data centres consumed approximately 1,000 kWh per square metre, which is about 10x the power consumption of a typical American household. BLOOM, an LLM, utilised 914kWh over an 18-day period while running on 16 NVIDIA A100 40GB GPUs, managing an average of 558 requests/hour.  Climbing Costs As per an article by Sequoia, if $1 is spent on a GPU, approximately another dollar is spent on energy costs for running that GPU in a data centre, and taking into consideration how the companies buying them will need to m