Choosing between buying or renting a GPU involves a similar decision-making process as any other asset, say a car. Renting a GPU over the cloud has the same benefits as renting a car. The financial burden is considerably lesser, and there is support available from cloud providers. Buying an on-premises GPU system means having to pay a one-time, fixed cost in the beginning. Companies factor in various parameters when looking into GPUs, the types of models being trained, the space of hyperparameters and the cost of electricity in the locality. 

Comparison

On-premises GPUs can be used by the companies at their disposal. Companies don’t have to consider costs when repeatedly using them and trying different methods. Sometimes when the company is using large datasets and very complex models, having on-premise GPUs helps developers with multiple training runs and gives them more time to experiment. In this instance, companies prefer to buy GPUs so that they have unlimited time. In fact, the more times an on-premises GPU system is used, the bigger is the return on the company’s investment. Also, if the company is involved with industries that work with sensitive data like healthcare and finance, it makes more sense to own the GPUs as they can be kept behind the company’s firewall protection. 

Renting GPUs is easier for small startups, researchers and students and is more suited for companies that want to experiment with more training models and don’t want to stick to a certain size or configuration of GPUs. 

Even though there is no thumb rule for it, a brand new startup usually builds prototypes on the cloud. When the company is growing and has raised enough money, it usually purchases a GPU-powered data centre. 

Factors 

Costs: While buying physical GPUs is expensive, renting cloud GPUs is chargeable on an hourly basis. With cloud GPU prices for AWS and Google Cloud increasing overtime, unpaid students and academics have also resorted to renting out GPUs from cryptocurrency miners, who are usually among the first to get their hands on them.

The global supply chain shortage weighed heavily on GPUs in 2021. However, in 2022, GPU prices dropped by 11 per cent in February, according to a digital trends report. Prices for some of the GPUs fell as far as 25 per cent when compared to January. The report stated that prices for all GPUs for the past two generations from both NVIDIA and AMD decreased. For older GPUs, the decline was even sharper. Out of a total of 33 GPUs, 20 had a fall in price in double digits. 

                Dropping prices, Source: digitaltrends

Availability: There is a fab-capacity shortage for manufacturing cutting-edge silicon, which has hit the GPU supply chain hard. It has become harder for interested customers to compete with cryptocurrency miners and scalpers. Users who are in desperate need have bought GPUs from scalpers, but this is far from being a secure option. There is a big possibility that the listing is a scam or the graphics cards have been used to mine cryptocurrency and have run their course. Graphic cards don’t have a measurement that tracks their usage time which makes it next to impossible for second-hand users to determine to what extent the card is already used.

There are ways to monitor GPU stock through social media spaces like Discord and Telegram, with notifications being sent by communities and bots. There are also automated email services like CamelCamelCamel, which send a notification mail immediately when a GPU is restocked. However, these services could also be available only to certain specific websites, which means that products normally run out of stock quickly after the notification is received. Instead, it is more effective to keep track of inventory at a local physical store because of lesser demand. 

        Source: ImageNet

Release of next-generation GPUs: Purchasing a GPU is a matter of timing. Even with falling prices, it is advisable to wait until companies have released new generations of graphic cards this year. While NVIDIA announced its H100 TensorCore GPU, which will be deployed in the market by the third quarter, at its annual GTC conference recently, AMD is yet to announce its new generation graphic cards. The release of new, pricier GPUs will most likely push down the prices of previous generation graphic cards even further. A day ago, Intel, too, said it would release its first Arc desktop GPU by “early summer.” 

Even if companies or individuals want to buy GPU systems, several market watchers are suggesting a waiting period until prices level even further. An increased supply of GPUs in the market could also be responsible for this. Earlier, NVIDIA said that it expects the GPU supply shortage to plateau by mid-2022. Pat Gelsinger, CEO of Intel, has also said that the supply chain crisis for GPUs will ease up this year and normalise by 2023. 

              Source: Reddit

The GPU shortage has pushed up prices for graphic cards that are a couple of generations old to highly inflated rates on eBay. Consequently, used GPUs have also entered the market in a big way. But it depends on whether buyers are willing to use older cards with less memory and save more money instead. It is entirely up to the company to decide whether the deal makes sense to them or not.