What is Meta doing with 150K GPUs?

In Silicon Valley, the talk of the town in 2023 has been about who acquired how many GPUs, given the shortage of NVIDIA H100s. According to a recent report from Omdia Research, both Meta and Microsoft have received 150,000 H100 GPUs from NVIDIA, surpassing allocations to Google, Amazon, and Oracle. 

The quantity is notably substantial for Meta, the parent company of Facebook and Instagram, especially considering that it's not a cloud service provider like Microsoft, but rather the creator of the open-source Llama models. 

From Microsoft's standpoint, it's evident that they are supplying GPUs to support OpenAI in developing their upcoming models. Unverified leaks suggest that GPT-4 underwent training using approximately 25,000 NVIDIA A100 GPUs over a period of 90 to 100 days.

In addition to running its social media products, such as Instagram and Facebook, and running ads on them, Meta has been at the forefront of various AI research and open-source developments. This definitely hints that the company has been using its GPUs for good.

According to a blog post, Meta aimed to acquire 25,000 or more GPUs, potentially up to 100,000 in 2023.

A Year of Meta Projects 

Since the beginning of the year, Meta has been consistently rolling out new products, prompting the need for more GPUs. Moreover, Meta recently celebrated 10 years of FAIR Fundamental AI Research, its AI lab. While others are building mass products in B2C space, Meta is entirely focused on the developer ecosystem with its slew of open source models. 

Source: Meta 

Starting in February, Meta released LLaMA, followed by Toolformer—a model trained to determine the optimal APIs to call, when to call them, what arguments to pass, and how to best incorporate the results into future token predictions.

In May, Meta introduced ImageBind, an AI model capable of binding data from six modalities at once, including images and video, audio, text, depth, thermal and inertial measurement units (IMUs). 

In July, it introduced the multimodal model 'CM3leon' (pronounced: chameleon), capable of both text-to-image and image-to-text generation. During the same month, Meta unveiled LLaMA's successor, Llama 2.

In August, Meta announced that DINOv2, its computer vision model trained through self-supervised learning to produce universal features, is now available under the Apache 2.0 license. 

In September, Meta hosted Meta Connect 2023, integrating AI features across its social media apps, including WhatsApp, Instagram, and Messenger. The company introduced AI stickers, powered by Llama 2 and Emu, for real-time image generation, seamlessly converting text into diverse, high-quality stickers.

Furthermore, as part of its ongoing commitment to advancing communication technologies, Meta released Seamless Communication. This is a family of AI-powered language translation models designed to foster more natural and real-time communication across different languages.

Comprising three components — SeamlessM4T v2, SeamlessStreaming, and SeamlessExpressive — this innovative release showcases Meta's dedication to pushing the boundaries of AI applications.

Simultaneously, Meta introduced AudioCraft, an open-source generative AI framework which enables users to generate high-quality, realistic audio and music from simple text prompts. It is a single codebase for all generative audio needs, including music, sound effects, and audio compression.

Most recently, Meta introduced Emu Video, which leverages the Emu model for text-to-video generation based on diffusion models. This unified architecture for video generation tasks can respond to a variety of inputs: text only, image only, and both text and image."

Llama 3 Won’t be on Diet 

Now that we know Meta has lots of GPUs and where they used them, it won't be hard for them to make Llama 3 soon. As Meta keeps using Llama 2 in their products, they've gotten really good at getting the most out of it.

https://twitter.com/bindureddy/status/1732190117547454933

Rumors suggest that Meta has now turned its attention to Llama 3, aiming to enhance it even beyond the capabilities of GPT-4. According to several speculations, Meta will launch Llama 3 early next year. The best part about it is that it is going to be open source for research and commercial use as well. 

However, in a podcast with Lex Fridman, Meta chief Mark Zuckerberg expressed his contemplation over open-sourcing Llama 3. Additionally, he shared his intention to integrate Llama 3 into Meta products, potentially leading to an increased demand for GPUs.