Apple has been at the forefront of almost all the major technological developments since the beginning. Three years ago, the company was giving tough competition to Google, Microsoft, and Amazon in the AI race as well by acquiring the most number of AI startups. Recently, Apple also acquired WaveOne, a startup developing AI algorithms for video compression. But when it comes to the current capabilities of large language models (LLMs) and generative AI, the company has been surprisingly quiet.

Some might even say that the company is merely being stubborn, refusing to adapt to the changing landscape of AI. But there might be legitimate reasons as to why Apple is staying away from this popular field. One such reason is also that the company values privacy and security above all else. 

While LLMs can definitely be useful in certain applications, they also require vast amounts of data to be effective. This data often includes personal information, and the more data a model has access to, the greater the risk of privacy breaches and leaks. Apple has always been very careful with user data, and have maintained that they don’t want to take any unnecessary risks in this arena.

But it also seems like the company is feeling like it may be missing out. According to a report by The New York Times, Apple had held an internal event focusing on AI and LLMs. The participants included the Siri team and reported that it is testing “language-generating concepts”. Moreover, 9to5Mac reported that Apple has introduced a framework for ‘Siri Natural Language Generation’ in tvOS 16.4. 

However, even this does not reveal whether the company is actually building its own LLM-based technology or adopting one from others. 

Interestingly, Apple had published a paper in 2021 alongside OpenAI, Google, Stanford University, UC Berkley, Harvard University, and Northeastern University, titled Extracting Training Data from Large Language Models. The paper talks about the security and privacy issues of training language models on sensitive data. Since then, Apple has been quiet about language models. Moreover, the researcher from Apple who worked on the paper, Úlfar Erlingsson, left Apple the same year to join Lacework. He had also been a researcher at Microsoft and Google Brain before joining Apple.

On the other hand, recently Apple released Transformers architecture which is optimised for Apple Silicon—the technology that is behind the current LLM and GPT technology. It is quite possible that the company is making strides in language models or maybe even developing its own chatbot. 

“What happens on your iPhone, stays on your iPhone” 

As mentioned before, Apple’s highest priority is users’ privacy and security. That comes with its own set of challenges for the company, particularly in the context of AI. 

Firstly, since privacy is the primary concern, the company has been pushing federated learning on all its devices. This means that the device does not share any personal data with anyone else. This would also mean that training the model on the device itself—without sending the data on the cloud—requires a lot of resources when it comes to LLMs. 

Therefore, it all comes down to a simple question. Can Apple run this huge system of models on single devices, i.e., on edge, like a private LLM? There are ways to solve it and interestingly, OpenAI’s co-founder Ilya Sutskever said in a conversation with Jensen Huang at NVIDIA GTC 2023 that he also believes that the company should aim towards that—model compression.

GPT-X types of models require a huge amount of cloud computing power to run on our browsers. Compressing the models using methods such as knowledge distillation and quantisation can significantly reduce the size of the models, but also hamper their performance and capabilities. 

However, Meta’s LLaMA model showed the capability of running smaller models with fewer parameters on a single device. If we build models using these, they might be best suited for on-device processing, even though they might not be as powerful as ones running on cloud, but that is only for now. Development in the field is evidently advancing at a rapid rate at the moment.  

Additionally, further advancements have shown that it is possible to build and run ChatGPT-like models on a single-device. Colossal-AI leveraged a PyTorch-based implementation, covering all the stages of training a model with only a 1.62 GB GPU memory usage, which is a single consumer-grade GPU. Similar breakthrough was also achieved by Hugging Face by enabling 20-billion-parameter models on a smaller GPU. 

Then, there are releases like BigScience Workshop’s Petals, which enable running LLMs with more than 100 billion parameters on a device. But the problem is that it works with a shared network, by loading a smart part of your model onto other devices, sort of like Torrent. The problem is that these releases still do not tackle the biggest concern that Apple has—privacy. 

Yet considering what these models can do, this is definitely good news for Apple. Utilising M1 capabilities for its models is a good way to go ahead and build a private LLM, and the developer ecosystem is definitely growing around it. 

Further, Huizi Mao from NVIDIA explains how Apple should be worried about LLMs as they are proving to be a powerful human-computer interface. This can definitely benefit Apple to build a brand new OS/ecosystem, just like how Apple presented the touch screen to the world.

But What if Apple Does Not Care?

Recently, a lot of researchers like Gary Marcus, Yoshua Bengio, Elon Musk, and a thousand others have signed a petition to pause any further training beyond GPT-4. What appears to be a fear-driven agenda might only be a concern that OpenAI is progressing too much too fast. Interestingly enough, Steve Wozniak, co-founder of Apple, has also signed on the petition. 

Several reasons can be drawn for this particular move. One can be that the company actually believes that models like GPT-4 or ChatGPT are a danger, or perhaps useless for them. Or, it can be that Apple has plans to build an alternative to Google and OpenAI’s models. If the latter is true, the company has to be quick about it. 

However, if the former is true, Apple has switched from being a research and technology company to a consumer-product or a hardware company. Though even that shouldn’t stop them from making a chatbot based on LLMs, as that’s what’s minting the most amount of money for all the big tech companies at the moment. 

Apple is more focused on AR/VR at the moment. Most of its investment has been channelled to that space. There is no apparent reason for the company to invest in LLM-based technology at the moment with so much competition, while also sacrificing its privacy and security concerns.

Moreover, staying out has proven good for Apple. The company is taking the same approach with AI as it did with search engines. Apple receives around $20 billion per year from Google to use their search engine in Safari. Now, with Microsoft’s Bing getting better, it might be up to Apple to decide which way to go.

Thus, it might be safe for Apple right now to stall the idea of building LLMs and just partnering with the other giants to integrate theirs in Apple devices.