Article complete

Apple Optimises LLMs for Edge Use Cases

Apple tackles the challenge of efficiently running LLMs that exceed the available DRAM capacity

Apple Optimises LLMs for Edge Use Cases

Key takeaways

  • Apple introduces a method to run large language models on devices with limited DRAM capacity.
  • Utilize flash memory to store model parameters, optimizing data transfer and access speed.
  • Employ techniques like 'windowing' and 'row-column bundling' for efficient inference on large models.
  • Achieve up to 25x faster inference compared to traditional loading methods on CPU and GPU.
  • Integrate generative AI into iOS 18 to enhance Siri and other applications.

Takeaways drafted with AI from the article text.

https://twitter.com/1littlecoder/status/1737353316634374312

Their method involves constructing an inference cost model that aligns with the behavior of flash memory, guiding optimization efforts in two crucial areas: reducing the volume of data transferred from flash and reading data in larger, more contiguous chunks

Within this flash memory-informed framework, Apple employs two principal techniques. First, "windowing" strategically reduces data transfer by reusing previously activated neurons, and second, "row-column bundling," tailored to the sequential data access strengths of flash memory, increases the size of data chunks read from flash memory.

These methods collectively enable running models up to twice the size of the available DRAM, with a 4-5x and 20-25x increase in inference speed compared to naive loading approaches in CPU and GPU, respectively.

This research is significant as Apple plans to integrate generative AI capabilities into iOS 18. The new OS will leverage generative AI technology to enhance Siri and the Messages app, enabling them to answer questions and auto-complete sentences more effectively. Apple is also exploring the potential use of generative AI in apps such as Apple Music, Pages, Keynote, and Xcode.

Apart from Apple, Samsung recently introduced Gauss, its own on-device LLM. According to reports, Gauss is set to be incorporated into the upcoming Galaxy S24 smartphone, slated for release in early 2024. The company intends to integrate this language model into its devices such as phones, laptops, and tablets to enhance the capabilities of its smart devices.

Moreover, Google has announced its on-device LLM, called Gemini Nano, which is set to be introduced in the upcoming Google Pixel 8 phones, offering capabilities such as Summarize in the Recorder app and Smart Reply in Gboard. 

Published

Corrections: AIM corrects errors of fact and notes the change on the story. Spotted one? Tell the editors. Read our editorial standards.

About the author

Tech Journalist

Siddharth is a media graduate who loves to explore tech through journalism and putting forward ideas worth pondering about in the era of artificial intelligence.

All stories
Belamy · Daily

The day’s AI news, before your first meeting.