Apple Optimises LLMs for Edge Use Cases

Apple has published a paper ‘LLM in a flash: Efficient Large Language Model Inference with Limited Memory’ outlining a method for running LLMs on devices that surpass the available DRAM capacity. This involves storing the model parameters on flash memory and bringing them on demand to DRAM.

Their method involves constructing an inference cost model that aligns with the behavior of flash memory, guiding optimization efforts in two crucial areas: reducing the volume of data transferred from flash and reading data in larger, more contiguous chunks

Within this flash memory-informed framework, Apple employs two principal techniques. First, "windowing" strategically reduces data transfer by reusing previously activated neurons, and second, "row-column bundling," tailored to the sequential data access strengths of flash memory, increases the size of data chunks read from flash memory.

These methods collectively enable running models up to twice the size of the available DRAM, with a 4-5x and 20-25x increase in inference speed compared to naive loading approaches in CPU and GPU, respectively.

This research is significant as Apple plans to integrate generative AI capabilities into iOS 18. The new OS will leverage generative AI technology to enhance Siri and the Messages app, enabling them to answer questions and auto-complete sentences more effectively. Apple is also exploring the potential use of generative AI in apps such as Apple Music, Pages, Keynote, and Xcode.

Apart from Apple, Samsung recently introduced Gauss, its own on-device LLM. According to reports, Gauss is set to be incorporated into the upcoming Galaxy S24 smartphone, slated for release in early 2024. The company intends to integrate this language model into its devices such as phones, laptops, and tablets to enhance the capabilities of its smart devices.

Moreover, Google has announced its on-device LLM, called Gemini Nano, which is set to be introduced in the upcoming Google Pixel 8 phones, offering capabilities such as Summarize in the Recorder app and Smart Reply in Gboard.