Apple researchers have developed a family of large multimodal language models called MM1, which can process and generate both text and visual data, according to a research paper presented last week. The study at Apple’s research labs aimed to build performant multimodal large language models (MLLMs) through careful ablation of various architectural components, data sources, and training procedures.

Read the paper here. 

The researchers found that image resolution and the capacity of the visual encoder had the highest impact on model performance, while the specific method of combining visual and text data mattered less. 

They also discovered that a careful mix of different data types was crucial, with interleaved image-text documents helping with few-shot learning, traditional captioned images boosting zero-shot performance, and including text-only data maintaining strong language understanding capabilities.

Based on these insights, the team developed the MM1 model family, ranging from three billion to 30 billion parameters, including dense and mixture-of-experts variants. After scaling up training, MM1 achieved state-of-the-art results on various multimodal benchmarks during pre-training.

Following further instruction tuning on a curated 1 million example dataset, the final MM1 models demonstrated competitive performance across 12 multimodal tasks, such as visual question answering and captioning. Notably, MM1 could perform multi-image reasoning and few-shot learning, critical capabilities enabled by the team’s careful multimodal pre-training approach.

This paper builds upon previous research into areas like CLIP for learning visual representations from natural language supervision, and autoregressive models like GPT for text generation. However, it is one of the first detailed studies focused specifically on large-scale multimodal pre-training.

The researchers hope their insights will accelerate progress, as Apple is reportedly in talks to integrate Google’s Gemini generative AI models into upcoming iPhone software.