Apple unveils additional information on the LLM model within Apple Intelligence used to assist in summarizing text, correcting spelling errors, adjusting words, or prioritizing the importance of various texts. The fundamental component is the Apple Foundation Models, which are Apple’s own models.
The Apple Foundation Models are trained on the AXLearn framework released by Apple as open-source since 2023. This model is built from JAX and XLA, with the actual models trained on Google’s TPU chips and Apple’s GPUs. The training data is purchased or extracted from websites through AppleBot. Websites can opt to include a robots.txt file to prevent Apple from extracting their content. The final tuning is done through human reinforcement learning feedback (RLHF).
The Apple Foundation Models are divided into two: on-device models support 49K terms, while server models support 100K to accommodate additional languages. In practical use, each feature is fine-tuned using a combination of LoRA 2-bit and 4-bit, averaging to 3.5-bit. These models can deliver responses within 0.6ms and generate answers at 30 tokens/s on an iPhone 15 Pro.
Apple tested the Apple Foundation Models, comparing the on-device models with open-source models, successfully outperforming Mistral-7B, Phi-3-mini, and Gemma-7B, despite the models being as small as 3B. In contrast, server models compete with GPT-3.5-Turbo and even surpass GPT-4-Turbo in certain aspects.
TLDR: Apple introduces the LLM model in Apple Intelligence for text summarization and correction, trained on Apple Foundation Models utilizing RLHF and achieving impressive performance metrics in testing.
Leave a Comment