WWDC 24: Running Mistral 7B with Core ML

WWDC’ 24 is the moment Apple officially unveiled Apple Intelligence and reiterated their commitment to efficient, private, and on-device AI. During the keynote and the sessions that followed, they demonstrated Apple Intelligence, which powers a huge array of AI-enhanced features that show practical uses for everyday tasks. These are not *AI-for-the-sake-of-AI* shiny demos. These are time-saving, appropriate (and fun!) helpers that are deeply integrated with apps and the OS, that also offer developers a number of ways to include these features within their own apps.

Apple Intelligence features can only work this well because of the vertically integrated software stack that harnesses Apple Silicon's capabilities to the fullest. Apple also offers a platform for developers to run models on-device, known as Core ML. This software stack allows you to run ML models across all 3 compute units (CPU, GPU & Neural Engine) available on Apple Silicon hardware.

In this blog post, we’ll be exploring some of the best new Core ML features to replicate the Mistral 7B example Apple showcased in the WWDC’24 Deploy machine learning and AI models on-device with Core ML session, where they use a fork of swift-transformers to run a state-of-the-art LLM on a Mac. This is a high-quality model with more than 7 billion parameters that pushes the capabilities of consumer hardware today. You can also check out WWDC’24 Bring your machine learning and AI models to Apple silicon session, where part of the Mistral 7B conversion process is shown.

Let’s see what steps to take to run it as efficiently as possible, and learn the new tools available in iOS 18 & macOS Sequoia.

This is what we’ll be building today:

Video: Mistral 7B running with Core ML.

TL;DR

By the end of this blog post, you will have learnt all the new goodies accompanying the latest macOS release AND you will have successfully run a 7B parameter model using less than 4GB of memory on your Mac.