LLM Inference on Edge: A Fun and Easy Guide to run LLMs via React Native on your Phone!

As LLMs continue to evolve, they are becoming smaller and smarter, enabling them to run directly on your phone. Take, for instance, the DeepSeek R1 Distil Qwen 2.5 with 1.5 billion parameters, this model really shows how advanced AI can now fit into the palm of your hand!

In this blog, we will guide you through creating a mobile app that allows you to chat with these powerful models locally. The complete code for this tutorial is available in our EdgeLLM repository. If you've ever felt overwhelmed by the complexity of open-source projects, fear not! Inspired by the Pocket Pal app, we will help you build a straightforward React Native application that downloads LLMs from the Hugging Face hub, ensuring everything remains private and runs on your device. We will utilize llama.rn, a binding for llama.cpp, to load GGUF files efficiently!

Why You Should Follow This Tutorial?

This tutorial is designed for anyone who:

Is interested in integrating AI into mobile applications
Wants to create a conversational app compatible with both Android and iOS using React Native
Seeks to develop privacy-focused AI applications that operate entirely offline

By the end of this guide, you will have a fully functional app that allows you to interact with your favorite models.

0. Choosing the Right Models

Before we dive into building our app, let's talk about which models work well on mobile devices and what to consider when selecting them.

Model Size Considerations

When running LLMs on mobile devices, size matters significantly:

Small models (1-3B parameters): Ideal for most mobile devices, offering good performance with minimal latency
Medium models (4-7B parameters): Work well on newer high-end devices but may cause slowdowns on older phones
Large models (8B+ parameters): Generally too resource-intensive for most mobile devices, but can be used if quantized to low precision formats like Q2_K or Q4_K_M