A language model running on a phone can now generate responses fast enough to support practical applications without sending every prompt to a remote server. Google’s published LiteRT-LM benchmarks report 52 tokens per second and a 0.3-second time to first token for Gemma 4 E2B on the Samsung Galaxy S26 Ultra GPU. The model package is approximately 2.58 GB.
Those figures come from Google’s own tests, not an independent benchmark, and they do not represent every phone. Still, they demonstrate what modern mobile hardware can achieve.
The more interesting question is what developers should build with this capability. Adding a chatbot to an existing app is relatively straightforward. Building a copilot that reliably completes useful tasks, protects personal information and works on the devices people actually own requires a different approach.
What Makes an On-Device Copilot Useful?
An on-device large language model (LLM) runs its inference locally rather than relying on a remote model server. Depending on the application, it can summarize text, extract information from images, rewrite messages or translate natural-language instructions into structured application actions.
Consider an expense-tracking app. A user photographs a restaurant receipt. The model extracts the merchant, date and total, then prepares a transaction for review. The app validates the fields before saving anything.
The model handles language and interpretation. The application remains responsible for data storage, permissions and business rules.
This separation is essential. A convincing answer is not proof that the underlying task was completed correctly.
Choose the Right Model and Runtime
Developers have several options, and the simplest route is not always the most flexible.
- Android: Google’s ML Kit GenAI APIs expose supported on-device capabilities through Gemini Nano. They are a sensible starting point for summarization, rewriting, proofreading and other supported tasks.
- Custom Android inference: LiteRT-LM supports developer-selected models and accelerated inference, giving teams more control over model choice and deployment.
- iOS: Apple’s Foundation Models framework provides native access to an on-device model, including structured generation and tool calling on eligible devices.
- Cross-platform deployment: Projects such as llama.cpp and ExecuTorch offer alternative routes for running custom models.
Do not choose a framework simply because its model performs well on a flagship phone. Check device coverage, memory requirements, model distribution, hardware acceleration and the amount of integration work your team can support.
There is another constraint with managed Android inference: availability varies by device and model version. Google’s ML Kit guidance also describes per-application inference quotas and restrictions on background execution. A product that depends on uninterrupted background inference must account for those limits before release.
Smaller Models Can Be Better at Specific Tasks
Developers often assume that a larger model will produce a better mobile assistant. That is not necessarily true when the task has a narrow scope.
Google’s FunctionGemma evaluation illustrates the alternative. Its 270-million-parameter model achieved 58% on Google’s Mobile Actions evaluation before fine-tuning. After training it on the specific function-calling workflow, the reported result rose to 85%.
That improvement applies to the evaluated task; it does not mean fine-tuning will produce the same gain in another app. But the engineering lesson is useful: train and test a model against the actions your product needs rather than expecting general conversational ability to guarantee correct execution.
For an assistant with a fixed set of functions, a small specialized model may be a better fit than a general-purpose model that consumes more memory and still struggles with the application’s expected output.
Build a Safe Path from Prompt to Action
Suppose a user types, “Remind me to call the supplier tomorrow at 10 a.m.” The copilot needs to interpret the request and prepare a valid reminder. It does not need unrestricted access to the phone.
A practical implementation follows these steps:
- Parse the request. Ask the model to produce a structured reminder with a title, date and time.
- Validate the result. Check required fields, resolve the relative date using the device’s local date and timezone, and reject malformed values.
- Confirm ambiguity. Ask the user for missing information rather than inventing a date or recipient.
- Execute approved code. The app creates the reminder through its own authorized function.
- Report the outcome. Confirm success only after the application has saved the reminder.
Apple’s Foundation Models tool interface supports connecting model decisions to application code. However, a tool call should never bypass permission checks or validation merely because the model requested it.
Local AI Still Needs Memory Management
A model’s context window is not the same as permanent memory. It is the amount of information the model can process within a session.
Apple specifies a 4,096-token context window for its on-device foundation model. Instructions, conversation history, tool definitions and generated responses all consume that budget. Long sessions can therefore exhaust the available context even when the model itself continues running normally.
For a personal knowledge copilot, avoid repeatedly sending every saved note into the prompt. Index authorized documents locally, retrieve relevant passages and provide only those passages to the model. This retrieval-augmented generation approach reduces unnecessary context and gives the application source material against which it can check an answer.
Keep the system honest about missing information. If the relevant document was never indexed or the search returns nothing useful, the assistant should say so rather than inventing an answer.
Test the Experience on Real Devices
Benchmarks are useful for screening models, but they cannot tell you whether your application feels reliable after repeated use. Test the complete workflow on lower-end, mid-range and flagship phones.
At minimum, measure:
- Task success and incorrect tool calls.
- Time to first token and total completion time.
- Peak memory use, battery consumption and sustained performance.
- Offline completion rates and recovery from inference failures.
- Whether private information enters unauthorized logs, backups or network requests.
Include difficult inputs. Test blurry receipts, ambiguous reminders, long documents and requests that should trigger a clarification rather than an action. For regional products, evaluate the actual languages, accents and writing patterns your users employ instead of assuming general multilingual support is sufficient.
Privacy Is an Architectural Choice
Local inference can reduce the amount of sensitive information sent to model providers. It does not automatically secure everything stored on a phone.
Prompt injection remains a concern when models process imported documents or retrieved text. Google’s or Apple’s local execution environment does not make model output inherently trustworthy. The OWASP guidance on excessive agency explains how overly broad permissions and poorly controlled model actions can create security problems.
Keep available tools narrowly scoped, validate every output, protect local data and require confirmation for consequential actions. If a cloud model is needed, define what information may leave the device and obtain the appropriate user consent.
Start with One Workflow
The best first mobile copilot is not an unrestricted assistant that can do everything. It is a focused feature with clear inputs, measurable success and a safe way to handle uncertainty.
Choose one job, establish a non-AI baseline, integrate the smallest suitable model and test it on real hardware. Expand only when the evidence shows that the model improves task completion without imposing unacceptable latency, memory use or privacy risks.
That is how on-device AI becomes useful software rather than another chatbot inside an app.
