Building an AI agent that runs entirely on your iPhone.

I wanted to find out how much of a useful AI product could fit inside one phone.

Not a chat screen wrapped around a hosted API. Not a demo that works only while the app remains open. I wanted an agent that could accept a real task, keep working through the iOS lifecycle, research current information, cite what it used, and turn the answer back into speech—all without shipping the user's prompts, transcripts, or voice sample to an inference service.

The result is Agent, a new open-source Swift application. You can type or dictate something like “find the best summer camps for my nine-year-old near Chicago.” The app plans searches, reads a bounded set of pages, reasons over those excerpts with Apple Intelligence's on-device model, saves the sources beside the result, and generates a short summary designed to be heard rather than read.

That product sentence is simple. The mobile engineering underneath it is not.

An agent is a lifecycle problem

The easiest agent demo is a single async function attached to a button. That falls apart as soon as the phone locks, the app backgrounds, iOS expires the task, or the process is killed.

Agent treats work as persisted state. Every task is a SwiftData model with a small state machine—queued → running → completed/failed—plus timestamps, a progress log, cited sources, a spoken summary, and detailed findings. The UI observes those records; it does not own the work.

A singleton AgentRunner drains the queue from two entry points. The foreground path runs when the app becomes active or a new task is created. The background path uses BGProcessingTask, registered before launch finishes because that is a hard requirement of BGTaskScheduler.

Cancellation is part of the design rather than an error case. When iOS calls the expiration handler, the current task is cancelled and returned to the queue. If the process dies before cleanup, the next launch finds jobs stranded in running and requeues them. The system provides at-least-once execution without pretending the operating system owes the app unlimited background time.

The UI shows an agent. The architecture sees a recoverable state machine.

SwiftData contexts are not thread-safe, so every mutation from the runner goes through an @ModelActor. Views continue to read through @Query on the main context. That separation keeps persistence correct without pushing database details into the SwiftUI layer.

The model never gets a network connection

Current information creates an uncomfortable choice for a private, local agent: either give the model network tools or accept stale answers. I chose a third path. The application owns retrieval; the model only sees text the application has already admitted.

The flow is deliberately finite:

plan queries → search → interleave + dedupe → read N pages → excerpt → ground

Apple's Foundation Models framework first produces a guided list of search queries. The web layer sends them to DuckDuckGo or Brave, round-robins results so one query cannot consume the entire budget, canonicalizes URLs, and downloads a configurable number of pages. Each response is capped at 1.2 MB. The reader prefers <article> or <main>, reduces HTML to text, and falls back to the search snippet when a page cannot be read.

Only then does the model reason. It receives numbered excerpts and must cite those numbers inline. Sources are persisted on the job and shown as tappable links beneath the findings. If the on-device context window overflows, Agent retries with progressively shorter excerpts before falling back to unresearched general knowledge.

This boundary improves reliability as much as privacy. A small local model cannot wander through arbitrary tool calls, retrieval costs are predictable, and citation numbering stays deterministic.

It also makes the privacy story honest. When research is enabled, the task text goes to the selected search provider and the referenced pages are downloaded. Settings say that plainly. Turn research off and the entire task stays local.

Speech is not a feature; it is a pipeline

The other half of Agent is a complete local voice stack. It handles dictation for new tasks, live transcription, playback of completed summaries, and zero-shot voice cloning from a short reference recording.

AudioCapture taps AVAudioEngine in the hardware's native 44.1 or 48 kHz format and converts it to the 16 kHz mono Float32 stream the recognizer expects. A 20-million-parameter streaming Zipformer transducer decodes that stream on a private serial queue, publishing partial text while the user speaks and finalizing a segment when endpoint detection finds a pause.

Dictation and transcription share the same engine but not the same output path. In dictation mode, partial and finalized text go to composer callbacks rather than the transcript screen. There is only one capture session, so enrollment and recognition are mutually exclusive instead of racing for the microphone.

For synthesis, ZipVoice runs on another serial queue. A voice profile is intentionally small: a reference WAV file plus the words spoken in it. The model conditions on that pair at generation time, so cloning requires no training job and the sample never leaves the phone. A Vocos model turns the generated mel spectrogram into 24 kHz audio for playback.

Several small details separate a pipeline from a demo. Prompt audio is cached by file path, and every new enrollment gets a new filename so stale audio cannot survive in the cache. Near-silent samples are rejected before they produce a useless clone. The recognizer's uppercase, unpunctuated output is normalized for display, while terminal punctuation is added before synthesis because it materially improves prosody.

Keeping native inference off the main thread

The speech models run through sherpa-onnx and ONNX Runtime, delivered as prebuilt iOS .xcframework binaries. Swift reaches the runtime through a bridging header and C API. I kept the upstream Swift wrapper unchanged and added the missing ZipVoice call in a separate extension, which means the vendored wrapper can be replaced cleanly when the runtime is upgraded.

Thread ownership is explicit. SwiftUI and published state stay on the main actor. Audio capture callbacks arrive on the AVAudioEngine render thread. Recognition and synthesis each have private serial queues. SwiftData background work goes through the model actor. The agent runner is a Swift task that awaits each boundary.

That architecture is intentionally unglamorous. Audio callbacks cannot wait for UI work. Model inference cannot freeze scrolling. Database contexts cannot cross executors. Each subsystem has one owner and a narrow surface area.

The constraints are part of the project

Local inference makes one category of complexity disappear—accounts, tokens, usage billing, hosted model availability—and replaces it with another.

• The speech and voice models total 229 MB, putting the installed app at roughly 257 MB.
• Apple Intelligence reasoning is conditional on Foundation Models being available; a mock brain keeps the rest of the pipeline testable elsewhere.
• Background processing runs at iOS's discretion and cannot be exercised normally in the simulator.
• The keyless DuckDuckGo provider parses a no-JavaScript results page, so Brave is available as a keyed alternative when markup stability matters.
• The local network-call idea—send text instead of compressed audio and re-synthesize the caller's voice on the receiver—is still only a loopback experiment. The networking layer is not built.

I would rather make those constraints visible than hide them behind a broad “private AI” label. Good mobile architecture is often the work of defining what the system cannot promise, then making every failure recoverable and every boundary inspectable.

Why open source it?

This project is meant to be read as well as run. There are no Swift Package Manager or CocoaPods dependencies. The project file is generated from project.yml with XcodeGen, and one idempotent script reproduces the vendored frameworks and models. The README documents the architecture, source map, model sizes and licenses, thread ownership, failure behavior, and the tradeoffs above.

That is the part I most wanted to share. The interesting work in mobile AI is not putting a prompt behind a button. It is fitting model behavior into an operating system that can suspend you, an audio stack with real-time constraints, a persistence layer with executor rules, a device with finite memory and storage, and a privacy promise precise enough to verify.

Agent is available on GitHub. I hope it is useful as an app, but even more useful as a concrete example of what on-device AI looks like once it has to behave like real iPhone software.

---

Frequently asked questions