How to Use Uzu: The Beginner's Guide

CodexFuse Team · 2026-11-03T22:45:09.392Z · 12 min read min read

How to Use Uzu: The Beginner's Guide

Learning how to use Uzu starts with one clear idea: Uzu is an open-source, MIT-licensed inference engine from Mirai that runs large language models directly on your own device, using Apple Silicon's unified memory and Metal GPU for speed. The project describes itself in its README as "a high-performance inference engine for AI models" that lets you "deploy AI directly in your app with zero latency, full data privacy, and no inference costs." One disambiguation before we go further: this guide covers only the inference engine at github.com/trymirai/uzu — not Uzu the Raku static site generator, and not Yuzu the Nintendo Switch emulator, which only sounds similar. Once Uzu is running, it can sit underneath your local AI apps and agents, for example the kind of app built in our Dify beginner guide. This Uzu tutorial walks you from install to your first chat, then to serving models over a local API.

What Is Uzu, and Who Is It For?

Uzu is an inference engine, not a chatbot app. It is the layer that loads a model, runs the math, and hands tokens back — the engine room that an app, an agent, or a CLI talks to. Its Rust core ships bindings for Swift, Python, and TypeScript, so you can embed it in an iOS app, a Python service, or a Node project. Uzu currently supports Apple Silicon (iOS and macOS) devices through Metal and unified memory, with a CPU backend as a fallback. If you are a developer who wants a model running privately on a Mac — behind your own agent, your own app, or just your own experiments — that is exactly who Uzu is for.

Think of the local-AI stack in three layers. At the bottom sits the engine (Uzu), which does the raw inference. Above it sits the runner or manager that fetches models and keeps them organized — Ollama plays that role for many people. Above that sit the apps and agents that consume the inference, like the workflows in our Dify beginner guide. Uzu is the bottom layer, and it is designed to be fast there: its Metal kernels are written for Apple Silicon, and its newest versions add speculative decoding that drafts many tokens at once.

How Do You Install Uzu?

You install Uzu through your language's normal package flow — there is no installer download and, at the time of writing, no official Homebrew path, so stick to the four methods the README documents. This uzu install section pins version 0.6.15, released October 30, 2026, and I re-verified every command below against the live project pages on November 3, 2026. Uzu moves fast at this age, so if a command ever fails, the README is the authority.

1. Add the package to your project. In Python, that is uv add uzu==0.6.15. Rust, Swift, and TypeScript each have their own one-line form, listed in the table below.
2. Pick a model identifier. Uzu uses its own model naming, for example alibaba:qwen3.5:0.8b:mirai:mirai-m:4. The published model list lives on Mirai's site, and the medium (mirai-m) and large (mirai-l) tiers tell you the quantization: medium is 4-bit asymmetric integer, large is 8-bit symmetric.
3. Download the model on first run. The engine downloads the model for you automatically — the README's promise is that "everything from model downloading to inference configuration is handled automatically."
4. Open a chat session. You create an engine from an engine config, then ask the engine for a chat session on the downloaded model.
5. Send messages and read the replies. You pass chat messages to the session and get replies back, with streaming and tool calling available when you need them.

Those five steps are the whole arc: add the package, name the model, let it download, open a chat, and talk. The sections below show what each step looks like in practice.

How Do You Run Your First Uzu Chat?

You run your first Uzu chat by creating an engine, downloading a model, and opening a chat session. In Python the shape of the code follows a fixed order: build an EngineConfig, create the Engine, fetch the model with engine.model(...), download it with engine.download(model), open a session with engine.chat(model, ChatConfig.create()), and get replies with session.reply(messages, ChatReplyConfig.create()). Every step after the first is asynchronous, so the quick-start example wraps it in an async main function.

A few engine features matter the moment your first chat works. Chat instances can be shared, so you can load a model into memory once and reuse it across sessions instead of reloading it. Tool calling is built in: you decorate a Python function with uzu_tool_function, add it to the session with session.add_tool, and the model can call it during a reply. Structured output is supported through JSON-schema grammars, so you can ask the model to answer in a strict shape your code can parse. And if you would rather not write code at all, the project ships an interactive CLI — one quirk to know is that the installable CLI binary is called mirai, not uzu, so you run it with cargo run --release -p cli and preselect a model with --model trymirai/Qwen3.5-4B-M.

How Do You Serve Models With Uzu's Local API?

You serve models with Uzu by starting its built-in server, which speaks the OpenAI chat-completions format on your own machine. The command is cargo run --release -p cli -- server --model trymirai/Qwen3.5-4B-M, and the server listens on 127.0.0.1:8000 by default, with host and port overrides available. The two endpoints you need are POST /v1/chat/completions, which supports streaming when you set "stream": true and honors temperature, top_p, top_k, and max_tokens, and GET /v1/models for listing what is loaded.

The reason a local OpenAI-compatible API is useful is that almost every AI tool already speaks that format. You can point your coding agent, your test harness, or your own scripts at http://127.0.0.1:8000 instead of a cloud endpoint and keep every token on your Mac. Developers building agent workflows, like the ones in our Claude Code beginner guide, use exactly this pattern to swap cloud inference for local inference without rewriting their tooling. Because the traffic never leaves localhost, your prompts, documents, and data stay private by construction — the "full data privacy" half of the project's tagline.

Uzu vs MLX: Which Runs Faster on Apple Silicon?

On Apple Silicon, Uzu's speed story is speculative decoding: it runs speculative budgets of 16 to 32 tokens ahead of the main model and verifies them in one pass on kernels written for the M5's GPU neural accelerators, where older speculative approaches draft only a few. This Uzu Apple Silicon specialization is the reason its M5 numbers pull so far ahead of general-purpose builds. Mirai announced this speculative-decoding release for the Qwen3.6 27B model on Apple M5-series chips, and published its own comparison figures: against MTPLX (MLX plus speculative decoding) it claims almost 2x, and against llama.cpp over 3x at comparable quantization levels. Those are the maker's own claims, and you should treat them as the vendor's numbers, not independent measurements.

Independent measurements exist too. In October 2026, analyst Vidit Shrimali benchmarked Uzu against llama.cpp and MLX on the same base M5, the same model, and the same coding prompt: Uzu wrote 92 to 117 tokens per second, while llama.cpp and MLX sat at 22 and 25 tokens per second respectively. Mirai's published figures for the larger Qwen3.6 27B Mirai-M checkpoint (15.6 GB, measured on an M5 Max with 128 GB of memory) put Uzu at 35.54 tokens per second without speculation and 114 tokens per second with it — a 3.2-times gain from the speculative path alone. Shrimali's own caveat is worth repeating: "Acceptance changes with the workload. Hardware, quantization quality, context length, and thermal state can also alter the result." Your numbers will differ with your chip, your model, and your prompt.

Uzu vs llama.cpp: What Actually Changes?

Against llama.cpp, what changes is not just speed but the programming model. llama.cpp is a C++ inference server you talk to over HTTP or through bindings; Uzu is an engine library you embed directly in your app, with the model lifecycle — download, configuration, memory management — handled for you. Uzu's first-class features are the ones app builders reach for: tool calling, JSON-schema structured output, streaming replies, and shared chat instances. If your goal is to drop a fast local model inside an iOS or macOS app, Uzu's Swift and Python bindings and its unified-memory design are the reason to look at it instead of wrapping a server.

The honest counterweight is maturity. llama.cpp supports far more hardware and model formats and has years of community tooling around it; Uzu supports only Apple Silicon devices today and uses its own model format, with a companion converter called lalamo for optimizing and converting models for on-device use. Uzu is also young — version 0.6.15 with rapid releases — so expect APIs and install commands to move. Choose Uzu when you are building on Apple hardware and want maximum on-device speed with a clean embedding API; stay with llama.cpp when you need broad hardware coverage or the widest model compatibility.

What Should You Know Before You Install Uzu?

Before you install Uzu, know its four real limits. First, it is early software: 0.6.15 landed on October 30, 2026, and at this pace the install commands and APIs can change between versions — pin the version and re-check the README. Second, speculative decoding is chip-gated: the fast path is built for Apple M5-series chips, and a user report on the project's own issue tracker shows an M5 Ultra getting no speculative gain because the supported-chips list did not include it yet. Third, only Apple Silicon (iOS and macOS) is supported right now, with a CPU fallback — there is no Windows or Linux GPU path. Fourth, Uzu speaks its own model format, so you work from Mirai's published checkpoints or convert your own with lalamo.

Set your environment up for fast iteration too. If you are containerizing your dev setup, our Docker beginner guide covers the workflow, though note that Uzu's Metal acceleration needs the host Mac's GPU — containers on macOS do not pass the GPU through, so run the engine natively and keep the container for everything around it. And keep your expectations honest about the numbers in this guide: the 92–117 tokens-per-second figures are one analyst's measurements on one machine, and the 114 tokens-per-second figure is the maker's own. Measure on your own chip before you quote a number to anyone.

Sources

Every fact in this guide was verified against its primary source on November 3, 2026: the trymirai/uzu GitHub README (install commands, tagline, license, CLI and server usage, tool calling); the GitHub Releases API for trymirai/uzu (version 0.6.15, published 2026-10-30); the PyPI page for uzu 0.6.15 (Python install method); the Mirai docs FAQ (MIT license, device support, free-use statement); the twiscan mirror of Mirai's speculative-decoding announcement on X (speculative budgets, M5 chips, MTPLX/llama.cpp comparisons); Vidit Shrimali's Medium analysis "Stop Waiting on Local LLMs: How Uzu Supercharges Inference Speed" (October 11, 2026; independent 92–117 tokens-per-second benchmarks and Mirai's Qwen3.6 27B figures); the Towards AI Rust-trending roundup (3,158 stars, +573 per week as of November 1, 2026); the dailydoseofds coverage of the same benchmarks; and GitHub issue #964 on trymirai/uzu (speculation chip-gating on M5 Ultra). The star count of 3,208 was read live from the repository page the same day.

Frequently Asked Questions

Is Uzu open source?

Yes. The project's README states plainly, "This project is licensed under the MIT License," and the docs FAQ confirms the uzu library is fully open source under MIT. You can use it in commercial apps under those terms.

Is Uzu free to use?

Yes. The Mirai docs FAQ asks "Is it free to use?" and answers that it is completely free to use. There is no paid tier for the engine itself at the time of writing.

Which devices does Uzu support?

Apple Silicon devices only — iOS and macOS — through Metal and unified memory, with a CPU backend as a fallback. There is no Windows, Linux GPU, or Android support in the current version.

How does Uzu compare to llama.cpp and MLX?

On a base M5 with the same model and prompt, an independent October 2026 benchmark measured Uzu at 92–117 tokens per second against 22 for llama.cpp and 25 for MLX. Mirai's own figures for Qwen3.6 27B show 114 tokens per second with speculative decoding versus 35.54 without. Beyond speed, Uzu embeds as a library with built-in tool calling and structured output, while llama.cpp covers far more hardware and model formats.

Which models work with Uzu?

Uzu uses its own model format, with published checkpoints listed on Mirai's site and named in a tiered scheme — for example, alibaba:qwen3.5:0.8b:mirai:mirai-m:4, where the mirai-m tier means the 4-bit medium quantization. You can convert other models for on-device use with the companion tool lalamo.