How to Use LocalAI: The Beginner's Guide (First Docker Run, the Volume Rule, the Gallery, and the API-Key Lockdown)

CodexFuse Team · 2026-10-31T10:36:44.398Z · 13 min read

The complete beginner guide to LocalAI: first Docker run, the four-volume persistence rule, installing models from the gallery, your first /v1/chat/completions call, and the API-key lockdown.

To learn how to use LocalAI, think of it as the one piece between you and a private ChatGPT: it is a self-hosted server that speaks the same OpenAI API language, so any tool that talks to OpenAI can be pointed at your own machine instead. This beginner guide takes you from zero to a running LocalAI server: your first Docker run, the volume rule that stops your models from evaporating, installing a model from the gallery, your first real API call, and the API-key lockdown you must set before anything touches the internet. The guide assumes you can run containers — the Docker beginner guide gets you that far — and every fact below was checked against LocalAI's official docs and release notes on October 31, 2026.

What is LocalAI?

LocalAI is a free, open-source, self-hosted inference server that acts as a drop-in replacement for the OpenAI REST API, letting you run LLMs, generate images, and produce audio locally or on-premises with consumer-grade hardware, supporting multiple model families and architectures (LocalAI official quickstart, crawled October 31, 2026). It exposes the same endpoints OpenAI clients already speak — /v1/chat/completions, /v1/models, embeddings, image generation, and audio endpoints — so existing tools built for OpenAI can be repointed to your own hardware without changing their logic (dev.to quickstart, crawled October 29, 2026).

What sets LocalAI apart from its siblings is that it is a server first, not a desktop app: it ships a built-in web interface for chatting with models, a model gallery for one-click installs, a CLI for scripting, built-in AI agents with Model Context Protocol tool support, and even distributed multi-node scaling (official quickstart). If Ollama is the terminal app on your laptop, LocalAI is the API box you aim your whole stack at.

What do I need before running LocalAI?

You need a machine that stays on and a working Docker setup. Every command below assumes Docker Engine is installed and your user can run docker without sudo — the Docker beginner guide covers that setup — and if you want LocalAI on real server hardware instead of a home machine, the what is VPS hosting guide explains the box most people deploy it on. A laptop, a spare Linux computer, or a VPS all work for CPU inference; GPU users need an NVIDIA card with the NVIDIA Container Toolkit (or AMD/Intel with the matching runtime).

The first decision is CPU vs GPU, and LocalAI publishes image tags for each: localai/localai:latest is the CPU image, while GPU builds are latest-gpu-nvidia-cuda-12, latest-gpu-nvidia-cuda-13, latest-gpu-hipblas (AMD ROCm), latest-gpu-intel, and latest-gpu-vulkan (official containers docs, crawled October 31, 2026). Vulkan works across vendors and is the fallback when no matching CUDA, ROCm, or Intel image fits your card. CPU works fine for testing the whole arc — a small model answers in seconds — but anything interactive needs a GPU.

Plan your disk: models are gigabytes, not megabytes, and you want the model folder on fast local disk. Check you have tens of gigabytes free with df -h before downloading anything (official containers docs).

How do I run LocalAI in Docker for the first time?

Run it once the plain way first, because this is the command every guide builds on — and it is the command that teaches you the volume lesson:

docker run -p 8080:8080 --name local-ai -ti localai/localai:latest

This starts LocalAI with the API and web interface on port 8080, but it does not install any models — standard images ship with no pre-configured models, so you configure them yourself (official containers docs). Open http://localhost:8080 and you will see the web interface ready, with no models to chat with yet.

Confirm the server is alive before doing anything else:

curl http://localhost:8080/readyz

If that returns an OK, the API is up. This is also the exact probe the official Docker Compose healthcheck uses — ["CMD", "curl", "-f", "http://localhost:8080/readyz"] — so it is the canonical way to check (official containers docs). Note the tag: v4.12.1 was released on October 23, 2026 as a security-fix patch release (GitHub mudler/LocalAI releases API), and you can pin to it with localai/localai:v4.12.1 instead of latest if you want reproducible deploys.

The volume rule: which folders must I persist?

Here is the rule that saves every beginner exactly once: container files that are not stored in a volume are lost when the container is recreated during an image upgrade — mount the paths you want to keep. LocalAI exposes four of them, and the official docs say to mount all four if you want to preserve installed models, backends, settings, and application data:

The beginner-friendly form is bind mounts under one working directory — the container paths must be exactly /models, /backends, /configuration, and /data (official containers docs):

docker run -ti --name local-ai -p 8080:8080 \
  -v $PWD/models:/models \
  -v $PWD/backends:/backends \
  -v $PWD/configuration:/configuration \
  -v $PWD/data:/data \
  localai/localai:latest

Prefer named volumes over bind mounts if you are on Windows or macOS, or if /backends lands on a filesystem that cannot create symlinks (backend OCI images contain symlinks, and some CIFS/SMB mounts silently materialize them as regular files, wasting disk — official containers docs). The pattern is the same: docker volume create localai-models (and the other three), then -v localai-models:/models, and so on.

One precedence rule to memorize: every CLI flag is also settable as an environment variable, and environment variables take precedence over flags (dev.to quickstart, crawled October 29, 2026). When two settings disagree, the environment variable wins — so treat env as the authoritative place for configuration.

How do I install a model from the gallery?

You install models two ways, and the web interface is the beginner path. Open http://localhost:8080, go to Models → Explore, search for qwen3-4b, and click Install on its entry. qwen3-4b is the gallery's small, CPU-friendly Qwen3 model that also supports tool calling, so it doubles as the testbed for the agents page later (official quickstart). Wait for the download to finish, then open the Chat page, pick qwen3-4b from the model dropdown, and send a message.

The CLI route installs the same model without the browser:

local-ai models install qwen3-4b
local-ai models list   # list what the gallery offers

Or preload a model at startup — when starting LocalAI you can pass models to install automatically before the API comes up: local-ai run qwen3-4b (official quickstart). This also accepts gallery names, HuggingFace URIs, and Ollama-model references, so a URL or huggingface://… path is a valid install source.

Two details that matter later: gallery installs automatically detect your GPU (NVIDIA, AMD, Intel) and download the appropriate backend — you do not hand-pick binaries (official quickstart). And the repo's own docker-compose.yaml installs phi-2-chat from the gallery by default, so if you boot from Compose and get a model you did not pick, that is why (official containers docs).

How do I call the LocalAI API?

Check what is loaded first: curl http://localhost:8080/v1/models lists the installed models (official containers docs). Then send a real chat request exactly the way you would against OpenAI:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-4b",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Point any OpenAI SDK at http://localhost:8080 and change nothing else — the same URL scheme powers the rest of the surface: embeddings, image generation, audio/TTS, and transcription endpoints (official quickstart; glukhov.org FAQ, crawled October 29, 2026). LocalAI also speaks the Anthropic Messages API and the Open Responses API, so clients written against either of those work too (official quickstart).

This drop-in shape is why the whole self-hosted AI stack plugs in here: the AnythingLLM beginner guide shows a RAG app that can use LocalAI as its provider, and the LangChain beginner guide covers the code layer that aims OpenAI-shaped calls at whatever endpoint you give it. Set the endpoint to LocalAI, and nothing else in your code changes.

How do I lock LocalAI down with an API key?

LocalAI runs fully open on localhost by default — fine on your own laptop, dangerous on any machine someone else can reach. If you are exposing LocalAI remotely, the official docs give you two protection options (official quickstart, crawled October 31, 2026):

-e LOCALAI_API_KEY=your-strong-key-here

The simple form is LOCALAI_API_KEY: it gates access with a single key. Know the catch — API keys grant full admin access with no role separation, so whoever holds the key holds the server (official quickstart). The fuller form is LOCALAI_AUTH=true, which switches on multi-user support with admin/user roles, OAuth login, per-user API keys, and usage tracking.

The operational rules: never bind :8080 to a public IP without a key set — the web UI's import flow is powerful and unauthenticated (queue research, October 29, 2026). For remote production use, put the API behind HTTPS with a reverse proxy (the Vultr deploy guide walks the Traefik route) and add the key on top. Since keys grant full admin, treat the key like a root password: store it in a password manager and rotate it if it ever appears in a log, screenshot, or chat history.

The whole beginner arc in one numbered list:

1. Run the container with all four volumes mounted (the volume rule).
2. Hit /readyz to prove the API is up.
3. Install qwen3-4b from Models → Explore (or local-ai models install).
4. Chat with it in the web UI to prove inference works.
5. Call /v1/chat/completions with curl to prove the API is OpenAI-shaped.
6. Set LOCALAI_API_KEY before the machine touches any untrusted network.
7. Pin the image tag (e.g. v4.12.1) once you are happy, so upgrades are deliberate.

LocalAI vs Ollama: which should I run?

Run LocalAI when you want an OpenAI-compatible server; run Ollama when you want the terminal-first app on your own machine. The comparison below draws the line — the Ollama beginner guide covers the desktop lane in full:

The practical rule: if every developer on the team will call the same endpoint from their own code, LocalAI is the box in the middle; if you just want to chat with a local model from your own terminal, Ollama is the shorter path. Both are open source, both are free, and they are siblings — LocalAI even accepts ollama://model:tag references when installing models (official quickstart).

What are the beginner mistakes with LocalAI?

Sources

LocalAI official quickstart (localai.io) — crawled October 31, 2026: drop-in OpenAI REST API definition; Web UI; Models → Explore qwen3-4b install; local-ai run / local-ai models install; /v1/chat/completions curl; LOCALAI_API_KEY and LOCALAI_AUTH; MCP agents; Anthropic/Open Responses APIs; built-in images/agents/audio. LocalAI official containers docs (localai.io) — crawled October 31, 2026: CPU/GPU image tags; /readyz healthcheck; four persistent volumes with CLI flags and env vars; the lost-on-recreate warning; named volumes; backend symlinks on CIFS/SMB. GitHub mudler/LocalAI releases API — October 31, 2026: v4.12.1 released October 23, 2026 (security fix: hardened external URL policy). Dev.to LocalAI quickstart (rosgluk) — crawled October 29, 2026: env vars take precedence over CLI flags; /readyz; gallery install via Web UI or CLI. Explore.n1n.ai LocalAI quickstart — crawled October 31, 2026: latest-aio-cpu all-in-one image with pre-mapped OpenAI model names; $PWD/models volume; LOCALAI_* env table. glukhov.org LocalAI FAQ — crawled October 29, 2026: OpenAI-compatible endpoints for chat/embeddings/images/TTS/transcription; gallery install via Web UI or local-ai models install; secure-with-API-key lesson. Docker docs "Local Models" — updated October 27, 2026: docker run -p 8080:8080 -v ./models:/models localai/localai:latest-cpu. Vultr "How to Deploy LocalAI" — crawled October 31, 2026: directory structure + .env + Docker Compose with Traefik HTTPS.

Frequently asked questions about using LocalAI

Is LocalAI free?

Yes. LocalAI is free and open source (official quickstart) — the only costs are your hardware, electricity, and the time you spend on setup. There is no paid tier gate on the features covered in this guide.

Can I use LocalAI with the OpenAI Python SDK?

Yes — that is the entire design. Point the SDK's base URL at http://localhost:8080 (and pass your key if you set LOCALAI_API_KEY) and your existing OpenAI code runs unchanged, because LocalAI mirrors the OpenAI endpoint shapes (official quickstart).

Why do my models disappear when I update the container?

Because you ran without volumes: container files not stored in a volume are lost when the container is recreated during an image upgrade (official containers docs). Mount /models, /backends, /configuration, and /data — all four — and your models, backends, settings, and app data survive upgrades.

How do I pick between the CPU and GPU images?

Use localai/localai:latest (or a pinned version like v4.12.1) for CPU; pick the GPU tag matching your card — latest-gpu-nvidia-cuda-12/-cuda-13 for NVIDIA (plus --gpus all), -hipblas for AMD, -intel for Intel, -vulkan as the cross-vendor fallback — and add the matching --device flags (official containers docs).

Is LocalAI safe to expose on the internet?

Not out of the box: it runs fully open on localhost by default and its API keys grant full admin access with no role separation (official quickstart). Before any remote exposure, set LOCALAI_API_KEY (or LOCALAI_AUTH=true for multi-user) and put the API behind HTTPS via a reverse proxy.