How to Use AnythingLLM: The Beginner's Guide (First Install, First Model, First Workspace, and the Private-Data Rule)

CodexFuse Team · 2026-10-31T04:08:41.015Z · 16 min read

Learn how to use AnythingLLM: install the desktop or Docker app, connect Ollama, create your first workspace, master the embed-vs-attach rule, and chat with your documents privately.

Learning how to use AnythingLLM is the fastest way to get a private AI assistant that can read your own documents. AnythingLLM is a free, open-source desktop and Docker app from Mintplex Labs that bundles document chat, retrieval-augmented generation (RAG), and AI agents into one interface — with more than 60,800 GitHub stars, it is the most-installed local AI workspace of its kind. This AnythingLLM tutorial walks the full arc: desktop-vs-Docker decision, first install, first model, first workspace, the embed-vs-attach private-data rule, cited answers, agents, and the API.

What Is AnythingLLM, and Why Would a Beginner Pick It?

AnythingLLM is an all-in-one, local-first AI application: you connect any large language model (local or cloud), point it at your files, and chat with your documents in a normal chat window. Everything runs on your own machine by default — documents, embeddings, the vector database, and the chat history never have to leave your hardware.

The reason beginners pick it over wiring their own RAG stack is that the whole pipeline ships in one download. Upload a PDF, and AnythingLLM extracts the text, splits it into chunks, converts the chunks to vectors with an embedding model, and stores them in its built-in LanceDB vector database — no separate servers, no code, no configuration files. When you ask a question, it embeds your query, finds the most relevant chunks by cosine similarity, and hands that context to the language model for a grounded answer.

It also plays well with the rest of your stack. AnythingLLM exposes a full REST API (workspace management, document upload, chat completions), so once your workspace answers well you can drive it from your own code — a REST API tester is handy for trying those endpoints before you write a line of integration code. The 30-plus supported providers range from OpenAI, Anthropic, and Google to local runners like Ollama, LM Studio, and the desktop app's own built-in engine.

How Do You Use AnythingLLM for the First Time?

Use this eight-step arc — it is the complete beginner path from download to your first cited answer, and it takes about ten minutes on a normal laptop.

1. Decide desktop or Docker. Desktop is the one-download path for a single user; Docker is the multi-user server path for teams. Pick desktop unless you need logins and roles.
2. Install it. The AnythingLLM install on desktop is one download: get the installer for macOS, Windows, or Linux (AppImage) from anythingllm.com and launch it. Docker: pull mintplexlabs/anythingllm:master and run it on port 3001 with a persistent storage volume.
3. Choose your first model. On first boot you can download a built-in model (Llama-3, Phi-3) that runs on your CPU/GPU with zero external setup, connect Ollama, or paste a cloud provider key.
4. Configure the embedder. A chat model writes answers; a separate embedding model turns your files into searchable vectors. Set the embedder too — most beginners forget this step and then wonder why documents will not index.
5. Create your first workspace. Click + New Workspace in the sidebar and name it after the project the documents belong to.
6. Upload documents and embed them. Drag PDFs or docs into the workspace sidebar and let AnythingLLM chunk and embed them into the workspace's vector store.
7. Ask your first question. Chat normally; AnythingLLM retrieves the relevant chunks and answers from them, with citations back to the source.
8. Tighten accuracy next. Switch the workspace to Query mode or turn on Accuracy Optimized search when answers need to be strictly source-bound.

Should You Install the Desktop App or Docker?

Install the desktop app if this is your first time. The AnythingLLM Docker image is the same product, but it exists for servers, teams, and production — multi-user logins, role-based permissions, and data connectors like GitHub, Confluence, YouTube transcripts, and website crawlers live on the Docker/Cloud side.

The minimum bar for Docker is modest — 2 GB of RAM and 10 GB of disk — but local models are what actually consume hardware, and that depends on model size, not on AnythingLLM itself. An 8-billion-parameter model wants roughly 8 GB of RAM; a 70B model wants 40 GB or more. If Docker is new to you, our Docker beginner guide covers the runtime basics before you deploy anything into it.

How Do You Connect AnythingLLM to Ollama?

Connect AnythingLLM to Ollama when you want 100% offline, private AI with the widest model choice. The AnythingLLM Ollama pairing is the most popular local setup, and it is five clicks once Ollama itself is running.

First, install and start Ollama, then pull two models — one chat model and one embedding model. They are separate settings in AnythingLLM, and the embedder is the one beginners forget:

ollama serve
ollama pull llama3.1:8b
ollama pull nomic-embed-text

Then open AnythingLLM, go to Settings (the gear icon), and set the LLM provider to Ollama. The URL you enter depends on where each piece runs — this is the single most common connection failure, so get it right the first time:

Pick your model from the dropdown and save. Then go to Settings > Embedder, select Ollama again, choose nomic-embed-text, and use the same URL. The embedding model is set system-wide, so choose carefully — changing embedders later forces a full re-embed of every document, which is slow on large libraries.

If even Ollama feels like too much setup, the desktop app's built-in engine is the true zero-config path: Settings > LLM Provider > AnythingLLM, pick a model to download, wait, and chat. No external services at all. For a deeper tour of local model management, our Ollama beginner guide walks through installation, model picking, and GPU configuration.

Which model size should a beginner pull?

Match the model to your RAM — a practical pairing as of mid-2026. The chat model does the reasoning; the embedding model (nomic-embed-text, about 274 MB, CPU-friendly) does the retrieval, and it barely touches GPU memory.

What Is an AnythingLLM Workspace?

A workspace is an isolated environment that keeps one project's documents, embeddings, and conversations separate from everything else. Each workspace has its own embedded documents, its own chat history, and — on Docker and Cloud — can even use a different language model, system prompt, and temperature than its neighbors.

Create one per project, client, or knowledge domain: click + New Workspace in the sidebar, name it, optionally add a description, and start uploading. A workspace for "Q3 contracts" and a workspace for "home renovation quotes" will never mix each other's documents, which is exactly the isolation you want when answers must come from the right pile of files.

On the Docker deployment, workspaces also carry the permission model: Admins get full access and user management, Managers can create workspaces and manage documents, and Default Users can only chat inside the workspaces assigned to them. One caution worth knowing up front — within a workspace, anyone who can see the workspace can see all of its embedded documents. There is no per-document permission sync, so keep sensitive piles in separate workspaces.

Should You Embed or Attach Your Documents?

Learn this choice before you upload anything — the AnythingLLM embed vs attach decision is the private-data rule of the product, and getting it backwards is the most common beginner regret. AnythingLLM gives you two ways to feed a document to the model, and they behave very differently.

Embedding is the RAG path: the document is split into chunks, converted to vectors, and stored in the workspace's vector database. When you ask a question, only the most relevant chunks are pulled in — and the embedded document becomes available in every thread inside that workspace. Attaching is the drag-and-drop path: you drop a file into the chat window (or click the + icon in the prompt input), AnythingLLM inserts the document's full text into that one conversation, and it stays limited to that single thread.

A good rule of thumb, borrowed from real-world usage: embed the documents you will need in every conversation (your CV, your standing contracts, the product manual), and attach the documents that belong to one specific thread (this week's offer letter, a single invoice). AnythingLLM also offers pinning — click the pin icon next to a document to inject its full text into every prompt, bypassing chunking entirely. Pinning is ideal for a short policy or spec you always want the model to read in full, as long as it fits the context window.

How Do You Get Accurate, Cited Answers Instead of Hallucinations?

Switch the workspace to Query mode when answers must come from your files and nowhere else. Every workspace has a chat-mode toggle under Workspace Settings → Chat Settings: Chat (the default) blends document context with the model's general knowledge and conversation history, while Query answers only from retrieved document chunks — no chat history, and it refuses rather than inventing when nothing relevant is found.

Then tune retrieval instead of blaming the model. Chunk size and overlap are set per workspace, and a solid starting point is roughly 500-token chunks with about 10% overlap, retrieving 5–7 chunks per query. AnythingLLM also ships an "Accuracy Optimized" search mode that pulls more candidate chunks and re-ranks them before answering — slightly slower, noticeably better in most cases. More vectors is not always better: extra chunks add noise, so raise the count only when answers feel thin.

For everyday document Q&A, the default LanceDB vector database needs zero configuration and scales to millions of vectors on disk — most users should never change it. The alternatives (Chroma, Pinecone, Milvus, Qdrant, Weaviate, PGVector, Zilliz, AstraDB) exist for specific infrastructure needs, and switching requires re-embedding everything, so treat the default as the default.

How Do AnythingLLM Agents Work?

Type @agent in any chat to hand the conversation to AnythingLLM's agent mode. The agent can call tools to do multi-step work: searching your embedded documents, summarizing them, browsing the web, scraping pages, saving files, listing documents, generating charts, and even querying databases through the SQL Connector.

RAG Search and Summarize Documents are always enabled; the rest are toggled under Settings > Agent Skills. Web browsing needs a search API key (SerpApi or SearchApi), and the SQL Connector needs its database configuration. As of 2026 the desktop app also speaks the Model Context Protocol — agents can load tools from MCP servers (file systems, web search, databases) without writing a custom skill, with MCP Tools supported.

Not every model makes a good agent. Use 8 billion parameters or more, prefer tool-calling models with native function calling, and 8-bit quantization tends to behave more reliably than 4-bit for agent work. If the agent ignores its tools, the usual culprits are a too-small model, disabled skills, or a missing API key — check those three before anything else.

Beyond chat agents, Agent Flows is the no-code visual builder: drag components onto a canvas, connect steps, configure triggers, and save automated workflows like scheduled web scraping or document-processing pipelines. Power users can also write custom agent skills in code, and the Community Hub (disabled by default for security) lets you browse shared ones.

How Do You Use the AnythingLLM API and Chat Widget?

Use the REST API when AnythingLLM's answers are good and you want them inside your own product. The developer API covers workspace management, document upload and embedding, chat completions, and user management, with interactive Swagger docs at /api/docs on your instance.

A chat call is one POST with a Bearer API key:

curl -X POST "http://localhost:3001/api/v1/workspace/my-workspace/chat" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"message": "What are the key findings?"}'

Keep that API key in the provider integration and your own secrets management — never in screenshots, shared snippets, or public code. For a website integration, the chat widget embeds with one script tag: set data-embed-id and data-base-api-url, point the script source at your instance's /embed/anythingllm-chat-widget.min.js, and you get a customizable support-chat bubble (position, assistant name and icon, window size, auto-open on load) backed by your workspace.

AnythingLLM vs Open WebUI: Which Should You Choose?

Choose AnythingLLM when documents are the job; choose Open WebUI when the chat interface itself is the job. They are sibling products in the local-AI space — not rivals — and many people run both: AnythingLLM as the document workspace, Open WebUI as the daily chat UI.

The center of gravity differs. AnythingLLM is the all-in-one RAG product: first-class document handling with citations, chunk controls, pinning, and Query mode, a zero-config desktop app for individuals, and Docker for small teams. Open WebUI is the extensible ChatGPT-style interface: server-first deployment, stronger enterprise authentication (OAuth2, LDAP, SSO), and hybrid search — it assumes you already have Ollama running and brings the polish.

If you want the full Open WebUI path instead, our Open WebUI self-host guide covers the Ollama-plus-Open-WebUI stack on a cheap VPS — same private-chat goal, different product. And if you would rather build AI apps than chat with documents, our Dify beginner guide covers the no-code builder lane: Dify builds the app, AnythingLLM is the ready-made app.

Beginner Mistakes to Avoid

Sources

Setup facts, provider URLs, and feature details verified against the following pages, all fetched during this article's research: the AnythingLLM setup guide on Local AI Master (installation options, Ollama URL table, embedder and vector-database notes, agent tools, pricing); TechRadar's AnythingLLM walkthrough (local-first approach, the embed-vs-attach distinction with worked examples); the official AnythingLLM documentation on using documents (attaching via drag-and-drop or the + icon, context-window overflow behavior, document pinning, workspace RAG settings); DataCamp's AnythingLLM guide (architecture: frontend, Express server, document collector; workspaces as isolated rooms; API at /api/docs); the Open WebUI vs AnythingLLM comparison (deployment forms, auth, permissions, connectors); and the AIKetra AnythingLLM review (Mintplex Labs background, 30+ providers, cloud pricing verified September 2026).

Frequently Asked Questions

Is AnythingLLM free?

Yes. The desktop app and the Docker deployment are free forever under the MIT license with full features. Mintplex Labs also sells managed cloud hosting — Starter at $50/month for up to 3 users and 100 documents, Pro at $99/month, and custom Enterprise — but self-hosting costs nothing beyond your own hardware and any cloud model API keys you choose to connect.

Can AnythingLLM run completely offline?

Yes, when you pair it with local providers. The desktop app's built-in engine downloads models like Llama-3 and Phi-3 and runs them on your CPU or GPU with no external services, and Ollama plus a local embedder like nomic-embed-text keeps the entire RAG pipeline — chunking, embeddings, vector storage, and chat — on your machine with zero internet needed.

What is the difference between embedding and attaching a document?

Embedding chunks a document into the workspace's vector database so it is searchable from every thread in that workspace — best for reference libraries. Attaching drops the document's full text into a single chat thread via drag-and-drop or the + icon — best for one-off reads. If an attached file exceeds the model's context window, AnythingLLM offers to embed it instead.

How do I connect AnythingLLM to Ollama?

In AnythingLLM go to Settings → LLM Provider, choose Ollama, and enter the right URL for your setup: http://127.0.0.1:11434 for the desktop app, http://host.docker.internal:11434 for Docker on Windows or macOS, or http://172.17.0.1:11434 for Docker on Linux. Pick your model, save, then repeat under Settings → Embedder with nomic-embed-text using the same URL.

AnythingLLM or Open WebUI — which should a beginner install first?

Install AnythingLLM first if your goal is chatting with documents: its RAG pipeline, workspaces, citations, and Query mode are built for that job with a zero-config desktop app. Install Open WebUI first if your goal is a polished daily chat interface with enterprise login options and you already run Ollama. They complement each other, and many self-hosters run both.