How to Use vLLM: A Beginner's Guide to Serving LLMs at Speed

CodexFuse Team · 2026-10-31T03:57:27.257Z · 13 min read min read

vLLM turns your GPU into an OpenAI-compatible LLM server with one command. Install, serve, call the API, tune sampling, run Docker, and size your GPU memory — the complete beginner guide.

vLLM turns your GPU machine into an OpenAI-compatible LLM server with one command. This guide shows how to use vLLM from zero: install it, serve your first model, call the API, tune sampling, run it in Docker, and size your GPU memory so nothing crashes at load time.

What is vLLM, and how do you use it?

vLLM is an open-source inference and serving engine for large language models, built by UC Berkeley's Sky Computing Lab. It uses two core techniques to serve models faster than the standard approach: PagedAttention, which manages the GPU's KV cache in small blocks instead of one long contiguous reservation, and continuous batching, which rebuilds the request batch at every step so finished requests leave and waiting ones join immediately. The project reports up to 24x higher throughput than traditional serving methods. Most importantly for beginners, vLLM speaks the OpenAI API protocol: anything that talks to OpenAI's API can talk to vLLM by just changing the base URL.

How to use vLLM in five steps: install it in a fresh Python environment on a Linux machine with an NVIDIA GPU; run vllm serve with a model name from Hugging Face; confirm the server is up at http://localhost:8000; send your first request to /v1/chat/completions; then tune sampling parameters and memory flags to fit your GPU.

1. Install vLLM with pip in a fresh Python environment on a GPU server.
2. Serve a model with vllm serve <model-id> — vLLM downloads the weights from Hugging Face automatically.
3. Call the OpenAI-compatible endpoint at http://localhost:8000/v1/chat/completions.
4. Tune sampling with temperature, top_p, and max_tokens, then profile GPU memory.
5. Deploy with the official Docker image when you move to production.

Before you run your first vllm serve, you can sanity-check the command you are planning with the vLLM config checker — it parses the model name and flags, scores readiness, and catches common mistakes like a port that is already taken.

What hardware and software do you need to run vLLM?

vLLM needs a real NVIDIA GPU for production serving — the CPU path exists for development and testing only, never for production throughput. The official requirements list Linux as the operating system, Python 3.8 through 3.12, and a GPU with compute capability 7.0 or higher (V100, T4, RTX 20xx, A100, L4, H100, and newer cards all qualify).

VRAM is the hard ceiling that decides everything else. As a practical rule, a model in FP16 or BF16 precision needs about 2 bytes per parameter just for the weights — a 7-billion-parameter model takes roughly 14 GB before the KV cache is even allocated. A 2026 vLLM setup guide puts the practical minimum at 16 GB of VRAM for models in the 1 to 8 billion parameter range. The detailed sizing table later in this guide works the numbers out for common model sizes.

How do you install vLLM?

Install vLLM in a fresh Python environment — the project recommends against reusing an existing PyTorch installation, because vLLM's compiled CUDA kernels are tied to specific PyTorch and CUDA builds and a stale environment produces binary-incompatibility errors. A fresh virtual environment avoids the whole class of problems.

# 1. Fresh Python environment
python3 -m venv vllm-env
source vllm-env/bin/activate

# 2. Install vLLM (prebuilt wheels; the current release is v0.32.0)
pip install vllm

# 3. Verify the install and the GPU
nvidia-smi
python -c "import vllm; print(vllm.__version__)"

If nvidia-smi is not found or shows no GPU, stop here and fix the NVIDIA driver installation before continuing — vLLM cannot see a GPU the driver does not expose. You can also install with uv pip install vllm --torch-backend=auto, which the vLLM documentation recommends for faster, more reliable dependency resolution.

How do you serve your first model with vLLM?

The vllm serve command is the current standard entrypoint — the older python -m vllm.entrypoints.openai.api_server form is deprecated and prints a warning, so use vllm serve in everything you write from now on. One command starts the OpenAI-compatible server, with vLLM downloading the model weights from Hugging Face automatically on first run.

# Serve a small open model (downloads weights automatically)
vllm serve Qwen/Qwen3-0.6B

# Serve on a custom port
vllm serve Qwen/Qwen3-0.6B --port 8100

# Serve over a Unix domain socket
vllm serve Qwen/Qwen3-0.6B --uds /tmp/vllm.sock

By default the server starts at http://localhost:8000. Models behind a gated Hugging Face agreement (such as Meta's Llama family) need an access token: export HUGGING_FACE_HUB_TOKEN before serving, and pass --trust-remote-code only for models whose custom code you trust. Expect the first startup to take a few minutes — vLLM profiles the GPU and warms up before it reports ready.

How do you call the vLLM API from your app?

Since vLLM implements the OpenAI API protocol, any OpenAI client works as a drop-in replacement — just point it at your server. A plain curl request to /v1/chat/completions is the quickest proof the whole chain works end to end.

# 1. Check the server sees the model
curl http://localhost:8000/v1/models

# 2. Send your first chat completion
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-0.6B",
    "messages": [
      {"role": "user", "content": "Explain PagedAttention in two sentences."}
    ],
    "temperature": 0.7,
    "max_tokens": 200
  }'

In Python, swap the base URL and everything else stays identical: initialize the OpenAI client with base_url="http://localhost:8000/v1" and keep using chat.completions.create exactly as you would against OpenAI. This is why teams call vLLM a drop-in replacement — the application code never changes, only the endpoint does.

How do you tune vLLM's sampling parameters?

Sampling parameters decide how the model picks its next token, and they are the same knobs whether you pass them through the API or the offline Python API. The three you will touch first are temperature (how creative the picks are), top_p (the nucleus of top candidates to sample from), and max_tokens (how long the answer may run).

from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen3-0.6B")
params = SamplingParams(
    temperature=0.8,   # lower = deterministic, higher = creative
    top_p=0.95,        # sample from the top 95% probability mass
    max_tokens=200,    # cap the answer length
    repetition_penalty=1.05,  # discourage looping on the same phrase
)
outputs = llm.generate(
    ["Explain PagedAttention in two sentences."], params)
print(outputs[0].outputs[0].text)

Start from temperature 0.7–0.8 and top_p 0.95 for chat-style tasks, and drop temperature toward 0.1–0.2 when you need deterministic, factual answers. A repetition_penalty just above 1.0 (1.05–1.1) quietly kills the looping behavior small models sometimes show. Keep max_tokens tight — every extra token costs GPU time, and long answers on a small model are where quality visibly degrades.

How do you run vLLM in Docker with a GPU?

The official vllm/vllm-openai image on Docker Hub is the most reliable way to run vLLM in production — it bundles the right CUDA build and avoids the host's driver-version headaches entirely. The run command needs the NVIDIA runtime, the GPU flag, a mounted Hugging Face cache, and --ipc=host for the shared memory vLLM uses under the hood.

docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HUGGING_FACE_HUB_TOKEN=<secret>" \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:v0.32.0 \
    --model Qwen/Qwen3-0.6B

Pin the image tag to a release version instead of latest so upgrades are deliberate — latest can move under you mid-deployment. The mounted cache volume means the model downloads once and persists across container restarts, and the HF token goes in an environment variable, never in the image. The Docker beginner guide covers volumes, port mapping, and the NVIDIA container runtime in more depth if you are new to containers.

How much GPU memory does vLLM need?

The GPU-memory rule is simple: weights first, then KV cache. Weights cost about 2 bytes per parameter in FP16/BF16, and the KV cache — which holds the attention state for every in-flight request — grows with context length, batch size, and max sequence length. vLLM profiles the GPU at startup and sizes the cache from the memory it can see, and the --gpu-memory-utilization flag controls the fraction it is allowed to take.

This table is computed from the 2-bytes-per-parameter rule with headroom for the KV cache — treat it as a floor, not a guarantee, because long contexts and big batches eat the cache fast. When you share a GPU with other processes, lower --gpu-memory-utilization so vLLM leaves them room. If a model fails to load with an out-of-memory error, shrink --max-model-len or --max-num-seqs before you blame the GPU — those two flags are the biggest levers on cache size. All of vLLM's engine flags accept human-readable suffixes, so --kv-cache-memory-bytes 4G and --max-model-len 32K both work directly. Many beginners run their first GPU server on a rented VPS hosting plan — check the VRAM tier before you pick one.

vLLM vs Ollama: which should you use?

vLLM and Ollama serve the same models but optimize for opposite jobs. Ollama is the local runner: it shines on a laptop or desktop, installs in minutes, and is the fastest way to chat with a model on your own machine. vLLM is the serving engine: it shines on a GPU server, where its continuous batching and PagedAttention push many concurrent requests through at the lowest per-request cost.

The decision rule: if one person runs one model on one machine, start with Ollama. If many users or services hit the same model and you care about throughput, cost per request, or an OpenAI-compatible endpoint, run vLLM. Teams often use both — Ollama for local development, vLLM behind the production API.

How do you monitor and benchmark a vLLM server?

vLLM ships its own benchmarking and testing tools, so you never have to guess at performance. The vllm bench serve command load-tests a running server, and vllm chat gives you a terminal chat with optional timing statistics.

# Quick interactive chat with timing stats
vllm chat --stats

# Load-test the running server (install bench extras first)
pip install "vllm[bench]"
vllm bench serve \
    --model Qwen/Qwen3-0.6B \
    --backend openai-chat \
    --num-prompts 50

For production traffic, enable --enable-prefix-caching to reuse KV cache for repeated prompt prefixes (system prompts shared across requests are the classic win), and cap --max-num-seqs so one traffic spike cannot queue the server into timeout territory. Watch the server's streaming token rate during the bench run — that single number is the fairest comparison between your configurations.

Common beginner mistakes with vLLM

Frequently asked questions

Can vLLM run on a CPU or Mac?

There is a CPU Docker image for development, but production serving needs a real NVIDIA GPU — vLLM's speed comes from GPU kernels, and the CPU path cannot deliver the throughput. For Apple Silicon or plain laptops, Ollama is the practical choice; save vLLM for a Linux GPU server or a GPU-backed cloud instance.

How is vLLM different from the Hugging Face Transformers library?

Transformers loads a model and generates text one request at a time in your Python process. vLLM is a serving engine: it wraps the model in a persistent OpenAI-compatible server, batches concurrent requests continuously, and manages GPU memory with PagedAttention so dozens of users share one GPU efficiently.

Does vLLM work with LangChain and other frameworks?

Yes — point LangChain at the server's OpenAI-compatible endpoint the same way you would point it at OpenAI, just with a different base URL. This drop-in compatibility is why vLLM fits existing RAG and agent stacks without code changes.

Which models work with vLLM?

vLLM supports the popular open-weight families — Llama, Qwen, Mistral, Gemma, Phi, and vision-language models — on the Hugging Face Hub. Gated models need a HUGGING_FACE_HUB_TOKEN with access approval. Always run vllm serve with the model's exact Hub ID (for example Qwen/Qwen3-0.6B), not a local folder path, unless you have downloaded the weights yourself. Treat the Hugging Face token like a password — the token-secrecy habits in the GitHub beginner guide apply to HF tokens in exactly the same way.

How do I update vLLM to a new version?

In a virtual environment, re-run pip install --upgrade vllm in the same fresh environment, or pull the new vllm/vllm-openai image tag and re-run the container with your pinned version tag. Check the release notes first — flag names occasionally change between versions, and the current release is v0.32.0.

Sources

• vLLM project — CLI Guide (docs.vllm.ai, crawled 2026-10-31): vllm serve as the OpenAI-compatible server command; subcommands chat, complete, serve, launch, bench, collect-env, download-kernels, run-batch, preload; --port, --uds, --help=all; human-readable integer suffixes.
• vLLM project — Installation docs (docs.vllm.ai, crawled 2026-10-31): Linux OS, Python 3.8–3.12, GPU compute capability 7.0+; fresh conda/venv recommended; pip install vllm; prebuilt wheels compiled with CUDA 12.1/12.4.
• vLLM project — Using Docker docs (docs.vllm.ai, crawled 2026-10-31): official vllm/vllm-openai image; docker run with --runtime nvidia --gpus all, HF cache volume, HUGGING_FACE_HUB_TOKEN, -p 8000:8000, --ipc=host.
• vLLM project — Quickstart (docs.vllm.ai): from vllm import LLM, SamplingParams; temperature=0.8, top_p=0.95; offline batched inference; OpenAI-compatible server at localhost:8000.
• vllm-project/vllm GitHub releases API (2026-10-31): latest release v0.32.0, published 2026-10-18.
• Glukhov.org — vLLM Quickstart: High-Performance LLM Serving in 2026 (2026-10): api_server entrypoint deprecated in favor of vllm serve; prebuilt wheels target CUDA 12.9.
• dev.to — What Is vLLM: Fast LLM Inference Engine Explained (87 days old): UC Berkeley Sky Computing Lab; PagedAttention + continuous batching; up to 24x throughput claim; OpenAI-compatible API drop-in replacement.
• Medium — Serving an OCR Model in Production with vLLM (2 days old): vLLM 0.25.1 production run on a single 48GB GPU; --enable-prefix-caching and --max-num-seqs; vllm bench serve --backend openai-chat.
• Medium — Getting Started with VLLM: Installation, Setup and Inference (Dinesh R): pip install vllm; nvidia-smi verification; fresh venv; air-gapped install path.
• tech-insider.org — How to Set Up vLLM: 12 Steps, 90 Min (2026): 16GB practical minimum VRAM for 1–8B models; uv install recommended; --ipc=host for tensor parallel setups.