How to Use LitGPT: The Beginner's Guide (Install, First Chat, the Finetune Lesson, and Serving)

CodexFuse Team · 2026-11-06T06:56:50.698Z · 23 min read min read

How to Use LitGPT: The Beginner's Guide (Install, First Chat, the Finetune Lesson, and Serving)

If you want to learn how to use LitGPT, the short answer is that you install the Python package, download a small open model such as Phi-2, chat with it from your terminal, and then finetune it on your own data with one command when you are ready. LitGPT is Lightning AI's open-source toolkit for running, finetuning, pretraining, evaluating, and serving more than 20 open large language models on your own hardware. This litgpt tutorial walks the whole arc in order: install, first chat, one-shot generation, the LoRA finetune lesson, and serving the finished model as a web API.

One name to get straight before anything else: LiteLLM is a different product from LitGPT, and the two are easy to confuse because the names look alike. LiteLLM is BerriAI's open-source AI gateway and LLM proxy, a tool that routes your calls to other providers' hosted models through one OpenAI-compatible API. LitGPT is Lightning AI's toolkit for working with open models locally, and it never routes anything to a hosted provider.

This guide is about LitGPT only, and every command, version, and number below comes from LitGPT's own docs, checked on 6 November 2026.

Before you install anything from the internet, build one safety habit: if you download a tool, a script, or an installer from a third-party mirror, check the file first with a SHA-256 hash generator before you run it. And if you plan to do your finetuning on a machine you control rather than your laptop, a VPS hosting plan gives you a server of your own, which is the kind of setup that GPU work needs.

What Is LitGPT?

LitGPT is Lightning AI's open-source Python library and command-line tool for pretraining, finetuning, evaluating, chatting with, and serving more than 20 large language models, built from scratch with no abstraction layers. You give it a model name and an action, such as litgpt chat microsoft/phi-2, and it downloads the weights, loads the model, and runs it. The project began in May 2023, it is released under the Apache 2.0 licence, and its code lives at github.com/Lightning-AI/litgpt.

The mental model that makes LitGPT click is its order of operations: finetune first, serve second. You take a pretrained open model, teach it your data with a small finetuning run, test it in chat, and then serve the result as an API. That is the whole arc of this guide, and every section below follows it in the same order.

Two cautions belong up front because they shape how you should read the rest of this guide. First, the current release is 0.5.13, dated 29 June 2026, which is 130 days before this guide, while commits on the main branch continued into late October 2026, so the installed release can lag behind the docs. Second, the command names in the docs are in flux: the current README uses unified commands such as litgpt finetune, while the tutorials folder still documents older subcommands such as litgpt finetune_lora, so this guide uses the README's unified form throughout.

How to Use LitGPT in 5 Steps

If you only need the shape of the whole job, here it is in five steps. Each step has its own section below with the exact commands and the mistakes to avoid.

1. Install LitGPT with pip install 'litgpt[all]' on Python 3.10 or newer.
2. List the models with litgpt download list and download a small one, such as litgpt download microsoft/phi-2.
3. Chat with it in your terminal using litgpt chat microsoft/phi-2.
4. Finetune it on your own JSON data with litgpt finetune microsoft/phi-2 --data JSON --data.json_path my_custom_dataset.json --data.val_split_fraction 0.1 --out_dir out/custom-model.
5. Test the result with litgpt chat out/custom-model/final and serve it with litgpt serve out/custom-model/final.

What Do You Need Before You Start?

You need Python 3.10 or newer, pip, and a machine with a GPU if you plan to finetune. LitGPT's own finetuning docs say finetuning needs at least one GPU with about 24 GB of memory, such as an RTX 3090, for a 3-billion-parameter model with the default settings. That sounds like a lot, and it is, but the memory section below shows how the right settings cut that number to about 6 GB.

You also need disk space for model weights, which run to several gigabytes per model, and a Hugging Face account for gated models. Some models, such as Llama 2, require you to accept their licence terms on Hugging Face and pass an access token with --access_token before LitGPT can download them. Small open models such as Phi-2, SmolLM2, and TinyLlama need no token at all, which is why this guide starts with Phi-2.

One honest gap in the docs: CPU-only chat is not verified in any fetched LitGPT page. The repo has CPU tests and the README says it runs on low-memory GPUs, but that is not the same as a supported CPU-only chat. If you have no GPU, start with the install and the Python API sections, and test chat on a GPU machine before you promise yourself a CPU-only setup.

How Do You Install LitGPT?

A litgpt install takes one command for most beginners: pip install 'litgpt[all]'. The tutorials recommend this full install, which pulls in every dependency the finetuning, serving, and quantization workflows need.

The README's quick start shows a different form, pip install 'litgpt[extra]', and the two are not identical. The [all] form is the beginner-safe full install from the tutorials, while [extra] is the README's minimal form. Pick [all] unless you already know which extras you are skipping, because a missing extra shows up later as a confusing import error in the middle of a finetuning run.

If you prefer to install from source, clone the repo with git clone https://github.com/Lightning-AI/litgpt, move into it with cd litgpt, and install it in editable mode. The README gives two paths: uv sync --all-extras if you use uv, or pip install -e ".[extra,compiler,test]" if you use pip. You do not need both.

After installing, check that the command is on your path by running litgpt --help, which lists every workflow. If the shell cannot find litgpt, your pip install went to a different Python than the one on your path, and the fix is to install into the same environment you run commands from.

Which Model Should You Start With?

Start small. LitGPT supports more than 20 model families, from 135-million-parameter SmolLM2 up to 405-billion-parameter Llama 3.3, a size range of 3,000 times, and a beginner has no reason to start at the top of it. A small model downloads in minutes, chats on a single consumer GPU, and finetunes in the memory budget the docs actually measured.

Run litgpt download list to see every supported model, then download one small starter with litgpt download microsoft/phi-2. The docs' own default example model is Microsoft's Phi-2, a 2.7-billion-parameter model, and it is the right first pick for most people because every README example uses it.

The download converts the weights into LitGPT's own checkpoint format and stores them under ./checkpoints/microsoft/phi-2. You do not need to move them or convert them yourself. If you would rather try a different starter, the table below lists the small models the README's model tables verify, with their sizes and authors.

One warning about model names: the Hugging Face repo id is part of the command, so microsoft/phi-2 and meta-llama/Llama-3.2-3B-Instruct are different strings that point at different models. Copy the id exactly from litgpt download list or the README's model tables. A near-miss id either fails to download or, worse, downloads a different model than the one you meant.

How Do You Chat With a Model in LitGPT?

A litgpt chat session is the fastest way to see whether a model works. One command, litgpt chat microsoft/phi-2, opens an interactive prompt in your terminal, and the model answers line by line: type a prompt such as "What do Llamas eat?" and it replies.

Chat auto-downloads the weights if you have not downloaded them already, so the first run takes as long as the download. After that, the same command starts in seconds. If your prompt runs over several lines, add --multiline true so the chat does not send your message on the first Enter.

Chatting with your own finetuned model uses the same command with a checkpoint path instead of a model name: litgpt chat out/custom-model/final. The finetune section below writes its result to that path, and this is how you test it.

If a download asks for a Hugging Face access token, the model is gated: accept its licence terms on the Hugging Face model page first, then pass your token with --access_token. That token rule applies once per gated model, and it is the single most common reason a first chat fails. Small open models such as Phi-2 never ask for it.

How Do You Generate Text With One Command?

Chat is interactive, but sometimes you want one answer and then your terminal back. The generate command does that: litgpt generate stabilityai/stablelm-base-alpha-3b --prompt "Hello, my name is" takes a prompt, prints the completion, and exits. The same form tests a finetuned checkpoint: litgpt generate "out/lora/final" --prompt "Recommend a movie to watch on the weekend."

The second form is the useful one after finetuning, because it points at your own checkpoint directory instead of a base model. It is also the fastest way to compare a base model against your finetuned version: run the same prompt against both and read the difference.

If you would rather call the model from Python than from the shell, LitGPT has a small Python API. The README's quick start shows the whole thing in a few lines: from litgpt import LLM, then llm = LLM.load("microsoft/phi-2"), then text = llm.generate("Fix the spelling: Every fall, the family goes to the mountains."), then print(text). The same code works for any supported model name.

LLM.load() downloads the checkpoint from the Hugging Face Hub automatically if it is not already local, and you can pass a local path instead. Generation supports streaming with llm.generate("hi", stream=True), saving with llm.save("path/to/save/directory"), and multi-GPU distribution with llm.distribute(). This is the litgpt python api in its current form, but note that its own docs call it a work-in-progress draft that is experimental and subject to change, so expect the surface to move between releases.

How Do You Finetune a Model With LoRA in LitGPT?

The litgpt finetune lesson is the heart of this guide, because finetuning is the reason most people install LitGPT at all. Finetuning takes a pretrained model and trains it further on a smaller, specialized dataset, so it learns your task, your tone, or your domain. LitGPT's default method is LoRA, which freezes the big model and trains only a small set of adapter weights on top of it.

LoRA is the right default for a beginner because it trains far fewer numbers than full finetuning. The default settings use a rank of 8 and an alpha of 16, which gives a scaling factor of 16 divided by 8, or 2.0. A litgpt lora run freezes the big model and trains only a small set of adapter weights on top of it, and that is why it fits on a single GPU.

Your data comes first, before any training command. For a custom JSON dataset, LitGPT expects a JSON array of objects with an instruction field, an output field, and an optional input field. The README's worked example downloads a finance dataset with curl -L https://huggingface.co/datasets/ksaw008/finance_alpaca/resolve/main/finance_alpaca.json -o my_custom_dataset.json, and you can swap in your own file with the same shape.

With the dataset ready, one command runs the finetune: litgpt finetune microsoft/phi-2 --data JSON --data.json_path my_custom_dataset.json --data.val_split_fraction 0.1 --out_dir out/custom-model. It auto-downloads the base model weights, trains the LoRA adapter on your data, and writes checkpoints to the output directory. The --data.val_split_fraction 0.1 flag holds back 10 percent of your data for validation, so you can see whether the model is actually improving instead of just memorizing.

When training finishes, test the result in chat with litgpt chat out/custom-model/final, then serve it with litgpt serve out/custom-model/final. Those two commands are the same ones from the earlier sections, pointed at the new checkpoint.

If you need finer control than command-line flags, LitGPT takes a YAML recipe file instead. Recipes live in config_hub/finetune/<model>/lora.yaml in the repo, and you can pass one with --config and override any single key on the command line, such as --lora_r 4. The keys a beginner actually needs are the checkpoint and output paths, the precision, the quantization, the LoRA rank and alpha, the data settings, and the training settings.

The defaults are sensible: rank 8, alpha 16, dropout 0.05, learning rate 0.0003, and seed 1337.

• checkpoint_dir: the base model checkpoint path.
• out_dir: where checkpoints and logs go, such as out/finetune/qlora-llama2-7b.
• precision: bf16-true, bf16-mixed, or 32-true.
• quantize: bnb.nf4 for QLoRA, plus nf4-dq, fp4, fp4-dq, and int8-training.
• lora_r (default 8), lora_alpha (default 16), lora_dropout (default 0.05).
• data: the dataset class, prompt style, and val_split_fraction.
• train: batch sizes, epochs, max_seq_length, and learning_rate (default 0.0003).
• devices (default 1) and seed (default 1337).

Two practical notes close the lesson. First, LitGPT auto-merges the LoRA adapter for generate and chat, so you only need litgpt merge_lora "out/lora/step-002000" if you want to use the weights outside LitGPT. Second, older tutorials still show litgpt finetune_lora, litgpt finetune_full, and litgpt finetune_adapter as separate subcommands, while the current README uses the single unified litgpt finetune form shown above.

Use the unified form, and check litgpt finetune --help on your installed version if a flag behaves differently than this guide says.

How Much GPU Memory Does LitGPT Finetuning Need?

Finetuning memory is the question that decides whether your hardware can run the job at all, so here are the numbers LitGPT's own docs measured, not estimates. The docs finetuned StableLM 3B with LoRA on a single RTX 3090 with 24 GB of memory, and they measured the training memory for each precision and quantization setting. The pattern is the same for every model: full precision costs the most, and 4-bit quantization costs the least.

The numbers tell a clear story. Switching StableLM 3B from the default bf16-mixed to bf16-true plus 4-bit quantization cuts training memory from 26.92 GB to 6.35 GB, which is 20.57 GB saved, or 76.4 percent less, a 4.24 times reduction. The same trick on Llama 2 7B saves 7.16 GB, or 33.6 percent, taking it from 21.30 GB to 14.14 GB.

And the default setting runs out of memory entirely on the 7B model, which is why the docs' first recommendation is to change the precision before you buy a bigger GPU.

For plain inference, a 3B model needs about 7 GB of GPU memory in bfloat16, while a 70B model needs about 140 GB on a single device, which is 20 times as much. The docs say a 70B model needs at least four A100 40 GB GPUs, or two GPUs with 4-bit quantization, so do not plan a 70B chat on a single consumer card. As a sense check on the small end, Phi-2's 2.7 billion parameters weigh 5.4 GB in bfloat16, and the docs measured 5.92 GB of GPU memory running it through the Python API on an A10G, which is 0.52 GB of overhead, or 9.6 percent above the raw weights.

At the docs' measured speed of about 32 tokens per second on that card, 100 tokens take 3.1 seconds and 1,000 tokens take 31.2 seconds.

# LitGPT memory math, from the docs' measured numbers (GB)
# StableLM 3B LoRA: 26.92 -> 6.35 GB with bf16-true + bnb.nf4
print("saved GB:", round(26.92 - 6.35, 2))
print("percent less:", round((26.92 - 6.35) / 26.92 * 100, 1))
print("reduction factor:", round(26.92 / 6.35, 2))
# Llama 2 7B LoRA: 21.30 -> 14.14 GB with bf16-true + bnb.nf4
print("saved GB:", round(21.30 - 14.14, 2))
print("percent less:", round((21.30 - 14.14) / 21.30 * 100, 1))
# LoRA scaling: alpha / r
print("scaling r=8 alpha=16:", 16 / 8)
print("scaling r=4 alpha=16:", 16 / 4)
# Phi-2 weights at bf16: 2.7B params x 2 bytes
print("phi-2 weight GB:", 2.7e9 * 2 / 1e9)
print("measured 5.92 GB overhead pct:", round((5.92 - 5.4) / 5.4 * 100, 1))
# Inference: 70B vs 3B, and token timing at 32 tok/s
print("70B vs 3B inference GB:", 140 / 7)
print("100 tokens at 32 tok/s:", round(100 / 32, 1), "s")
print("1000 tokens at 32 tok/s:", round(1000 / 32, 1), "s")
# Running it prints:
# saved GB: 20.57
# percent less: 76.4
# reduction factor: 4.24
# saved GB: 7.16
# percent less: 33.6
# scaling r=8 alpha=16: 2.0
# scaling r=4 alpha=16: 4.0
# phi-2 weight GB: 5.4
# measured 5.92 GB overhead pct: 9.6
# 70B vs 3B inference GB: 20.0
# 100 tokens at 32 tok/s: 3.1 s
# 1000 tokens at 32 tok/s: 31.2 s

How Do You Serve a Finetuned Model With LitGPT?

Serving turns your model into a web API that other programs can call. One command starts the server: litgpt serve microsoft/phi-2 for an out-of-the-box model, or litgpt serve path/to/microsoft/phi-2/checkpoint for your own finetuned checkpoint.

The server listens on port 8000 by default and takes POST requests at http://127.0.0.1:8000/predict with a JSON body of {"prompt": "..."}, returning the completion in the "output" field. The README's example shows the call from Python with the requests library, as requests.post("http://127.0.0.1:8000/predict", json={"prompt": "Fix typos in the following sentence: Example input"}) followed by print(response.json()["output"]), and it is the same shape from any language that can send an HTTP POST.

Two flags change how the server behaves. Add --stream true to get tokens one by one instead of waiting for the whole answer, and add --openai_spec true to expose OpenAI-compatible endpoints at /v1/chat/completions instead of /predict, so existing OpenAI client code can point at your local model with only a URL change. Run litgpt serve --help for the full flag list on your installed version, because the exact flag names beyond these two were not individually verified for this guide.

If you prefer to serve from a container instead of a bare install, the Docker beginner guide covers the runtime LitGPT would run inside.

LitGPT vs vLLM vs Ollama: Which Tool Fits Which Job?

LitGPT is not the only tool that runs open models, and picking the wrong one wastes a weekend. The three tools beginners meet most are LitGPT, vLLM, and Ollama, and they fit different jobs.

LitGPT is the pretrain and finetune lane: you use it when you want to teach a model new behavior on your own data. vLLM is the inference-serving lane: you use it when you want fast, high-throughput serving of a model you already have. Ollama is the local-chat lane: you use it when you want the simplest possible way to chat with a model on your own machine.

That division is the draw-the-line rule for this guide: LitGPT finetunes, vLLM serves at scale, and Ollama chats locally. If your job is to finetune a model and then serve the result yourself, LitGPT alone covers the arc. If your job is only to serve a finished model to many users at once, the vLLM beginner guide is the better starting point.

If your job is only to chat with a model on your laptop with no training at all, the Ollama beginner guide is the simpler path.

None of the three is a rival to the others, because they answer different questions. A common real workflow uses two of them: finetune in LitGPT, then serve the merged checkpoint with vLLM when traffic grows. Start with the tool that matches the job you have today, and add the second one only when the first stops being enough.

What Should You Not Do With LitGPT?

Do not start with pretraining. Pretraining teaches a model from scratch on a large pile of text, and LitGPT's own docs recommend that most users skip it and use pretrained weights instead, because it needs substantial compute and time. Finetuning a small pretrained model teaches it your task for a fraction of the cost, and it is the right first project.

Do not copy commands from old blog posts without checking them against the current README. Lightning AI's own chatbot tutorial, for example, warns on the page itself that it is outdated with respect to the latest LitGPT version and tells readers to check the official repo for current code samples. The same warning applies to any third-party guide older than the 0.5.13 release: the unified litgpt finetune command replaced the older litgpt finetune_lora subcommand, and old commands can fail silently or train the wrong thing.

Do not assume the Python API is stable. Its own docs describe it as a draft that may change between releases, so code written against it today may need updating after the next release. The CLI commands are the stable surface for a beginner to learn first.

And do not expect a CodexFuse LitGPT tool: CodexFuse does not offer one, and this guide is about the open-source project itself.

Finally, do not finetune on data you do not have the right to use, and do not serve a finetuned model without checking what it says. A model trained on biased or private data will repeat both, and serving it makes that your problem. Test every finetune in chat first, read its answers on your own data, and only then put it behind an API.

Sources

Every fact in this guide comes from a page fetched on 6 November 2026. Prices, versions, and command names are dated to that day, and the LitGPT docs move fast enough that a re-check is worth doing before you run a command that spends GPU time.

• LitGPT GitHub README — github.com/Lightning-AI/litgpt (install, commands, model tables, finetune, serve, evaluate, pretrain)
• PyPI litgpt JSON API — pypi.org/pypi/litgpt/json (version 0.5.13, Python 3.10+, classifiers)
• PyPI litgpt 0.5.13 project page — pypi.org/project/litgpt/0.5.13/ (release listing)
• GitHub releases API for LitGPT — api.github.com/repos/Lightning-AI/litgpt/releases/tags/v0.5.13 (release date 29 June 2026)
• GitHub commits API for LitGPT — api.github.com/repos/Lightning-AI/litgpt/commits (latest commit 28 October 2026)
• LitGPT finetune_lora tutorial — raw.githubusercontent.com/Lightning-AI/litgpt/main/tutorials/finetune_lora.md (measured GPU memory, QLoRA, merge_lora)
• LitGPT inference tutorial — raw.githubusercontent.com/Lightning-AI/litgpt/main/tutorials/inference.md (chat, generate, 3B and 70B inference memory)
• LitGPT Python API tutorial — raw.githubusercontent.com/Lightning-AI/litgpt/main/tutorials/python-api.md (LLM.load, generate, streaming, benchmark)
• LitGPT Zero to LitGPT tutorial — raw.githubusercontent.com/Lightning-AI/litgpt/main/tutorials/0_to_litgpt.md (full install, download, evaluate, pretrain)
• LitGPT deploy tutorial — raw.githubusercontent.com/Lightning-AI/litgpt/main/tutorials/deploy.md (serve, streaming, OpenAI-compatible endpoints)
• LitGPT finetune tutorial — raw.githubusercontent.com/Lightning-AI/litgpt/main/tutorials/finetune.md (older subcommand forms, LoRA config)
• LiteLLM vendor site — litellm.ai (LiteLLM is BerriAI's AI gateway and LLM proxy, a different product)
• Wikipedia, LiteLLM — en.wikipedia.org/wiki/LiteLLM (developer BerriAI, type LLM gateway)

Frequently Asked Questions

What is LitGPT used for?

LitGPT is used to pretrain, finetune, evaluate, chat with, and serve open large language models on your own hardware. Its most common job is LoRA finetuning: taking a pretrained model such as Phi-2 and teaching it a specific task on your own data, then serving the result. It supports more than 20 model families, from 135-million-parameter SmolLM2 to 405-billion-parameter Llama 3.3.

Is LitGPT free?

Yes. LitGPT is open source under the Apache 2.0 licence, so the software itself costs nothing and you can use it commercially. The real cost is hardware: finetuning needs a GPU, and the docs measured about 24 GB of GPU memory for a 3B model with default settings, or about 6 GB with bf16-true plus 4-bit quantization.

You pay for the machine, not for LitGPT.

Is LitGPT still maintained?

Yes, as of 6 November 2026. The latest commit on the main branch landed on 28 October 2026, nine days before this guide, and it added DeepSeek-R1-Distill-Qwen model configs. The last release, 0.5.13, dates from 29 June 2026, which is 130 days earlier, so commits are ahead of releases.

Check the PyPI page before installing, in case a newer release has landed since this guide.

Do I need a GPU to use LitGPT?

You need a GPU for finetuning, and the docs say at least about 24 GB of GPU memory for a 3B model with default settings, or about 6 GB with quantization. For chatting and one-shot generation, a GPU is also the documented path, with a 3B model needing about 7 GB of GPU memory in bfloat16. CPU-only chat is not verified in any fetched LitGPT page, so do not plan a CPU-only setup without testing it first.

What is the difference between LitGPT and LiteLLM?

LitGPT and LiteLLM are different products from different vendors with similar names. LitGPT is Lightning AI's toolkit for pretraining, finetuning, and serving open models locally. LiteLLM is BerriAI's AI gateway and LLM proxy, which routes calls to hosted providers through one OpenAI-compatible API.

They share no code, no commands, and no versions, so install instructions for one never apply to the other.