How to Use XGrammar: Structured Generation (JSON Schemas, EBNF) From Zero to Your First Grammar

CodexFuse Team · 2026-11-06T12:29:35.645Z · 14 min read min read

How to Use XGrammar: Structured Generation (JSON Schemas, EBNF) From Zero to Your First Grammar

Learning how to use XGrammar is the fastest way to stop a language model from handing you broken JSON. XGrammar is an open-source library from MLC AI that uses constrained decoding to guarantee the model's output follows a structure you choose, such as a JSON schema or a custom grammar. It promises 100 percent structural correctness, and it is the default structured generation backend in vLLM, SGLang, TensorRT-LLM, and MLC-LLM.

All facts in this guide are as of November 2026, and version numbers move fast, so treat the release details as a snapshot.

XGrammar works with any model you can load in Python, and it also runs inside the serving engines you already use. If you plan to serve the model in a container, our Docker beginner guide shows the setup, and the rest of this guide stays in plain Python. This xgrammar tutorial walks you from a bare install to your first grammar, then into EBNF mode and the vLLM and SGLang integrations.

What Is XGrammar and What Problem Does It Solve?

XGrammar solves the broken-output problem: without it, a model can return JSON with a missing brace, a wrong field name, or an invented value, and your parser fails. XGrammar constrains decoding so every token the model emits keeps the output valid, which means the result always parses. It enforces format, not meaning, so the model can still write a wrong answer inside a perfectly valid structure.

XGrammar gives you four ways to describe the structure you want, and each one fits a different job. Pick the mode that matches your output, and the rest of this guide shows each one in turn.

The project began on 28 June 2024, released version 0.1.0 in November 2024, and shipped XGrammar-2 in May 2026 with tool-calling support. The repository holds 1,993 stars and 260 forks under the Apache-2.0 license, as checked live on 6 November 2026. Big serving systems adopted it early: SGLang and MLC-LLM in November 2024, vLLM in December 2024, and TensorRT-LLM in January 2025.

How to Install XGrammar

Install XGrammar with one pip command, and check the version before you trust any tutorial. The current release is 0.2.8, published on PyPI on 24 September 2026, which makes it 43 days old on 6 November 2026. Search results may still show the older 0.2.6 page, and the docs carry a 0.2.9.dev label, so trust the PyPI page, not a search snippet, and do not mistake the development label for a release.

Wheels exist for Python 3.9 through 3.14, including free-threaded 3.14 builds, on Linux, macOS, and Windows. The library also exposes C++, JavaScript, and Swift APIs, and XGrammar-2 adds a Rust API. If you have not run a model on your own machine yet, our Ollama beginner guide is the quickest start, and the Quick Start below uses Hugging Face transformers instead.

How to Use XGrammar: Your First JSON Grammar

Your first grammar takes seven steps, and the official Quick Start walks the same path with Hugging Face transformers. The example below uses Llama-3.2-1B-Instruct, and it is quoted from the official Quick Start for XGrammar 0.2.8. Work through it once, and you will have the whole mental model.

1. Install xgrammar and transformers.
2. Load the model, tokenizer, and config.
3. Build the chat prompt and tokenize it.
4. Build TokenizerInfo from the tokenizer.
5. Compile a grammar with GrammarCompiler.
6. Wrap it in the Hugging Face LogitsProcessor.
7. Generate with the processor and decode the output.

First, prepare the model and the prompt:

import xgrammar as xgr
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig

device = "cuda"  # Or "cpu" if you don't have a GPU

model_name = "meta-llama/Llama-3.2-1B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_name, torch_dtype=torch.float32, device_map=device
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
config = AutoConfig.from_pretrained(model_name)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Introduce yourself in JSON briefly."},
]

texts = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
model_inputs = tokenizer(texts, return_tensors="pt").to(model.device)

Next, compile the grammar. The built-in JSON grammar is the simplest start, and the commented lines show the other two entry points:

tokenizer_info = xgr.TokenizerInfo.from_huggingface(tokenizer, vocab_size=config.vocab_size)
grammar_compiler = xgr.GrammarCompiler(tokenizer_info)
compiled_grammar = grammar_compiler.compile_builtin_json_grammar()

# Other ways: provide a json schema string
# compiled_grammar = grammar_compiler.compile_json_schema(json_schema_string)
# Or provide an EBNF string
# compiled_grammar = grammar_compiler.compile_grammar(ebnf_string)

Finally, generate with the grammar through the logits processor:

xgr_logits_processor = xgr.contrib.hf.LogitsProcessor(compiled_grammar)
generated_ids = model.generate(
    **model_inputs, max_new_tokens=512, logits_processor=[xgr_logits_processor]
)
generated_ids = generated_ids[0][len(model_inputs.input_ids[0]) :]
print(tokenizer.decode(generated_ids, skip_special_tokens=True))

One habit makes the whole thing work better: describe the expected output structure clearly in the prompt. Constrained decoding only affects the sampling stage, so the model's own probability distribution should already lean toward the structure you want.

How to Use the XGrammar JSON Schema Mode

JSON Schema mode is the workhorse of structured generation Python developers reach for first: you hand XGrammar a schema, and the model can only emit JSON that conforms to it. Compile it with grammar_compiler.compile_json_schema(json_schema_string), then use the same LogitsProcessor as the first example. The schema supports a Draft 2020-12 subset, including types, enum, const, anyOf, allOf, pattern, format, and numeric bounds.

import json

schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "age": {"type": "integer", "minimum": 0},
        "email": {"type": "string", "format": "email"},
    },
    "required": ["name", "age"],
    "additionalProperties": False,
}
compiled_grammar = grammar_compiler.compile_json_schema(json.dumps(schema))

A real gotcha from October 2026 shows why the details matter: with any_whitespace=False and no separators argument, XGrammar 0.2.8 still writes ": " after each key, so compact JSON gets rejected. Passing separators=(",", ":") to compile_json_schema fixes it. The same note measured the effect: on structure tokens, the allowed-token menu shrinks to 2 to 12 tokens out of GPT-2's 50,257, which is 0.004 to 0.024 percent of the vocabulary.

This is the mode behind the guaranteed JSON output LLM pipelines ask for: your API receives valid JSON every time, with no repair step and no retry loop. Use it for tool arguments, form extraction, and any endpoint that parses the model's answer.

How to Use XGrammar EBNF Mode

EBNF mode is for structures a JSON schema cannot describe, such as SQL, a mini programming language, or a strict output format with fixed keywords. You write the grammar in extended Backus-Naur form and compile it with grammar_compiler.compile_grammar(ebnf_string). XGrammar follows the GBNF specification from llama.cpp, so existing GBNF grammars work, and every grammar needs a root rule.

The vLLM docs show the idea with a small SQL grammar, which you can send as a structured-outputs grammar:

root ::= select_statement
select_statement ::= "SELECT " column " from " table " where " condition
column ::= "col_1 " | "col_2 "
table ::= "table_1 " | "table_2 "
condition ::= column "= " number
number ::= "1 " | "2 "

Rules use ::=, with choice (|), groups, quantifiers (? * + {n,m}), character classes, and string literals. All input formats are normalized to EBNF internally, then compiled through the same pipeline, so JSON Schema mode and EBNF mode share one engine. Choose EBNF when the output has fixed keywords or ordering rules that a schema cannot express.

JSON Schema vs EBNF: Which Mode Should You Choose?

Choose JSON Schema when the output is data, and EBNF when the output is language. JSON Schema is faster to write for API payloads, form extraction, and tool arguments, while EBNF wins for SQL, code, and formats with fixed keywords. Both modes guarantee valid output, so the choice is about which description is shorter, not which is safer.

When you are unsure, start with JSON Schema. If the schema starts fighting you with patterns and workarounds, that is the signal to rewrite the rule as EBNF.

How the Logits Processor Works

The logits processor is the whole trick: at each decoding step, XGrammar computes a mask of the tokens that can legally continue the output, sets every invalid token's probability to zero, and lets the model sample only from the valid ones. The mask is precomputed at compile time, so generation runs with near-zero overhead. XGrammar splits the vocabulary into context-independent tokens, whose validity never depends on parser state, and context-dependent tokens, which are checked live.

# Numbers checked on 6 November 2026 (sources: PyPI, MLC blog, justmalhar.com)
release = 43            # days since XGrammar 0.2.8 (24 Sep 2026)
repo_age = 861          # days since the repo was created (28 Jun 2024)
speedup = 534 / 5.37    # repetition compression, ms to ms
saving = (1 - 5.37 / 534) * 100   # percent of preprocessing time removed
menu_low = 2 / 50257 * 100       # smallest allowed-token menu, % of GPT-2 vocab
menu_high = 12 / 50257 * 100    # largest allowed-token menu, % of GPT-2 vocab
compile_80x = 800 / 80          # example: an 800 ms compile at 80x speedup, ms
tool_scale = 500 / 10           # tool count scaling in the XGrammar-2 test
print(release, repo_age, round(speedup, 1), round(saving, 1),
      round(menu_low, 3), round(menu_high, 3), compile_80x, tool_scale)
# Output: 43 861 99.4 99.0 0.004 0.024 10.0 50.0

The numbers behind the design are striking: XGrammar-2 cut repetition preprocessing from 534 ms to 5.37 ms, about a 99.4 times reduction, and reports up to an 80 times compilation speedup as tool counts scale from 10 to 500. The XGrammar-2 paper also reports near-zero end-to-end overhead, with the gap against unconstrained generation at or under 6 percent. XGrammar is best used to enforce format constraints, not to change the semantics of a response, so a model can still write a wrong answer inside a perfectly valid structure.

How to Use XGrammar in vLLM

In vLLM, XGrammar is already there: vLLM supports structured outputs with xgrammar or guidance as backends, and structured outputs are on by default in the OpenAI-compatible server. The backend is chosen with --structured-outputs-config.backend, and the default is auto, which picks a backend from the request details. CodexFuse does not offer an XGrammar tool, so this guide teaches the library itself, not a hosted shortcut.

Send a schema with the standard response_format field, or use the structured_outputs shape in extra_body with one of choice, regex, json, grammar, or structural_tag. Offline, use StructuredOutputsParams inside SamplingParams. That is the XGrammar vLLM path in one paragraph: the engine picks the backend, and your schema does the rest.

response = client.chat.completions.create(
    model="meta-llama/Llama-3.2-1B-Instruct",
    messages=[{"role": "user", "content": "Introduce yourself in JSON briefly."}],
    response_format={"type": "json_schema", "json_schema": schema},
)

Old tutorials are a trap here: the guided_json, guided_regex, guided_choice, guided_grammar, and guided_decoding_backend fields were removed in v0.12.0, so any guide still using them is outdated. Use structured_outputs in requests or StructuredOutputsParams in offline SamplingParams instead. If you have not served a model with vLLM yet, our vLLM beginner guide covers the setup.

How to Use XGrammar in SGLang

SGLang makes XGrammar the default: you can pass a JSON schema, a regular expression, or EBNF to constrain the output, and only one constraint parameter can be set per request. If no backend is specified, XGrammar is used as the default, and you switch with --grammar-backend outlines or --grammar-backend llguidance at server launch. The docs suggest XGrammar for its better performance and utility, and note that XGrammar currently uses the GGML BNF format.

The same prompt advice applies here as in the Quick Start: explicitly tell the model the format you want, because constrained decoding only shapes sampling. SGLang exposes the constraints through the OpenAI-compatible API response_format, the native /generate endpoint, and the offline engine API.

vLLM vs SGLang: XGrammar Defaults Compared

Both engines ship XGrammar, but they choose it differently: vLLM picks a backend per request, while SGLang defaults to XGrammar unless you say otherwise. The practical result is the same for most users, since a JSON schema request gets an XGrammar-backed answer either way.

If you already run one of these engines, you do not need to install anything extra to use XGrammar. If you are choosing an engine, pick it for the serving features you need, not for structured generation, since both give you the same guarantee.

Where XGrammar Sits vs Outlines

Outlines is the Python-first library a developer calls directly: you pass a Python type, a Pydantic model, a Regex, or a Lark grammar to a model wrapper, and its Rust outlines_core compiles JSON schemas to regex and indexes them as a finite-state machine. XGrammar is the grammar engine underneath the serving systems: it compiles JSON Schema, regex, EBNF, and Lark into a general context-free grammar with Earley parsing at runtime, and it ships as the default backend in vLLM, SGLang, TensorRT-LLM, and MLC-LLM. SGLang even offers Outlines as an alternative backend to its XGrammar default, so the two are complements at different layers, not rivals, and both are Apache-2.0.

Structural Tags and Tool Calling

XGrammar-2, released in May 2026, adds the Structural Tag mode for agentic work: outputs that mix reasoning, free-form text, and tool calls, with built-in styles for Llama, Qwen, DeepSeek, Kimi, and OpenAI Harmony models. The tag format carries the function name and arguments inside a strict structure, so a tool call can never arrive with a broken field or an invented name. On the BFCL-V3 benchmark, the MLC team reports the Structural Tag improves schema accuracy to 100 percent.

Tool calling only works when the model, the schema, and the server agree on the protocol. Our what MCP is guide explains the protocol servers use to expose tools, and our agentic AI explainer covers how agents plan multi-step work around those calls.

Sources

XGrammar, Quick Start, XGrammar 0.2.8, xgrammar.mlc.ai/docs/start/quick_start.html.

PyPI, xgrammar 0.2.8, released 24 September 2026, pypi.org/project/xgrammar.

GitHub, mlc-ai/xgrammar, 1,993 stars, 260 forks, Apache-2.0, created 28 June 2024, github.com/mlc-ai/xgrammar.

MLC blog, XGrammar-2: Fast and Customizable Structured Generation for Tool Calling and Agents, 4 May 2026, blog.mlc.ai/2026/05/04/xgrammar-2-fast-customizable-structured-generation.

vLLM docs, Structured Outputs, v0.32.0, docs.vllm.ai/en/v0.32.0/features/structured_outputs.

SGLang docs, Structured Outputs, updated 6 November 2026, docs.sglang.io/docs/advanced_features/structured_outputs.

justmalhar.com, TIL: xgrammar's JSON grammar sets your separators, 27 October 2026, justmalhar.com/notes/xgrammar-json-separators.

Metrale Engine Book, Constrained Decoding (XGrammar), book.dev.metrale.ai/deep-dives/xgrammar.

DeepWiki, EBNF Grammar Specification, mlc-ai/xgrammar, deepwiki.com/mlc-ai/xgrammar/4.1-compiler-architecture.

DeepWiki, JSON Schema Converter Deep Dive, mlc-ai/xgrammar, deepwiki.com/mlc-ai/xgrammar/5-grammar-matching.

Li et al., XGrammar-2: Dynamic and Efficient Structured Generation Engine for Agentic LLMs, arXiv:2601.04426.

Dong et al., XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models, MLSys 2025, arXiv:2411.15100.

dev.to, Outlines: Stop Parsing Broken JSON, Constrain the Tokens Instead, November 2026, dev.to/m_t_ramkrushna/outlines-stop-parsing-broken-json-constrain-the-tokens-instead-4546.

docs.rs, outlines_core 0.2.3, JSON-schema-to-regex finite-state-machine design, docs.rs/outlines-core/0.2.3/outlines_core.

Frequently Asked Questions

What Python versions does XGrammar support?

XGrammar ships wheels for Python 3.9 through 3.14, including free-threaded 3.14 builds, on Linux, macOS, and Windows. Install it with pip install xgrammar, or pip install "xgrammar[metal]" on Apple Silicon.

Should I use a JSON schema or EBNF with XGrammar?

Use a JSON schema when the output is data, such as tool arguments or API payloads, because it is faster to write and you often already have the schema. Use EBNF when the output is language with fixed keywords or ordering rules, such as SQL or a mini programming language.

Does XGrammar slow down generation?

Almost not at all, according to the project's own measurements: the mask is precomputed at compile time, and the XGrammar-2 paper reports near-zero end-to-end overhead, with the gap against unconstrained generation at or under 6 percent. Compilation itself is the heavier step, and XGrammar-2 reports up to an 80 times compilation speedup over the first version.

Can I use XGrammar without vLLM or SGLang?

Yes: the Quick Start runs XGrammar directly with Hugging Face transformers through xgr.contrib.hf.LogitsProcessor, with no serving engine at all. The vLLM and SGLang integrations are conveniences for production serving, not requirements.

Does XGrammar make the model's answers correct?

No: XGrammar guarantees the shape of the output, not the truth of it. A model can still invent a wrong value, pick the wrong tool, or answer the wrong question, and the output will be perfectly valid JSON while doing it.