How to Use dlt: The Beginner's Guide (First Pipeline, the Three-Phase Lesson, and the dbt Handoff)

CodexFuse Team · 2026-11-02T12:18:04.761Z · 13 min read min read

dlt is the open-source Python library that automates data loading. This beginner guide covers install, your first pipeline to DuckDB, dlt init, incremental loading, schema evolution, and the dbt handoff.

What Is dlt, Exactly?

dlt is "data load tool (dlt) — the open-source Python library that automates all your tedious data loading tasks." That is the project's own description, from its README. You point dlt at a source — a REST API, a CSV file, a SQL database, a Google Sheet — and it extracts the data, normalizes it into a destination-compatible schema, and loads it into your warehouse. You never write a CREATE TABLE statement by hand. This guide covers how to use dlt from your first install through incremental loading and the dbt handoff.

Before going further, the disambiguation: this guide is about the dlt Python library from dltHub. The bare letters "DLT" also name an automotive logging protocol (Diagnostic Log and Trace), an antihistamine tablet sold in India, and Databricks' Lakeflow Declarative Pipelines. None of those are this product — if you came for Delta Live Tables, you want Databricks' docs, not this page.

The promise that makes dlt worth learning: "dlt reads the data, creates the tables, and adapts when a source adds or renames a field. You get an alert and a migration instead of a failed load at 3am." That is the vendor's framing, and it captures the whole job — loading messy source data into clean tables without writing ETL boilerplate. And once your data is loaded, you transform it: the classic pairing is dlt for load plus our dbt beginner guide for the transform half.

How Do You Use dlt, Step by Step?

Learning how to use dlt comes down to five moves, and the rest of this guide walks each one: install the package, run your first pipeline into DuckDB, scaffold a real source with dlt init, add incremental loading, and hand the loaded tables to dbt for transformation.

1. Install: pip install "dlt[duckdb]" on Python 3.10 through 3.14.
2. First pipeline: call dlt.pipeline() with a destination of duckdb, then pipeline.run(data, table_name="users").
3. Scaffold: run dlt init <source> <destination> to pull in a verified source connector.
4. Go incremental: declare a cursor once — re-runs load only what changed.
5. Transform: hand the loaded tables to dbt; dlt even ships a dbt runner for this.

What Do You Install to Get Started?

You install the package dlt — never python-dlt. That older PyPI name belongs to a different, abandoned project; the real one is pip install dlt, straight from the dlt-hub/dlt README.

pip install "dlt[duckdb]"

The extras system is the one install decision that matters. The base package is destination-agnostic, and you bolt on what you need:

dlt supports Python 3.10 through 3.14, per the README — the older 3.9 floor you may see in dated tutorials is out of date, so check your interpreter before you start. It runs wherever Python runs: a laptop, a notebook (our Jupyter beginner guide covers the notebook side), an AWS Lambda, or inside an orchestrator. The project is Apache 2.0 licensed, owned by dlt-hub on GitHub, and the version current at write time is 1.32.0.

How Does Your First Pipeline Work?

Your first pipeline is three lines: declare the pipeline, hand it data, and name the table. Everything else dlt figures out.

import dlt

pipeline = dlt.pipeline(pipeline_name="users_demo", destination="duckdb", dataset_name="users_data")
load_info = pipeline.run(data, table_name="users")

The three constructor arguments each do one job. pipeline_name identifies the pipeline in traces and lets dlt restore its state and schemas on later runs. destination is the target — a single string you swap later. dataset_name is the logical group of tables (a schema in a relational database); skip it and it defaults to {pipeline_name}_dataset. The pipeline.run(data, table_name="users") pattern comes from dlt's own official tutorial.

When you call run, the pipeline executes three discrete steps — the three-phase lesson you need to internalize once, because every dlt behavior traces back to it:

What Does dlt init Actually Create?

dlt init is the scaffolding command: "Creates a pipeline in the current folder by adding existing verified source or creating a new one from template." One command, and a bare folder becomes a runnable pipeline project.

Running it does six things: it creates .dlt/config.toml and .dlt/secrets.toml plus a .gitignore, pulls in the verified source if you named a real one, generates a template for unknown sources, rewrites the pipeline scripts for your destination, adds sample configuration and credentials, and writes a requirements.txt. It is re-runnable — call it again to add more sources or swap destinations.

dlt init rest_api duckdb

That one line scaffolds a REST-API-to-DuckDB pipeline from the verified rest_api source. There is also a newer AI-harness flow (uvx dlthub-init) launched with the dltHub platform in 2026 — dlt is explicitly "built from the ground up for LLMs and coding agents" — but the classic dlt init still works and is the one to learn first.

Where Does Your Data Land When You Develop Locally?

It lands in DuckDB. This is a deliberate design choice, not a toy default: "dlt allows you to build and test your data pipelines locally with DuckDB and then switch out the destination for deployment." You develop against a local file, and moving to production means changing one argument.

The destination list shows why that works — the pipeline itself never changes, only the string:

"Change one argument to go from DuckDB to Snowflake, BigQuery, Databricks, Postgres, Iceberg, or a vector store. The pipeline itself does not change." MotherDuck's own docs confirm the mechanic: dlt "uses DuckDB as its local development destination... so a pipeline you develop locally against DuckDB loads into MotherDuck by switching the destination." This is why the dlt DuckDB combination is the standard starting point — the development loop is local and free, and deployment is a one-line change.

How Does Incremental Loading Work?

dlt incremental loading is how you stop re-loading the whole table every run: you declare a cursor field once, and dlt's state persists between runs so each re-run loads only what changed.

updated_at = dlt.sources.incremental("updated_at", initial_value="1970-01-01T00:00:00Z")
print(updated_at.last_value)

Under the hood, "dlt takes care of finding maximum/minimum cursor field values, removing duplicates, and managing the state with the last values of the cursor." For declarative sources like the REST API connector, you name the cursor path directly in config:

"incremental": {"cursor_path": "ts", "initial_value": "2024-01-01T00:00:00Z"}

The product page's framing is the one-liner to remember: "Incremental by default — Declare a cursor and a primary key once. State persists between runs, so a re-run loads what changed rather than the whole table again." This is the feature that separates a dlt pipeline from a cron job that dumps a full CSV nightly.

What Happens When Your Source Adds a Column?

Nothing breaks — and that is the point. During normalization, dlt inspects your data and computes the schema, and "the schema will automatically evolve to accommodate any future source data changes like new columns or tables." The vendor's punchline: "You get an alert and a migration instead of a failed load at 3am."

The dlt schema evolution behavior is the reason the vendor can promise you "an alert and a migration instead of a failed load at 3am." Traditional hand-built ETL fails when the upstream API adds a field — your typed schema rejects the new column, the load dies, and you find out in the morning. dlt detects the new column, evolves the schema, runs the migration during the load phase, and tells you. For production pipelines you can tighten this with schema contracts (evolve, freeze, or discard behaviors), but the default posture is: adapt, don't fail.

How Do Verified Sources Save You Time?

The dlt verified sources catalog is the set of dltHub-maintained connectors you pull into your project with dlt init <source> <destination> — a curated set covering the usual suspects like Kafka, MongoDB, Postgres CDC, Stripe, and HubSpot. They save you from writing the connector layer yourself: authentication, pagination, rate limits, and the cursor are already handled.

Three core sources ship with the library itself: rest_api (the generic REST client — the one most beginners start with), sql_database (reads from 30+ SQL databases), and filesystem (S3, GCS, Azure, Drive, SFTP). Beyond that, the "5000+ sources" claim comes from the LLM-native workflow: the typed, declarative primitives plus dlthub.com/context let you generate a working connector from a prompt — "often in a single shot," per the README. That number is the project's positioning, not a connector catalog you browse, but the mechanism is real: describe the API, get a pipeline.

When Should You Use dlt Instead of Airflow?

Search for dlt vs Airflow and you will find both docs and comparison pieces, but the question itself misframes the relationship — dlt runs inside orchestrators, it does not replace them. dlt is simply a Python package, so it "will run everywhere that Python runs — locally, in notebooks, on orchestrators." The official stance: "Keep the orchestrator you already have — Already on Airflow or dbt? dltHub runs beside them: one dlt pipeline, one Airflow task per resource, your DAG still driving incremental loads."

In practice that means the dlt Airflow helper maps your dlt resources into Airflow tasks, dlt deploy <pipeline>.py airflow-composer generates the DAG, and Airflow backfill drives incremental windows via allow_external_schedulers=True. There are official deploy walkthroughs for Dagster and Kestra too. One caution: the newer paid dltHub platform markets itself as "the orchestrator you were about to stand up is one you no longer need" — that claim belongs to the managed platform, not the open-source library. Keep the two products distinct.

What Comes After Loading: The dbt Handoff?

dlt is the LOAD half of ELT, and dbt is the TRANSFORM half — this is the industry pair, and dlt's own CEO says it plainly: "dbt stands for Data Build Tool. They're a transform layer tool. So they transform data that's already in your warehouse. And dlt is a data load tool. So it's loading the data where you want to do further transforms."

dlt's own blog documents the arc: extract and load with dlt, then the tables dlt created become sources in dbt for transformation. And the integration is one line deep — adding dlt's dbt runner to your Python script "configured my pipeline to automatically transform the data after loading it." If you are starting from zero on the transform side, read our dbt beginner guide next — and if you want the pipeline to run on its own schedule, our Docker beginner guide and the what is VPS hosting explainer cover where a Python pipeline actually lives in production.

What Are the 7 Mistakes Beginners Make With dlt?

Sources

1. dlt-hub/dlt README on GitHub — package self-description, install lines and extras, Python 3.10–3.14 support, 5000+ sources and LLM-native workflow claims, Apache 2.0 license, 6,050 stars (snapshot 2026-11-02)
2. dlthub.com/docs/reference/explainers/how-dlt-works — the three phases (extract, normalize, load), pipeline.run pattern, DuckDB local-development quote
3. dlthub.com/docs/general-usage/pipeline — dlt.pipeline(pipeline_name, destination, dataset_name) argument definitions
4. dlthub.com/docs/tutorial/load-data-from-an-api — official tutorial: pipeline.run(data, table_name="users"), incremental loading, dlt pipeline quick_start show
5. dlthub.com/docs/reference/command-line-interface — dlt init scaffolding steps, dlt pipeline info/show inspection commands
6. dlthub.com/docs/general-usage/incremental/cursor — cursor-based incremental loading, dlt.sources.incremental semantics
7. dlthub.com/context/pipeline/slack-block-kit-to-databricks — cursor_path config example for REST sources
8. dlthub.com/product/dlt — schema-evolution framing ("an alert and a migration instead of a failed load at 3am"), "runs where Python runs", download/community numbers
9. dlthub.com/blog/dlt-dbt-semantic-layer — the dlt-then-dbt ELT arc
10. dlthub.com/blog/dlt-motherduck-demo — dlt's dbt runner ("automatically transform the data after loading it")
11. Heavybit Open Source Ready podcast, ep. 31 — dltHub CEO on dlt vs dbt ("loading the data where you want to do further transforms")
12. dlthub.com/docs/hub/getting-started/oss-and-dlthub — verified sources, dlt init <source> <destination>
13. motherduck.com/docs/integrations/ingestion/dlt — "dlt uses DuckDB as its local development destination"
14. qdrant.tech/documentation/data-management/dlt — dlt as a vector-destination integration (pip install "dlt[qdrant]")
15. newreleases.io/project/github/dlt-hub/dlt/release/1.32.0 — 1.32.0 release notes (current version at write time)
16. datacamp.com/tutorial/python-dlt — DataCamp's Python dlt tutorial (install → init → REST API → DuckDB)
17. getorchestra.io/guides/dlt-concepts-getting-started — getorchestra's dlt concepts guide (referenced for existence; its code snippets were not used)
18. datainstitute.io — "dlt Tutorial: Data Integration with Python in 10 Minutes" (uv-based lean-stack tutorial)
19. tsecurity.de — dlt workshop recap (extract/normalize/load + incremental loading to DuckDB)

Frequently Asked Questions

Is dlt the same as Databricks DLT?

No. This dlt is the open-source Python "data load tool" from dltHub. Databricks DLT (Lakeflow Declarative Pipelines) is a different product from a different company. The bare letters also collide with an automotive logging protocol and a tablet — always check which "dlt" a page means.

Do I need Airflow to run dlt?

No. dlt is a Python package that runs anywhere Python runs — a laptop, a notebook, a Lambda, a cron job on a VPS. You add Airflow (or Dagster, or Kestra) when you have many pipelines to schedule; the official Airflow helper maps dlt resources into Airflow tasks when that day comes.

Can dlt replace dbt?

No — they are the two halves of ELT. dlt loads data into your warehouse and infers the schema; dbt transforms the loaded tables into analytics models. dlt even ships a dbt runner so the transform fires automatically after the load.

Which destinations does dlt support?

Twenty-plus, swapped with one string: DuckDB for local development, then Snowflake, BigQuery, Postgres, Redshift, Databricks, MotherDuck, ClickHouse, Athena, Iceberg, Delta, filesystem targets (S3/GCS/Azure), and vector stores like Qdrant. The pipeline code does not change.

Is dlt free?

The open-source library is Apache 2.0 licensed — free, commercial use included. dltHub also sells a managed platform with scheduling and verified-source hosting; keep the two products distinct when reading their docs.