How to Use Daft: A Beginner's Guide to the Distributed Dataframe

CodexFuse Team · 2026-11-02T10:25:49.515Z · 16 min read min read

Daft is a distributed Python dataframe engine with a lazy plan and a Ray runner. This beginner guide covers install, your first read_csv, the lazy-plan lesson, groupby, Ray, Delta Lake, Iceberg, and SQL — verified live on daft 0.7.26.

What Is Daft, Exactly?

If you are learning how to use Daft, start with this one sentence: Daft is a high-performance data engine for AI and multimodal workloads — a familiar Python dataframe API with a lazy execution plan that scales from your laptop to a Ray cluster. This daft python tutorial takes you from pip install to your first read_csv, the lazy-plan lesson that makes Daft click, your first groupby, the Ray scale-up, Delta Lake and Iceberg reads, and the SQL interface.

Every code block in this guide ran on daft 0.7.26 in a clean virtual environment while it was being written, and the plan outputs, row counts, and totals below are the real ones. The vendor describes Daft in its own words as "a high-performance data engine for AI and multimodal workloads" that can "process images, audio, video, and structured data at any scale." Daft is free and open source under the Apache 2.0 license, its latest stable release is v0.7.26 (uploaded to PyPI on Oct 6, 2026), and the project holds roughly 5,821 GitHub stars at the time of writing — a number that moves daily. If you run your Python data work in notebooks, this guide pairs naturally with our Jupyter beginner guide.

How to Use Daft for the First Time: 5 Steps

You use Daft in five steps, and the shape never changes no matter how big the data gets:

1. Install it. Run pip install -U daft in a Python 3.10+ environment — one package, no JVM, no cluster software.
2. Make a DataFrame. Point Daft at data with daft.read_csv("sales.csv"), daft.read_parquet("data/"), or build one in memory with daft.from_pydict({"a": [1, 2, 3]}).
3. Chain lazy operations. Write the query the way you would in pandas: df.where(df["amount"] > 100).select("region", "amount"). Nothing has run yet — Daft is only building a plan.
4. Trigger execution. Call a materializing operation — .collect(), .show(), .to_arrow(), or a write like .write_parquet() — and the plan runs.
5. Inspect and scale. Call .explain() to read the plan Daft built, and call daft.set_runner_ray() when the same code needs a cluster.

The rest of this daft tutorial walks each step with real output, so you can see the lazy plan fire in front of you.

How Do You Install Daft?

You install Daft with one line: pip install -U daft. The install docs say it plainly — "Install Daft with pip install daft" — and require Python 3.10 or newer. I verified this line in a fresh virtual environment on 2026-11-02: it installed daft 0.7.26 cleanly with no extra setup.

pip install -U daft

One caution, said once because it will save you real confusion: the old package name getdaft is legacy — the getdaft listing on PyPI is now a tiny stub pinned at 0.5.0 from August 2025, and the live extras all use the new name. If you follow an older tutorial that says pip install getdaft[deltalake], translate it to pip install -U "daft[deltalake]" before you run it.

Daft ships one base package and grows through extras. The extras worth knowing first are:

The full list also includes turbopuffer, unity, lance, kafka, gravitino, http, and mcap. Install only what your job needs — the base package already covers local CSV, Parquet, and JSON work. If you keep Daft pinned in a requirements.txt inside a versioned project, pair this section with our Git beginner guide.

Why Does My Daft Code Not Do Anything Until collect()?

Because Daft is lazy: your Python code builds an execution plan, and nothing reads or transforms data until you call a materializing operation. The architecture docs explain it directly: "Users express their workloads via the Python DataFrame API or the SQL interface. These APIs are lazy: operations construct a plan of execution, but they do not read or transform data yet. Internally, the workload is represented as a LogicalPlan: a tree of operators that describe what work should happen. Example operators include Source, Project, Filter, GroupBy, and Join."

Watch it happen. This is the real output of .explain() on daft 0.7.26 for a filter-and-select query — two lines of Python produced a three-operator plan, and the CSV was never touched:

import daft

df = daft.read_csv("sales.csv")
q = df.where(df["amount"] > 100).select("region", "amount")
print(q.explain())

== Unoptimized Logical Plan ==

* Project: col(region), col(amount)
|
* Filter: col(amount) > lit(100)
|
* GlobScanOperator
|   Glob paths = [sales.csv]
|   Has headers = true
|   File schema = order_id#Int64, region#Utf8, amount#Int64

The plan is a tree that reads bottom-up: scan the file, filter rows, project columns. Execution is triggered by "a materializing operation such as collect(), show(), to_arrow(), and write methods such as write_parquet()." Once you ask, the plan moves through Daft's three stages — Planning, Optimization, Execution — and the optimizer rewrites it before a single byte is read.

Ask for the optimized plan and you can watch the rewrite: the filter gets pushed down into the scan itself, and Daft fans the work out across 8 parallel scan tasks on this machine. Real output from .explain(show_all=True):

== Optimized Logical Plan ==

* Num Scan Tasks = 1
|   Pushdowns: {filter: col(amount) > lit(100)}
|   Filter pushdown = col(amount) > lit(100)

== Physical Plan ==

* ScanTaskSource:
|   Num Parallel Scan Tasks = 8

That pushdown is the payoff of laziness: Daft knew which rows you wanted before it opened the file, so it never read the rest. The optimizer applies rule-based rewrites (filter, projection, limit, and aggregation pushdowns, projection folding, pruning redundant repartitions) plus cost-based join reordering, per the architecture docs. Beginners should internalize exactly one sentence: build the query, then collect — Daft optimizes the whole thing at once.

Your First read_csv: How Do You Read and Aggregate a CSV?

You read a CSV the same way you would anywhere else — the difference is that daft.read_csv("sales.csv") returns a lazy DataFrame, not data. Build a six-row file, load it, and aggregate: every number below came out of a real run on daft 0.7.26.

import daft

df = daft.read_csv("sales.csv")          # lazy: nothing has run

totals = (
    df.groupby("region")
      .agg(df["amount"].sum().alias("total"))
      .sort("total", desc=True)
)

totals.show()                            # this runs the plan

region   total
------   -----
north    410
south    230
east     45

(Showing first 3 of 3 rows)

The sales file holds six orders; the groupby summed them into north 410, south 230, east 45 — exactly the rows the lazy plan promised. Two more facts worth knowing early: df.count_rows() returned 6 on that file, and daft.read_csv takes fine-grained kwargs for real-world files — has_headers, delimiter, schema, infer_schema, allow_variable_columns, hive_partitioning, and io_config among others. Parquet readers mirror the same argument style, and Daft also reads JSON.

Daft vs Polars: Which Dataframe Fits Your Data?

Daft and Polars are not rivals — they answer different questions, and the makers say so themselves. Polars describes itself as "an open-source library for data manipulation, known for being one of the fastest data processing solutions on a single machine," with a "multi-threaded query engine written in Rust and designed for effective parallelism." Daft's vendor page calls Daft "a distributed query engine for large-scale data processing in Python and is implemented in Rust," and adds the axis that matters: "Some workloads can quickly outgrow your local laptop's computational resources — Daft integrates natively with Ray for running dataframes on large clusters of machines with thousands of CPUs/GPUs."

Draw the line once: Polars is the fast single-machine frame; Daft is the distributed and multimodal engine with the lazy plan and the Ray runner. Daft's older docs add a second differentiator Polars never claimed — a "rich multimodal type-system" supporting "Images, URLs, Tensors and more," which is why the recent independent review of Daft is a multimodal production pipeline piece, not a single-machine benchmark.

Choose Polars when one fast box is enough and you want the quickest single-machine answer. Choose Daft when the data might outgrow the box, when the job mixes images or audio with tables, or when you want the plan optimized before anything runs. (A roughly two-and-a-half-year-old Data Engineering Central post ran a hands-on Daft-vs-Polars experiment log on S3 CSV reads — it reads as an experiment journal, not a tutorial, and its full body sits behind a paywall; it is not this guide's spine.)

Daft vs Spark: When Do You Outgrow One Box?

Daft vs Spark is the right comparison once a single machine stops being enough — and here the honest framing is my analysis, not a vendor claim. Spark is the long-established distributed engine with a JVM-based architecture and its own cluster manager; Daft pursues the same scale-up job with a pandas-like Python API, a Rust native runner for the one-box case, and the Ray runner when you need a cluster.

The practical difference for a beginner: Daft keeps one mental model from laptop to cluster — the code you wrote with read_csv and groupby above is the same code that runs distributed after daft.set_runner_ray(). There is no separate cluster API to learn, no JVM tuning, and no packaging a job for a different runtime. Spark earns its place in huge JVM-centric estates; Daft is the lighter answer when your team thinks in Python dataframes and wants distribution without a second stack.

How Does the Ray Runner Scale Your Same Code?

The Ray runner scales your same code because it executes the same lazy plan on a Ray cluster instead of your laptop — one call switches the runner, and your DataFrame code does not change. The vendor's Ray docs put it this way: "Once optimization completes, the plan is executed by either the native runner (single-machine) or the Ray runner (distributed)," where "the native runner is a streaming execution engine implemented in Rust, using the Tokio runtime for async I/O and multithreaded operations," and "the Ray runner is a distributed execution engine built on top of Ray."

import daft

# Run on a Ray cluster at this address
daft.set_runner_ray("ray://127.0.0.1:10001")

# Or: no address spins up a local Ray cluster on your machine
daft.set_runner_ray()

The current call is daft.set_runner_ray(...) — the older daft.context.set_runner_ray() form is superseded, so do not copy it from old tutorials. The docs also document autoscaling knobs (downscale_enabled=True, downscale_idle_seconds=60, min_survivor_workers=1) for clusters that should shrink when idle. Install the runner with the Ray extra: pip install -U "daft[ray]".

One honest boundary, said once: I ran every demo in this guide on the native runner in a sandbox — Ray was not installed there, so the cluster behavior above is docs-sourced, not live-tested by me. If you are standing up the cluster itself, a machine to host the Ray head node is the first thing you need — see what is VPS hosting — and containerizing the worker image keeps every node identical, for which our Docker beginner guide covers the basics.

How Do You Read Delta Lake and Iceberg Tables?

You read Delta Lake tables with the Delta Lake extra and one function: pip install -U "daft[deltalake]", then df = daft.read_deltalake("some-table") gives you a Daft DataFrame over the table, and df.write_deltalake("tmp/daft-recordbatch", mode="overwrite") writes one back. Spelling caution, said once: the widely linked delta.io integration blog was written in the old getdaft era and spells the reader read_delta_lake; current docs spell it read_deltalake — use the current spelling in new code.

Iceberg tables work through PyIceberg: install the Iceberg extra, load the table from a catalog, and hand it to Daft. The Apache Iceberg guide's own example reads:

import daft
from pyiceberg.catalog import load_catalog

table = load_catalog("default").load_table("nyc.taxis")
df = daft.read_iceberg(table)
df.show()

The guide's key promise is that "any filter operations on the Daft dataframe, df, will push down the filters, correctly account for hidden partitioning, and utilize table statistics to inform query planning for efficient reads" — the same pushdown mechanic you watched in the optimized plan section, now working against table statistics. Same honesty boundary as the Ray section: the Delta Lake and Iceberg paths above are verified against current vendor docs, not executed against a live table in my sandbox. If you refresh these tables on a schedule, our GitHub Actions beginner guide shows how to run the job nightly.

Can You Query Daft with SQL?

Yes — daft.sql() runs SQL against your DataFrames, and it auto-detects them by name. The SQL docs state: "Daft's daft.sql function automatically detects any daft.DataFrame objects in your current Python environment to let you query them easily by name." This is the real query I ran against the same sales data from the groupby section:

import daft

df = daft.read_csv("sales.csv")
res = daft.sql(
    "SELECT region, SUM(amount) AS total "
    "FROM df GROUP BY region ORDER BY total DESC"
)
res.collect()

It returned the same totals — north 410, south 230, east 45 — because it runs the same lazy plan machinery under the SQL text. For named tables, Daft also offers a Session: sess.create_temp_table("T", df) then sess.sql("SELECT * FROM T").show(), plus daft.sql_expr("A + B as C") for SQL expressions inside DataFrame code. SQL people get a door in; nobody has to rewrite the joins they already wrote.

7 Beginner Mistakes to Avoid with Daft

These are the traps beginners actually fall into, gathered from the docs' sharp edges and my own verification runs — including one I hit myself:

Sources

• Daft install docs — "Install Daft with pip install daft", Python 3.10+, extras list: https://docs.daft.ai/en/stable/install/
• Daft architecture docs — lazy plan, LogicalPlan operators, Planning/Optimization/Execution, runners, execution triggers: https://docs.daft.ai/en/stable/architecture/
• Daft Ray runner docs — daft.set_runner_ray(...), local cluster default, autoscaling options: https://docs.daft.ai/en/stable/distributed/ray/
• PyPI Daft — v0.7.26 release, extras inventory: https://pypi.org/project/daft/
• Eventual-Inc/Daft on GitHub — Apache 2.0 license, star count snapshot (5,821 on 2026-11-02): https://github.com/Eventual-Inc/Daft
• delta.io Daft–Delta Lake integration blog — older getdaft[deltalake] era, read_delta_lake spelling: https://delta.io/blog/daft-delta-lake-integration/
• Daft Delta Lake connector docs — current daft[deltalake] extra, read_deltalake, write_deltalake: https://docs.getdaft.io/en/stable/connectors/delta_lake/
• Apache Iceberg Daft guide — daft.read_iceberg(table), filter pushdown: https://iceberg.apache.org/docs/1.8.1/daft/
• Daft SQL docs — daft.sql auto-detects DataFrames by name: https://docs.getdaft.io/en/stable/sql/
• pola.rs — "one of the fastest data processing solutions on a single machine": https://pola.rs/
• dev.to — "Daft in Production: A Hard-Nosed Multimodal Pipeline Review Against Ray Data and Spark" (~20 days old, independent production review)
• Data Engineering Central — "Introduction to Daft (... vs Polars)" by Daniel Beach (Jun 17, 2024; hands-on experiment log, full body paywalled): https://dataengineeringcentral.substack.com/p/introduction-to-daft-vs-polars
• Author's own live verification — daft 0.7.26 installed via pip install -U daft in a fresh venv on 2026-11-02; read_csv/explain()/groupby().collect()/daft.sql() outputs quoted are the real run results

Frequently Asked Questions

Is Daft free?

Yes. Daft is open source under the Apache 2.0 license — the README states "Daft has an Apache 2.0 license," so you can use it in commercial projects at no cost. There is no paid tier of the library itself.

What Python versions does Daft support?

Python 3.10 or newer, per the install docs. The current stable release at the time of writing is v0.7.26, uploaded to PyPI on Oct 6, 2026 — a smaller release applying a dependency upgrade ahead of the larger v0.8.0.

Is Daft the same as Polars?

No — say it once and clearly. Polars is the fast single-machine dataframe; Daft is the distributed and multimodal engine with a lazy plan and a Ray runner. Same family, different job.

How do I run Daft on a cluster?

Install the Ray extra with pip install -U "daft[ray]", then call daft.set_runner_ray("ray://your-cluster:10001") — with no address, Daft spins up a local Ray cluster on your machine. Your DataFrame code stays exactly the same.

Can Daft read my Delta Lake tables?

Yes. Install pip install -U "daft[deltalake]" and read with daft.read_deltalake("some-table") (current spelling — the older read_delta_lake from old blog posts no longer exists). Iceberg tables work too, via daft.read_iceberg(table) with PyIceberg.