How to Use Numaflow: A Beginner's Guide to Stream Processing on Kubernetes

CodexFuse Team · 2026-11-02T11:59:56.111Z · 14 min read min read

Numaflow is a Kubernetes-native platform for stream processing. This beginner's guide covers installing it, declaring your first pipeline, writing a Python UDF, reading from Kafka, and autoscaling on back-pressure.

Learning how to use Numaflow is the fastest way to run stream processing on Kubernetes without standing up a Flink or Spark cluster. Numaflow is a Kubernetes-native, serverless platform for running scalable and reliable event-driven applications: you declare data pipelines as YAML custom resources, and the platform handles the buffers, scaling, and plumbing between stages. If you can write a Kubernetes manifest and a small Python function, you can ship your first streaming pipeline today. This numaflow tutorial walks the full beginner arc: installing the controller, declaring your first numaflow pipeline, the vertex lesson, writing your first numaflow python udf, reading from a numaflow kafka source, and riding the numaflow autoscaling curve — with every fact checked against the project's own docs on the day of writing. New to the platform underneath it all? Start with our Kubernetes beginner guide first, then come back — Numaflow assumes a working cluster.

What Is Numaflow, Exactly?

Numaflow is a Kubernetes-native, serverless platform for running scalable and reliable event-driven applications. That is the project's own one-line definition, and it is precise: "serverless" here means you do not manage the streaming infrastructure yourself — you declare what the pipeline should do, and Numaflow's controller reconciles the running state.

The project was created by the Intuit Argo team to address community needs for continuous stream processing, and it now lives under the numaproj organization on GitHub. Its tagline reads "Kubernetes-native platform to run massively parallel data/streaming jobs," it is licensed Apache-2.0, and at the time of writing the repository shows 2,837 stars and 183 forks — a snapshot that moves daily, but the direction has been steadily up. The latest release is v1.9.0.

Think of Numaflow as the missing middle layer between your event sources and your business logic: Kafka holds the events, your functions transform them, and Numaflow is the Kubernetes-native fabric that connects the two with buffers, ordering, watermarks, and autoscaling built in.

How Does Numaflow Work? The Mental Model

You declare a pipeline as a Kubernetes custom resource; Numaflow runs each processing stage as a vertex, moves data between vertices through inter-step buffers, and autoscales each vertex on back-pressure. That sentence is the whole mental model — everything else is detail.

A pipeline has three kinds of stages. A source vertex ingests events (from Kafka, from a generator, from Pulsar, and others). UDF vertices run your code — map for one-in-one-out transforms, reduce for windowed aggregation. A sink vertex emits results (to Kafka, to logs, to a black hole for testing, or to your own sink). Data flows between vertices through inter-step buffers, which give you durability and back-pressure handling for free: when a downstream vertex falls behind, the buffer fills and upstream stages feel the pressure instead of dropping data.

One line to keep you out of trouble: Numaflow is a stream processor, not a message broker and not a job orchestrator — it consumes from streaming sources like Kafka or Iggy rather than replacing them, and it processes live event streams rather than orchestrating batch DAG jobs the way Kestra does. Different layers, no rivalry.

How Do You Install Numaflow on Kubernetes?

Create the numaflow-system namespace and apply the install manifest, then deploy the JetStream-based inter-step buffer service once per namespace. The standard numaflow install kubernetes path is the two commands below — two kubectl commands get the platform running; the buffer service is the step beginners skip.

kubectl create ns numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/main/config/install.yaml

That manifest installs the controller as a Deployment named numaflow-controller in the numaflow-system namespace (its container is named controller-manager — not numaflow-controller-manager, which is the name half the internet's older blog posts use). Verify it is up with kubectl -n numaflow-system get deploy numaflow-controller.

For anything beyond a sandbox, pin a release instead of tracking main: https://github.com/numaproj/numaflow/releases/download/v1.9.0/install.yaml, with an optional validating-webhook-install.yaml alongside it. The manifest used to be called numaflow.yaml in older guides; the current official name is install.yaml.

Before you create any pipeline, install the inter-step buffer service — once per namespace, shared by every pipeline in that namespace:

kubectl apply -f https://raw.githubusercontent.com/numaproj/numaflow/main/examples/0-isbsvc-jetstream.yaml

Need somewhere to run the cluster itself? A single VPS is enough for learning — see what VPS hosting is before you pick a machine.

What Is a Numaflow Pipeline?

A Pipeline is a Kubernetes custom resource — apiVersion: numaflow.numaproj.io/v1alpha1, kind: Pipeline — that lists vertices, the processing stages, and edges, the from/to relationships that say how data flows between them. If you have written any Kubernetes YAML, the shape will feel familiar.

The project's own simplest example declares three vertices — in, cat, out — and two edges connecting them. It is short enough to read in full, and it is the canonical first pipeline:

apiVersion: numaflow.numaproj.io/v1alpha1
kind: Pipeline
metadata:
  name: simple-pipeline
spec:
  vertices:
    - name: in
      scale:
        min: 1
      # A self data generating source
      source:
        generator:
          rpu: 5
          duration: 1s
    - name: cat
      scale:
        min: 1
      udf:
        container:
          image: quay.io/numaio/numaflow-go/map-cat:stable # A UDF which simply cats the message
          imagePullPolicy: Always
    - name: out
      scale:
        min: 1
      sink:
        log: {}
  edges:
    - from: in
      to: cat
    - from: cat
      to: out

Read it top to bottom: the in vertex generates 5 messages per second, the cat vertex runs a user-defined function that passes each message through, and the out vertex logs everything. Each vertex carries its own scale block — minimum replicas here, full autoscaling config when you are ready. Apply it with kubectl apply -f, then watch it with kubectl get pipeline (short name pl).

Keep this manifest in version control and apply it from CI — our GitHub Actions beginner guide shows the pattern for automating kubectl applies on every push.

Pipeline vs MonoVertex: What's the Difference?

A MonoVertex is the simplest way to run Numaflow: a single vertex that reads from a source, optionally transforms the data, and writes to a sink. It needs no Inter-Step Buffer Service and has no edges. That is the vendor's definition, almost word for word — and it tells you exactly when to reach for it.

MonoVertex is its own custom resource (kind: MonoVertex, same v1alpha1 API version): source, optional transformer, and sink collapsed into one pod. If you search for numaflow monovertex examples, you will keep finding this same three-part shape — source, optional transformer, sink — because the CRD only models one vertex. It is the fastest thing to get running, and everything you learn there — sources, sinks, transformers, UDF containers — carries directly over to full pipelines, because both run the same components under the hood.

One caution the docs state plainly: windowed aggregation — grouping messages over time and reducing them — is a pipeline-only feature. A single-vertex MonoVertex cannot do it. And keep the hierarchy straight: MonoVertex is a feature, not the whole product. Graduate to a Pipeline when you need multiple processing stages, branching, joins, or windowed reduce.

What Is a Vertex in Numaflow?

A vertex is one processing stage of a pipeline: a source that ingests, a map or reduce UDF that transforms, or a sink that emits. Vertices are the nouns of Numaflow; edges are the verbs.

Source vertices ingest events — Kafka, Pulsar, a built-in generator for testing, HTTP, and more. A source can carry a transformer: your code running as a sidecar container inside the source vertex pod, talking to the platform over gRPC on a Unix domain socket, so you can reshape or filter events before they ever enter the pipeline.

Map vertices run your function once per message. Reduce vertices aggregate groups of messages over time windows — keyed or not, with fixed, sliding, or session windows — and because reduce needs persistence across restarts, you configure storage (a persistent volume claim) and parallelism via partitions instead of autoscaling. Sink vertices emit results to Kafka, Pulsar, the log, a black hole (for load testing), or your own user-defined sink.

How Do You Write a Python UDF for Numaflow?

A numaflow python udf is a container that runs your function behind a gRPC server; the pynumaflow library (0.9.0) wires the server for you. You write one handler function, the library serves it, and Numaflow calls it once per message.

The handler signature is fixed — keys, one datum in, a batch of messages out:

from typing import List
from pynumaflow.function import Messages, Message, Datum, Server

def my_handler(keys: List[str], datum: Datum) -> Messages:
    messages = Messages()
    # transform datum.value here, then append results
    messages.append(Message(b"hello", keys=keys))
    return messages

grpc_server = Server(map_handler=my_handler)
grpc_server.start()

Ship it as a container image: the project ships a sample UDF Dockerfile under its examples (a Python base image plus pip install pynumaflow — check the exact base tag at build time, since it moves), then reference the image in your vertex spec under udf.container.image. Beyond the basic map handler, pynumaflow offers MapServer and MapAsyncServer, plus MapT, Map Stream, Batch Map, Accumulator, UDSource, and UDSink for the fancier shapes.

Two habits that pay off immediately: keep the handler code versioned alongside your pipeline manifests — our Git beginner guide covers the workflow — and remember that every UDF is just a container image and remember that every UDF is just a container image — our Docker beginner guide covers the build-and-push loop you will use for every deploy.

How Do You Read from Kafka in Numaflow?

Point a source vertex at Kafka with the KafkaSource fields: brokers, topic, and consumerGroup. Numaflow treats Kafka as an external source it consumes from — your broker stays exactly where it is.

That consumerGroup spelling is the single most common typo in beginner Kafka sources — the API reference names it consumerGroup, and consumerGroupName will fail validation. TLS and SASL blocks are optional and cover the usual enterprise setups, so a numaflow kafka source works against managed Kafka offerings as well as your own cluster.

How Does Numaflow Autoscaling Work?

Numaflow provides built-in autoscaling that monitors buffer depth and processing rates, making scaling decisions based on actual workload rather than generic CPU/memory metrics. That is the vendor's description, and it is the key difference from a plain HorizontalPodAutoscaler: Numaflow scales on how much work is waiting, not on how hot the CPU is.

Each vertex carries a scale block with min and max replicas, lookbackSeconds ("lookback seconds to calculate the average pending messages and processing rate"), zeroReplicaSleepSeconds, and targetProcessingSeconds ("used to tune the aggressiveness of autoscaling for source vertices"). Vertices can scale all the way to zero and wake when messages arrive, so an idle pipeline costs you nothing — and v1.9.0 added cron-based autoscaling for source vertices and MonoVertices, for workloads that follow a schedule rather than a queue.

Two boundaries to memorize: autoscaling is not supported on reduce vertices — define partitions there instead — and event-time correctness comes from watermarks, with allowedLateness covering data that arrives after its window closed.

Numaflow vs Flink: Which Should You Pick?

Pick Numaflow when your platform is Kubernetes and you want streaming declared as versioned YAML; the project's creators frame it against the Flink and Spark world they came from. Vigith Maurice, a Numaflow cocreator at Intuit, put it this way in an Intel interview: "Today, streaming is associated mostly with data engineers who use systems like Apache Flink and Apache Spark. We wanted it to be accessible to everyone — application developers, machine learning (ML) engineers, and DevOps."

That quote is the honest basis for the comparison. Nobody at the vendor publishes a feature-matrix shootout, so here is a decision checklist built only from verified Numaflow facts:

Flink remains the incumbent for teams already invested in the JVM streaming ecosystem. Numaflow is the bet that your Kubernetes cluster can be your streaming platform too.

7 Beginner Mistakes to Avoid with Numaflow

Every row below comes from a real sharp edge in the docs or the API — learn them here instead of in production.

Sources

Every fact above was checked against these pages on the day of writing; versions, star counts, and release numbers move, so re-verify before you build.

• numaflow.numaproj.io — official documentation: quick-start (install, MonoVertex, pipeline guides), core concepts (pipeline, autoscaling), user guide (sources, transformers, reduce UDF, sinks), and the API reference (KafkaSource, scale fields)
• github.com/numaproj/numaflow — repository README (Intuit Argo team origins, tagline, Apache-2.0 license) and the releases page (v1.9.0 notes)
• pypi.org/project/pynumaflow/ — Python UDF SDK documentation (0.9.0: handler signature, server classes, Dockerfile sample)
• intel.com — "NumaProj: Kubernetes Project You Should Know About" (cocreator Vigith Maurice on Numaflow vs the Flink/Spark world)

Frequently Asked Questions

Is Numaflow a replacement for Apache Kafka?

No. Numaflow consumes from Kafka through its KafkaSource — brokers, topic, and consumer group — and processes the events downstream. Kafka still stores and transports the events; Numaflow is the compute layer that transforms them. You need both.

Do I need Kubernetes to run Numaflow?

Yes. Numaflow is Kubernetes-native to its core: a controller Deployment plus Pipeline and MonoVertex custom resources. There is no standalone binary mode — the install manifest targets a cluster, and every workload you declare is a Kubernetes object.

What languages can I write UDFs in?

Python (via the pynumaflow SDK) and Go are the documented paths, with map, reduce, source, sink, and transformer shapes. Because a UDF is just a container serving the UDF gRPC protocol, the platform is language-agnostic by design — any language that can serve gRPC over a Unix socket works.

What is the difference between a Pipeline and a MonoVertex?

A MonoVertex collapses source, optional transform, and sink into a single vertex with no buffer service and no edges — the simplest way to start. A Pipeline connects multiple vertices with edges through inter-step buffers, and it is the only place windowed aggregation (reduce) can run.

Is Numaflow free to use?

Yes. Numaflow is open source under the Apache-2.0 license — no license fees, no seat limits. Your only cost is the Kubernetes compute your own cluster consumes, which the scale-to-zero autoscaling keeps as low as your workload allows.