rivers: Best Asset Orchestration for Data Engineers in 2026

What Is rivers?

rivers is an asset orchestration platform built by Ion Elgreco. It lets data engineers and platform teams define pipelines in Python while its Rust control plane resolves the dependency graph and plans execution without a Python interpreter on the control plane. rivers is one of the best Asset Orchestration Platforms tools for data engineers and platform teams. The project claims sub-millisecond plan times on graphs with thousands of nodes, which is the number that matters when your DAG stops being toy-sized.

Quick Overview

Who Should Use rivers?

• Data platform engineers who need deterministic DAG planning, partition-aware execution, and a local dev loop that starts the scheduler and UI in one command.
• Analytics engineers modeling datasets as assets instead of opaque tasks, especially when downstream dependencies need to be inferred from Python call signatures.
• Platform teams on Kubernetes that want a CodeLocation deployment model, digest-pinned images, and operator-driven reconciliation.
• Teams with mixed Python and infrastructure constraints who want Python for definitions but Rust for the hot path.

Not ideal for:

• Teams that depend on the Airflow plugin ecosystem or a long tail of legacy operators.
• Simple cron-only jobs that do not justify asset modeling, SurrealDB, and a dedicated scheduler.
• Organizations that are unwilling to run SurrealDB or Kubernetes when they need production distribution.

Key Features of rivers

• Asset-based graph inference — define assets as Python functions, and rivers infers the dependency graph automatically from function signatures and asset composition.
• Rust scheduler and planner — graph resolution, partition mapping, and execution planning happen in compiled Rust, which is why large graphs stay under sub-millisecond planning time in the project’s own benchmark claims.
• Partition strategies — supports static, daily, hourly, custom cron windows, multi-dimensional partitions, and dynamic partitions for runtime-extensible datasets.
• IO handler matrix — built-in in-memory, pickle, and Delta Lake handlers; Delta support includes merge semantics, and the optional pyarrow and polars extras wire in dataframe-oriented types.
• Parallel and Kubernetes execution — Executor.parallel() fans out concurrent subprocess workers, while Executor.kubernetes() schedules one pod per step for cluster-native runs.
• Automation primitives — schedules, sensors, dep-aware triggers, and backfills are first-class, so recurring materializations do not need a separate orchestration layer.
• Single-binary local mode — rivers dev <module> starts embedded SurrealDB with RocksDB, the scheduler, and a web UI on :3000, which shortens the feedback loop for new pipelines.

rivers vs Alternatives

Pick Dagster if you want a mature asset orchestration workflow with more public examples and a larger ecosystem. Pick Prefect if Python ergonomics matter more than the orchestration runtime internals and you want a lighter authoring experience.

Pick Airflow if your team already has DAG operators, established on-call playbooks, and a lot of existing code. If your work is more about deployment automation than asset materialization, compare rivers with djevops. If you need adjacent data lifecycle tooling around storage and retrieval, DataHaven is a better companion than a replacement.

How rivers Works

rivers separates declaration from execution. Python defines assets and repository wiring, but Rust owns the control plane: graph resolution, partition mapping, run planning, and scheduler decisions. That design removes the Python interpreter from the coordinator path, so orchestration logic does not pay import-time latency or GIL contention when it is making decisions about the DAG.

The runtime stores state in SurrealDB, with RocksDB embedded for local development and a server deployment path for production. The UI is also compiled Rust, using Leptos SSR and WASM on axum, and it streams state changes via Server-Sent Events straight from the database. The Kubernetes operator adds a deployment layer on top of that model by reconciling CodeLocation objects into workloads that serve the Python module in cluster-native form.

import rivers as rs

@rs.Asset
def raw_data():
    return {'users': 100, 'events': 5000}

@rs.Asset
def summary(raw_data: dict):
    return '{} users, {} events'.format(raw_data['users'], raw_data['events'])

repo = rs.CodeRepository(assets=[raw_data, summary])
result = repo.materialize()
print(repo.load_node('summary'))

This example defines two assets, wires them into a repository, and materializes the graph without hand-rolled dependency plumbing. In a real project, rivers resolves the asset graph, schedules the work, and persists state so the next run can use partition-aware backfills or incremental materialization.

Pros and Cons of rivers

Pros:

• Rust handles the expensive orchestration work, so graph planning stays fast even as the DAG grows.
• Asset inference from Python functions cuts down on manual dependency wiring.
• Partitioning is unusually broad, covering time windows, multidimensional keys, and dynamic partitions.
• Kubernetes support is built in through an operator and CRDs, not bolted on later.
• The single-binary dev workflow reduces local setup time for new contributors.
• Delta Lake and dataframe extras make it easier to fit into modern analytics stacks.

Cons:

• The ecosystem is smaller than Dagster or Airflow, so you will find fewer community examples and third-party operators.
• Production use depends on SurrealDB and, for full distribution, often Kubernetes, which adds operational surface area.
• Teams that want a pure task-queue mental model will need to adapt to the asset-first abstraction.
• The scraped page does not expose a stable release tag or license badge, which makes procurement checks slower.
• Some IO paths require optional extras, so pip install rivers alone may not cover every data format you care about.

Getting Started with rivers

pip install rivers
pip install rivers[delta] rivers[pyarrow] rivers[polars]
rivers dev pipelines.analytics

The first command installs the base runtime, and the optional extras add Delta Lake plus dataframe support. When you run rivers dev, the platform boots embedded state, the scheduler, and the local UI on :3000, so you can inspect the asset graph before you push anything to Kubernetes.

Verdict

rivers is the strongest option for Python-defined asset orchestration when you want Rust-backed planning, Kubernetes-native deployment, and a one-command local dev loop. Its best strength is the split between declarative Python and a compiled scheduler; the caveat is a smaller ecosystem than Dagster or Airflow. Choose rivers if runtime architecture matters more than plugin count.