ZenithDB: Best AI Observability Databases for Agent Teams in 2026

What Is ZenithDB?

ZenithDB is a Rust-built, open-source columnar database from Polarity Inc. for AI agent traces, and ZenithDB is one of the best AI Observability Databases tools for AI platform teams, observability engineers, and agent-framework builders. It targets long, sparse, high-cardinality JSON spans and ships with HTTP, gRPC, and OTLP endpoints, plus SQL and ZenithQL. The repo currently spans 18 Rust crates and is still labeled alpha as of Feb 2026, so the core engine is real while the wire format can still move.

The practical pitch is simple: existing observability backends are expensive on this workload because they are optimized for short spans and shallow schemas. ZenithDB keeps trace data cheap to store and fast to scan by sorting segments on (trace_id, start_time, span_id), keeping trace-locality during compaction, and postponing wide-column decoding until filters have already cut the candidate set down.

Quick Overview

Who Should Use ZenithDB?

• Platform teams running agent-heavy products that emit thousands of sparse spans per request and need a store that does not choke on wide JSON payloads.
• Observability engineers who want SQL over traces, metadata, and annotations without copying the data into a separate warehouse first.
• Founders building AI tooling who need a local-first dev stack now and a path to S3, GCS, or Azure object storage later.
• Teams standardizing on OTLP and wanting one backend for ingestion, query, and index-heavy trace search instead of stitching together multiple systems.

Not ideal for:

• Teams that need a battle-tested production database today with frozen wire formats and long-term vendor guarantees.
• Workloads that are mostly logs or metrics and do not need trace-local compaction or sparse-column optimization.
• Shops that already standardized on a broad analytics lake and only need occasional trace lookup, not a dedicated trace store.

Key Features of ZenithDB

• Trace-local PAX segments — ZenithDB stores rows in a PAX-style segment layout and sorts by (trace_id, start_time, span_id), which keeps spans from one trace together during compaction. That design cuts scatter across the storage layer and makes trace-level filters far cheaper than a generic row store.
• Late materialization in scans — ZenithDB does not decode wide string columns until a row survives earlier predicates. For AI traces with large attributes maps, that means the engine avoids wasting CPU on rows that were already rejected by time, trace, or tag filters.
• Embedded full-text search — ZenithDB embeds Tantivy directly inside segments instead of outsourcing search to a sidecar index. That keeps keyword search over trace names, notes, and annotations close to the data and reduces index drift during compaction.
• Object-storage WAL — ZenithDB writes its WAL to object storage with conditional PUT semantics, then merges it with compacted segments at query time. That pattern fits S3-compatible backends and keeps the ingest path queryable as soon as the write is acknowledged.
• Multiple query surfaces — ZenithDB speaks SQL and ZenithQL, so teams can query with familiar relational patterns while still having a domain-specific language for trace-aware operations. The same engine also exposes REST, gRPC, and OTLP entry points for agents and collectors.
• Local-first development profile — ZenithDB can run with an in-memory MockCatalog and local filesystem object store, which means you can test ingest and query loops on a laptop without Docker or Postgres. That makes the first mile of development much cheaper than a distributed observability stack.
• Minimal Next.js console — ZenithDB ships with a web console that shows live segments, queries, compactions, WAL metrics, and a query runner. The console is intentionally thin, so the dashboard stays tied to real engine state instead of mocked fixture data.

ZenithDB vs Alternatives

Pick ClickHouse when your workload looks like standard analytics, because it is the safer default for broad SQL reporting and has a deeper production track record. Pick OpenSearch when the main job is fuzzy text search across event payloads and you care more about retrieval than columnar scan efficiency.

Pick Grafana Loki when your team mostly needs log retention and label filters, not deep JSON analytics or trace-local joins. Use ZenithDB instead when the data model is trace-centric, the rows are sparse, and the dominant cost is decoding too many unused columns.

ZenithDB also pairs well with OpenSwarm when you want agent orchestration on one side and trace storage on the other. If you need a lighter trace browser before committing to a new datastore, OpenTrace is the more minimal choice; if traces have to land in a broader analytics pipeline, DataHaven is the better companion.

How ZenithDB Works

ZenithDB uses a split architecture: a gateway built on axum and tonic, a writer that buffers into a memtable, a querier that combines planner and execution, a catalog backed by Postgres in production, and object storage for WAL and segment files. The central storage abstraction is a .zseg segment that holds compressed columns, per-column metadata, and footer information needed for fast pruning.

The storage design is opinionated. ZenithDB sorts segments by trace identity and time, then enforces trace-locality during compaction so one trace lands in one row group whenever possible. That matters because AI agent traces often include many small spans, late annotations, and nested attributes, so query latency falls more from avoiding needless decode work than from raw sequential throughput.

A typical first run looks like this:

curl -fsSL https://raw.githubusercontent.com/Polarityinc/zenith/main/install.sh | sh
zen serve --config examples/zenithdb.dev.toml
curl -s localhost:8080/v1/query

The installer drops the zen binary into your local bin directory, and zen serve starts HTTP on :8080 plus gRPC on :50051. On the default dev profile, ZenithDB uses an in-memory catalog and local filesystem object store, so you can test ingest, compaction, and query flows without provisioning Postgres or a bucket.

Pros and Cons of ZenithDB

Pros:

• Trace-aware storage layout reduces scan waste on sparse AI telemetry and makes segment pruning much more effective than generic row storage.
• Late materialization avoids decoding large string and JSON payloads for rows that are already filtered out.
• SQL plus ZenithQL gives teams a familiar query path and a domain-specific path for trace operations.
• Inline search indexing keeps text search closer to the segment, which reduces index drift and operational sprawl.
• Local dev mode works with no Docker, no Postgres, and no external cloud services for the first smoke test.
• Multi-protocol ingestion via HTTP, gRPC, and OTLP makes it easier to sit behind existing agent and collector pipelines.

Cons:

• Alpha status means on-disk formats and wire protocols can still change before 1.0.
• Production setup still assumes Postgres for the catalog, so it is not fully self-contained at scale.
• Rust toolchain requirements and protoc setup add friction for teams that expect a prebuilt binary for every environment.
• Narrow workload focus means it is less attractive as a general-purpose analytics engine than ClickHouse.
• Minimal console is useful for operators but not a full observability suite with dashboards, alerting, and long-term reporting.

Getting Started with ZenithDB

The fastest path is to install the zen binary and boot the dev config, because ZenithDB is designed to run locally before you wire it into a real bucket or catalog. If you need a source build, use Rust 1.87+ and protoc 3.21+.

# macOS and Linux installer
curl -fsSL https://raw.githubusercontent.com/Polarityinc/zenith/main/install.sh | sh

# start the local server
zen serve --config examples/zenithdb.dev.toml

# build from source if needed
git clone https://github.com/Polarityinc/zenith.git
cd zenith
cargo build --release
cargo run --release -p zen_cli -- serve --config examples/zenithdb.dev.toml

After startup, ZenithDB listens on HTTP :8080 and gRPC :50051, with data landing under ./data/ by default. The first config decision is whether to keep the default in-memory catalog for dev or switch to Postgres for production-like testing, and the second is which object store backend you want to target.

Verdict

ZenithDB is the strongest option for AI agent observability when you need cheap storage for long, sparse traces and want to query them with SQL. Its biggest win is the trace-local columnar engine, while the main caveat is alpha-stage format churn. Choose ZenithDB if you can accept that trade-off and need a purpose-built trace database.