Skip to content

index.mdx

Latest commit

 

History

History
73 lines (60 loc) · 2.94 KB

index.mdx

File metadata and controls

73 lines (60 loc) · 2.94 KB
Raw
title Introduction
description Declarative streaming data pipelines for ClickHouse.

StreamBuild compiles typed SQL models into ClickHouse tables and materialized views, plans their dependencies, and rebuilds them from retained streaming history.

StreamBuild lineage graph from a Kafka source through ClickHouse models

Each pipeline has an effective operational mode:

  • Direct mode is the default. stb build rebuilds selected models at their ordinary names and replays retained history without a publish step.
  • Virtual environments are an explicit opt-in. stb build creates deployment-specific tables, stb deployment audit assesses them, promotion switches stable views, and retained publications can be compared or rolled back as a complete graph.
  • Mixed builds stage selected virtual pipelines first, then apply selected direct pipelines in the same CLI invocation. Relationships cannot cross the mode boundary.

What StreamBuild manages

  • Strict, explicitly typed MODEL() SQL
  • Managed Kafka landing through ClickHouse's Kafka engine
  • Adopted ClickHouse stream tables that StreamBuild does not own
  • Materialized-view dependency graphs with driving and side-reference edges
  • Inclusive, at-least-once replay across offset, timestamp, landed-at, and cursor boundaries
  • SQL-native model tests, macro tests, and data-quality audits
  • Single-process scheduled audits using the warehouse clock
  • A local UI for plans, runs, deployments, lineage, Kafka lag, topics, and retained source messages
  • Offline compile artifacts for review and automation

StreamBuild currently supports ClickHouse only. It does not provide dbt compatibility, authored seeds, UDF management, or a generic multi-warehouse abstraction.

Choose a workflow

Build a small adopted-source pipeline against local ClickHouse. Understand destructive rebuilds, replay, failure, and rerun behavior. Stage, inspect, assess, compare, promote, roll back, diagnose, and clean deployments. Explore lineage, sources, messages, plans, quality, and durable runs. Run the checked-in Kafka-to-ClickHouse direct-mode example.

Core loop

stb discover
stb compile
stb plan
stb build
stb test
stb audit
stb dev

Run commands from anywhere inside a project, or pass --project-dir explicitly.

stb build already runs applicable audits. Use the separate stb audit command when you want to rerun live checks later without rebuilding.