Skip to content

Home

Jump to bottom
aeluce edited this page Oct 3, 2025 · 1 revision

Welcome to Flow!

Estuary Flow is a unified data platform: use pre-built pluggable connectors to quickly assemble pipelines, work with both real-time and batch data, and automatically evolve schemas as data changes.

While this wiki covers some Flow basics, you'll find the most comprehensive information on using Flow in our docs.

What Is Flow?

Estuary Flow supports flexible, yet simple, data movement for data your way.

  • Real-time: Millisecond latency with CDC-powered captures and optimized materializations
  • Batch: Predictably sync data on your own schedule with options for faster syncs during defined timeframes
  • ETL: Transform data before it lands in your destination(s):
    • Provide simple schema updates, like alternate names or field selection, through the UI
    • Implement more complex transformations using SQL or TypeScript
  • ELT: Transform data after load with a dbt Cloud integration

Learn more about Estuary's features and how it stacks up against comparable solutions.

How to Use Flow

You have options when using Flow. Estuary Flow comes with both dashboard and CLI components so you can manage pipelines through the UI or programmatically.

The Flow service also comes with several deployment options:

  • Public SaaS: Fully managed by Estuary
  • Private deployment: Enhanced security with your own private data plane
  • BYOC (Bring Your Own Cloud): Use existing cloud infrastructure paired with Estuary's native capabilities

These options allow you to balance simplicity with customization and security.

The Flow runtime is also available under a Business Source License. See the license if you are interested in self-hosting.

Set Up Your First Pipeline

You can try out Estuary and set up your first pipeline for free. When you register, you will set up your tenant, a unique namespace for your pipelines.

Pipelines consist of several parts, which you can mix and match to form one-to-many or many-to-many connections:

  • Captures: Captures discover data from source systems (databases, logs, SaaS systems, and more) and save them to durable data collections.
  • Collections: Centralized data collections representing a table, API responses, or other grouping of data. Can also be transformed into derived collections.
  • Materializations: Send data collections to destination systems with materializations.

You can create and manage each of these parts individually, either through the flowctl CLI or through the dashboard UI. The dashboard allows you to select and authenticate connectors (for captures or materializations), automatically building a configuration file based on your choices.

Find step-by-step tutorials for common pipelines:

On initial registration, you're automatically enrolled in Estuary's trial bucket, which securely stores data and recovery logs. This lets you quickly start testing out the platform and connectors.

For production use cases, you will likely want to switch to your own storage bucket.

About this Repository

Most Estuary Flow users can manage their integrations and data movement through Estuary's hosted SaaS offering without digging into the underlying technical structure. However, reviewing Flow in GitHub may be of interest if you want to compare the technical architecture with other solutions or are considering self-hosting Flow following the BSL.

This repository in particular forms the backbone of the Flow service and releases the flowctl CLI. To help you navigate the flow repo, here are some files and directories of note:

  • CLAUDE.md: While intended to provide context for AI tools, this document is a swift overview of the repo and associated commands
  • control_plane_readme.md: Describes components of Flow's control plane with tips for development
  • crates/: Rust crates that supply APIs, like control-plane-api, or provide packages, like flow-web
  • go/: Integration code between Flow and Gazette
  • examples/: Examples on working with Flow, particularly around derivations and reductions
  • site/: Documentation site (served at https://docs.estuary.dev/)

You can find related code in these additional repos:

  • estuary/ui: UI for the Estuary Flow dashboard
  • estuary/connectors: In-house capture and materialization connectors for databases, SaaS systems, blob storage, and more
  • gazette/core: Underlying broker logic for Estuary Flow, allowing the service to mix real-time and batch data using durable journals

Resources and Support

You can learn more about Estuary Flow with these resources:

For support:

Clone this wiki locally