Streams and files, together
Bring streams, database changes, HTTP events, and files into one product and operating system.
Turn live data into immediately queryable tables
Bring high volume live data from databases, streams, applications, and files into your lakehouse. Shape and enrich records as they arrive and deliver fresh data near-instantly to open table formats.
See how the ingestion engine fits your data stack.
Why the ingestion engine
Replace three tools with one. Get usable tables, faster.
Replace separate connector services, transformation jobs, and table writers with one product.
Shape data as it arrives
Filter, join, aggregate, enrich, and reshape records without waiting for a separate transformation job.
Output for the rest of the platform
Write to open tables that your existing tools can use.
Proven on 1M+ events / sec
Production throughput on a single pipeline, with headroom for bursts and volume spikes.
Data movement
Connect sources. Push events. Publish results.
The ingestion engine reads from connected sources, accepts events from applications, and writes the results to tables, streams, or files.
- ReadRead database changes, topics, streams, and files in batches.
- PushSend webhooks and application events directly over HTTP or gRPC. No broker required.
- PublishWrite processed data to supported tables, streams, and files.
Data movementThree paths · one runtime
Connected sourcesCDC · streams · files
The engine readsfrom sources
Applications pushHTTP · gRPC · webhooks
The engine acceptsevents
The engine processesRecords
Tables
Streams
Files
Streams
Files
Built-in CDC
Many databases. Less CDC infrastructure.
Replicate data from hundreds of MySQL and PostgreSQL databases into your lake. The ingestion engine captures existing data, keeps tables up to date as records change, and sends each table to the right destination.
- ReplicateMany tables from many databases in a single pipeline.
- ReuseThe same database changes feed multiple pipelines.
- DirectDatabase to Iceberg, without Debezium and Kafka in between.
- One serviceConnect databases without a separate CDC service for every pipeline.
CDC data pathMany databases · one CDC path
PostgreSQL · orders
MySQL · customers
PostgreSQL · inventory
The ingestion engine
Built-in CDC
From change log to table
01Capture
02Track progress
03Route
Snapshot first
changes continuously
orders
customers
inventory
Continuous processing
SQL for the common path. Specialized processing when you need it.
Transform data with SQL or call an external service without taking it out of the live pipeline.
orders.sqlRunning
INSERT INTO live_orders SELECT customer_id, sum(total) AS spend FROM order_events GROUP BY tumble(event_time, '5 min');
Continuous SQL
Transform before the data lands
Filter, join, aggregate, and reshape records as they arrive, including event-time windows and stateful updates.
User-defined functions
Extend the pipeline with your own logic
Use UDFs for custom transformations and enrichments when built-in SQL is not enough.
Built-in table management
Keep streaming tables query-ready
The ingestion engine keeps open lakehouse tables organized as new data arrives, without a separate maintenance job.
- Partition and organize files as they are written.
- Keep tables healthy as data grows.
- Keep data ready for the query engines you already use.
Table maintenance cycleAutomatic · continuous
Records Arrive
Live records
Small files accumulate
Records become files
Compaction Occurs
Files combine automatically
Table Made Query-Ready
Fewer, larger files
ResultQueries scan fewer, larger files.
Runtime and deployment
Place the ingestion engine in your data environment
Run the ingestion engine in the environment and compute model that fits your stack.
01
Where can it run?Broad environment support
AWS
Google Cloud
Azure
Private infrastructure
02
What the ingestion engine needs to run?Kubernetes · Object Storage
K8S
Kubernetes
Store
Object Storage
03
What stays consistent?Same runtime behavior everywhere
01
Distributed execution
Spread pipeline work across available machines.
02
Scale pipeline parallelism
Increase throughput by running more work in parallel.
03
Resume after failures
Resume processing without starting over.
Run close to your sources, storage, and catalogs.
Integrations
See where the ingestion engine fits
The engine works with the systems, storage, and catalogs already in your data platform.
Integration topologyConnect · process · deliver
Sources
PostgreSQL
MySQL
Kafka
Confluent
Kinesis
HTTP
gRPC
Files
The ingestion engine
One live
data path Connect
Process
Deliver
data path Connect
Process
Deliver
Outputs
Apache Iceberg
Delta Lake
Kafka
Files
Storage and catalog foundation
AWS S3AWS GlueGoogle Cloud StorageBigLakeAzure BlobREST catalogsHive Metastore
Before you decide
Ingestion Engine Questions
Common questions about adding the ingestion engine to the data stack you already have.
Do we need to replace our current stack?+−
No. Start with one pipeline and keep the tools you already use. Move more workloads to the ingestion engine when you are ready.
Is the ingestion engine a database or query engine?+−
Neither. The ingestion engine gets live data ready and sends it where it needs to go. Keep using your existing query, analytics, and BI tools.
Does the ingestion engine use its own storage?+−
No. Your data lands in your storage and open table formats, where other tools can use it directly.
Does our data have to leave our environment?+−
No. The ingestion engine runs inside your cloud or private environment. It processes data there and sends results only to destinations you configure.
Does the ingestion engine replace Kafka?+−
It can remove Kafka from pipelines where Kafka is only moving data from one system to another. Keep Kafka where it is already the shared event backbone.
Does the ingestion engine replace dbt, Spark, or Flink?+−
The ingestion engine can take over live transformations that currently need separate jobs. Keep those tools for batch work, analytics, or other jobs they already handle well.
What will our team still need to operate?+−
You still own your source systems, storage, and analytics tools. The ingestion engine handles the live data path between them.
How can we evaluate the ingestion engine?+−
Pick one real source and one destination. We’ll build it with your data so you can judge the speed, effort, and fit for yourself.
Book a demo on your own workloads
Reach out to book a demo, share challenges you're facing, and tell us how this fits into what you're currently working on or thinking about.
Prefer to self-serve? Problems we're solving →