Data Integration Patterns
Data integration is not a single pipeline shape. Some workloads periodically copy a complete dataset; others forward only new or changed records, react to incoming events, keep applications synchronized, or consolidate many sites into a central hub. DataZen provides reusable patterns for each of these needs so teams can start from a proven design instead of rebuilding capture, state, retry, and delivery logic for every project.
These patterns apply to traditional data-lake and warehouse ingestion as well as operational integration, business process automation, and remote-site replication. They support batch, near-time mini-batch, and real-time processing and can be combined as requirements evolve. For example, a pipeline may use a watermark to limit each read, synthetic CDC to remove unchanged rows, and one-way synchronization to deliver the resulting change set to several targets.
Move the data available during the current execution without retaining capture state between runs.
Remember a watermark, prior state, or time window so each execution can select the correct records.
Combine capture and delivery patterns to implement synchronization, aggregation, and resilient architectures.
Choosing a Pattern
Start with the source capability and the delivery guarantee you need. Use a snapshot when a full copy is inexpensive, a watermark when the source exposes a reliable increasing value, synthetic CDC when it does not expose native changes, and a CDC stream when the source already provides them. Composite patterns then define what happens to those records across one or more systems.
| Category | Pattern | Flow | When to use it |
|---|---|---|---|
| Stateless | Snapshot |  |
Capture an entire source dataset, optionally with a filter. Best for initialization, small datasets, and periodic full refreshes. |
| Stateless | Hook |  |
Receive data through a listener or webhook and immediately validate, transform, and forward the incoming payload. |
| Stateless | CDC Stream |  |
Consume and forward changes already produced by a database, message broker, or other native CDC engine. |
| Stateful | Watermark |  |
Read forward from the last successful high-watermark value for efficient incremental mini-batches. |
| Stateful | Synthetic CDC |  |
Compare source records with retained state and forward only inserts, updates, or deletes when native CDC is unavailable. |
| Stateful | Window Capture |  |
Re-read an overlapping time window to capture late-arriving or corrected records safely. |
| Composite | Watermark + CDC |  |
Limit the source read with a watermark, then remove unchanged records with CDC before downstream processing. |
| Composite | One-Way Sync |  |
Keep one or more targets aligned with a system of record while changes flow in one direction. |
| Composite | Two-Way Sync |  |
Synchronize systems that can each originate changes, with identity mapping and conflict handling defined explicitly. |
| Composite | Aggregation |  |
Normalize and centralize data from many systems or remote locations for monitoring, reporting, analytics, and AI/ML. |
Patterns Are Building Blocks
The table describes common starting points, not rigid products. A production pipeline can combine patterns and apply inline ETL, data-quality rules, schema normalization, cloud functions, or AI agents at any stage. Choose the smallest pattern that provides the required correctness, then add state and composition only when the source behavior or business outcome requires it. This keeps implementations easier to operate and change.