DataZen Documentation
DataZen User Guide

Pipeline Triggers

Triggers start or queue one or more pipelines after the current pipeline completes. Use them to sequence dependent pipelines, force-start writers after a reader, create processing loops, or multicast Change Logs to multiple targets.

Classic screen-based triggers are configured in DataZen Manager and are available for self-hosted agents. SQL CDC provides the TRIGGER operation for both cloud and self-hosted agents.

Trigger Conditions

Four conditions determine whether a configured trigger runs when the current pipeline completes:

Condition When the trigger runs
On Completion Whenever the pipeline completes, whether it succeeds or fails.
On Success (always) When the pipeline succeeds, whether or not it processed data.
On Success (with data) When the pipeline succeeds and processed data.
On Failure When the pipeline completes with an error.

When multiple pipelines are specified as triggers, their execution order is not guaranteed.

SQL CDC: TRIGGER vs. START

SQL CDC provides two ways to launch another pipeline. The important difference is when each operation is evaluated:

  • START is executed only when the script reaches that command. It can appear at different points in a script, including inside an IF block. If the script fails or exits before reaching the START command, the target pipeline is not started.
  • TRIGGER is evaluated for execution when the script completes. Its condition determines whether the target pipeline runs, including On Failure when an earlier operation caused the script to fail.

Use START when launching the next pipeline is part of the script’s immediate control flow. Use TRIGGER when the decision must be evaluated at script completion.

Looping

A trigger can call the same pipeline again. This pattern is useful when a pipeline processes a subset of records at a time and can fetch the next set automatically, such as by using a high watermark or a database column as a processing flag.

To avoid an infinite loop, use On Success (with data). The trigger stops firing when the pipeline no longer has records to process.

This SQL CDC example processes records in batches and triggers the same pipeline while data continues to be returned. The pipeline is named mypipelinename. 

-- Keep a high watermark and order by the modified date to keep getting the next batch of records progressively
SELECT * FROM DB [sql] 
  (SELECT TOP 250 name, modify_date FROM sys.tables WHERE modify_date > '@highwatermark' ORDER BY modify_date)
  WITH HWM 'modify_date';

-- if any records were returned, continue doing some work...
PRINT 'Working with the next @@rowcount records';

TRIGGER 'mypipelinename' ON SUCCESS_WITH_DATA;

Configuring Triggers in DataZen Manager

DataZen Manager Triggers tab with On Completion and On Success with data triggers
DataZen Manager: Triggers tab

The example defines two triggers: one On Completion and another On Success (with data). Use the add, edit, and delete actions to manage trigger definitions.

Temporarily disable all triggers for the pipeline by clearing Enable Job Triggers.

Related Topics