Runtime Variables

When a pipeline runs, DataZen automatically defines runtime variables that you can use in object names, scripts, HTTP URLs and payloads, data pipeline components, and SQL CDC operations. Some variables are available only during a read; others only during a write. Date and identity tokens are commonly used in target names, such as file names, to distribute output automatically.

Variable names are case-sensitive. Use the @ prefix when referencing them. For example: mydatabase.dbo.[target-tmp-@executionid]

Runtime variable names are reserved keywords. Creating an environment variable with the same name as a runtime variable is not supported and raises a warning. In the future this may become an error; that behavior is subject to change.

How Runtime Variables Differ from Environment Variables and DECLARE

Runtime variables are created automatically during job execution (for example @executionid or @highwatermark). They are read-only.
Environment variables are agent-level values you create and manage. Access them with #getvar() / #setvar() or SET. See Environment Variables.
Script parameters are declared in SQL CDC with DECLARE (for example DECLARE @lastUpdatedOn = '#getvar(lastUpdatedOn)'). They are script-scoped and separate from both runtime and environment variables.
A DECLARE @variable can be initialized from an environment variable of the same name. When that link is established, SET @variable updates both the script variable and the environment variable. In other words, script variables and environment variables may be linked using this approach.

Where to Use Runtime Variables

Runtime variables can be used wherever DataZen accepts dynamic names or expressions, including classic UI settings and SQL CDC scripts. They may also be combined with DataZen Functions.

Readers — HTTP URIs and payloads, database SQL commands, and other reader options. For example:
```
SELECT * FROM mytable WHERE timestamp > @highwatermark
```
Writers — target object names, payloads, and writer options. For example:
```
newcustomers-@executionid.json
```
Data pipelines — scripts and object names inside pipeline components or SQL CDC ETL. For example:
```
stage-@guid-@executionid.json
```

Initialization and finalization scripts — classic UI completion scripts, and SQL CDC PUSH DB scripts using ON_INIT and ON_COMPLETION (see PUSH DB). Finalization / completion scripts also have access to @status, which can be treated as a string ('success' / 'failure') or as an integer (1 / 0 respectively). For example:

if ('@status' == 'failure') BEGIN INSERT INTO warnings (jobkey, rundate, changelog, msg) VALUES ('@jobkey', '@rundate', '@packagefile', 'Error detected!') END
if (@status == 0) BEGIN INSERT INTO warnings (jobkey, rundate, changelog, msg) VALUES ('@jobkey', '@rundate', '@packagefile', 'Error detected!') END

Variable Reference

The following tables summarize runtime variables and where they are available. In SQL CDC and elsewhere, reference each name with the @ prefix (for example @executionid).

Common Variables

Variable	Description	Reader	Writer
`@agentid`	Cloud agent identifier currently executing the pipeline.	X	X
`@executionid`	Unix timestamp in milliseconds of the execution start time. Unique for a given job execution.	X	X
`@executionguid`	Execution session identifier (GUID) for the current pipeline run.	X	X
`@guid`	Unique GUID of the job. Remains unchanged unless the job is recreated.	X	X
`@ispreview`	`1` during preview operations; `0` when the pipeline is executing live.	X	X
`@jobkey`	Name of the job. Unique at any given time. If a job is deleted and recreated with the same name, `@jobkey` stays the same, but `@guid` changes.	X	X
`@peakmb`	Peak memory used by the pipeline, in MB.	X	X
`@maxpeakmb`	Maximum peak memory the pipeline is allowed to consume, in MB.	X	X
`@rundate`	Date/time of the execution start (normally aligned with `@executionid`).	X	X
`@upsertcolumns`	List of key columns that identify a unique record. When CDC is engaged with `CAPTURE`, holds the key columns used for uniqueness.		X

Reader / Paging Markers

Variable	Description	Reader
`@highwatermark`	Current high watermark value when paging or HWM is applied on the source (SQL, HTTP High Watermark). Empty string if none exists. See High Watermark Values.	X
`@highwatermarknull`	Same as `@highwatermark`, or injects the `NULL` string when no watermark is set.	X
`@pagingmarker`	Paging marker in use during SELECT HTTP paging.	X
`@pagingcount`	0-based page count during SELECT HTTP paging.	X
`@pagingindex`	1-based page index during SELECT HTTP paging (`@pagingcount + 1`).	X
`@recordcount`	When HTTP paging uses a simple offset strategy, the current 0-based record count so far (for example 0, 100, 200…).	X
`@recordindex`	When HTTP paging uses a simple offset strategy, the current 1-based record index so far (for example 1, 101, 201…).	X
`@sourcebatchindex`	1-based page index being processed by a `SELECT BATCH DB` operation.	X

Writer Variables

Variable	Description	Writer
`@batchid`	Unique identifier for each batch of write operations when batching applies (for example, 500 records with Max Batch Count = 100 creates 5 batches).	X
`@packagefile`	Name of the Change Log being applied or replayed against the target system.	X
`@sourceexecutionid`	Unix timestamp in milliseconds of the execution that created the Change Log being applied or replayed. For a normal reader-to-writer flow, this matches the reader’s `@executionid`.	X
`@sourceobject`	Name of the source object, if known, such as an HTTP URI, file name, or queue name. Normally blank for database sources unless a CDC table is the source.	X
`@sourceobjectname`	When the source object is a file name or URL, the resource or file name portion of that object.	X
`@startindex`	1-based index of the next batch of items to process on the target (`@totalwrite + 1`).	X
`@status`	Overall outcome of the write/completion path (`success` / `failure`, or `1` / `0`). Available during finalization / completion script execution.	X
`@targetobject`	Name of the target object, such as a database table name or HTTP target URI.	X
`@totalwrite`	Total number of write operations completed so far. Larger Change Logs may contain multiple inner batches; this value increases after each completed inner batch. For example, if an inner batch has 1000 records and Max Batch Count is 100, the inner batch is processed 10 times, and `@totalwrite` increases by 1000 only after that inner batch finishes successfully.	X

Count Variables

These variables report the current in-memory pipeline dataset. Unlike other runtime variables that use a single @ prefix (for example @executionid), count markers use a double @@ form (for example @@rowcount).

Variable	Description	Reader	Writer
`@@rowcount`	Number of records currently in the data pipeline.	X	X
`@@colcount`	Number of columns currently in the data pipeline.	X	X