Fan Out Records into Files

Overview

When you need to write a file using the current content of the pipeline data set, you may want to write them all into a single file or spread the records across multiple files (fan out). The process of writing files can be done using the SINK DRIVE and the PUSH INTO DRIVE operations. The process for fanning records is the same.

Fanning records across multiple files is done through the naming convention used in the CONTAINER and/or the FILE setting, regardless of the type of file created (Parquet, JSON, XML, CSV).

Fanning out records work against all drive endpoints, including FTP sites and cloud endpoints. If the folders/containers do not exist, they will be automatically created if DataZen has sufficient permissions.

Without Fan Out

To write all records into the same file, the CONTAINER and FILE settings must not use a field value from the data set and cannot use date/time markers (such as YYYY). This example writes a single file using the executionId of the job properties to write the file. Rerunning this command will create a different file with all available records.

SINK INTO DRIVE [adls]
    FORMAT 'PARQUET'
    CONTAINER 'inbound'
	FILE '/load/data_@executionid.parquet'
	COMPRESSION 'snappy';

With Fan Out

To write all records across multiple file, the CONTAINER and/or FILE settings must use a field value from the data set and/or date/time markers (such as YYYY). This example writes multiple files based on an existing pipeline data set field called stateCode and distributes them across multiple folders/containers based on a date (note: if the DATE_FIELD option is not set to an existing field in the pipeline data set, the current execution time is used intead).

Using field names may result in file names being invalid depending on the target endpoint. You are responsible for ensuring the field values provided are not null and valid for the target drive.

SINK INTO DRIVE [adls]
    FORMAT 'PARQUET'
    CONTAINER 'inbound[YYYY]'
	FILE '/load/[MM]/data_{{stateCode}}.parquet'
	COMPRESSION 'snappy';