DataZen Documentation
DataZen User Guide

Pipelines as Tools

DataZen lets data engineers build governed pipelines, expose them as MCP Tools, and test them with a real LLM inside the Portal — before wiring those same tools into an external agent platform such as Claude Desktop, Cursor, or Hermes.

The end-to-end flow is:

  1. Configure an HTTP/S connection as an AI Endpoint
  2. Expose a Reader or Direct pipeline as an MCP Tool
  3. Develop and validate tools in Portal AI Chat
  4. Connect external agents with the datazen-pipeline-mcp package

Portal AI Chat is the built-in playground for this loop. When the AI Endpoint and MCP Tools are set up correctly, you can discover tools, call them with natural language, and refine pipeline definitions without leaving the agent portal.

Step 1: Configure an AI Endpoint

Create or edit an HTTP/S connection that points to your LLM provider or gateway. Enable AI Endpoint and set defaults such as model, max tokens, temperature, maximum tool loops, and system prompt.

These defaults drive Portal AI Chat and can be overridden per session. Full field reference: HTTP Connection: AI Endpoint.

Web Portal HTTP connection with Enable AI Endpoint and AI Configuration defaults
Web Portal: Enable AI Endpoint on an HTTP/S connection.

Step 2: Expose a Pipeline as an MCP Tool

Open a Reader or Direct pipeline in the SQL Editor (Portal or DataZen Manager). Write the SQL CDC script, declare optional parameters with DECLARE, then click MCP Tool to define the tool name, descriptions, input fields, and output fields. Enable the pipeline as an LLM tool and save.

Matching DECLARE names to MCP input field names lets the LLM pass filters or arguments into the script. See MCP Tool for configuration details and a full example (including optional state-code filtering).

Web Portal MCP Tool configuration dialog for a drivers pipeline
Web Portal: MCP Tool definition for a pipeline exposed to the LLM.

Step 3: Develop and Test in Portal AI Chat

Under Playground → AI Chat, select the AI Endpoint connection and model, then chat with the agent. Ask which tools are available, try sample prompts from the MCP Tool definition, and verify that optional parameters behave as expected.

This step is meant for data engineers: iterate on the pipeline script and tool metadata in the same portal where you test the LLM’s use of those tools. When the conversation and tool results look right, the same MCP definitions are ready for external agents.

Web Portal AI Chat listing available MCP tools such as drivers, weather search, and Zendesk search
Web Portal: AI Chat lists MCP Tools exposed by the agent (for example drivers).

Step 4: Connect External Agent Platforms

When tools are validated in AI Chat, expose them to any MCP-compatible client with the open-source package datazen-pipeline-mcp. The package is a local stdio MCP server: the client (Claude Desktop, Cursor, Hermes, MCP Inspector, and others) launches it and authenticates to your DataZen agent with a Service Token.

Create a Service Token under Administration → Service Tokens (or in DataZen Manager). See Security Overview for token scoping. Then add the MCP server to the client config. Typical Claude Desktop / Cursor JSON: 

{
  "mcpServers": {
    "datazen-pipeline": {
      "command": "npx",
      "args": ["-y", "datazen-pipeline-mcp"],
      "env": {
        "DZ_AGENT_ID": "<your agent id>",
        "DZ_CLIENT_ID": "<your client id>",
        "DZ_SECRET": "<your secret>",
        "DZ_FUNCTIONS_BASE_URL": "https://<environment>.ipaas.sc.enzounified.com/api"
      }
    }
  }
}

For Hermes, register the same command and environment variables in the mcp_servers block of the Hermes config (YAML), preferably with ${VAR} placeholders and real values in $HERMES_HOME/.env. Full options, cloud vs self-hosted routes, and refresh behavior are documented on the npm package page.

DZ_FUNCTIONS_BASE_URL must include the /api suffix. For cloud agents, DZ_AGENT_ID is required; for self-hosted deployments set DZ_DEPLOYMENT=selfhosted and omit the agent id.

After the client connects, the tools you built and tested in Portal AI Chat appear in the external agent. Updates to pipeline MCP definitions refresh automatically (clients that honor tools/list_changed need no restart).

Related Topics