HTTP Connection: Rate Limiting

Most SaaS and cloud services throttle requests when you call an endpoint too frequently. When you create an HTTP/S connection in DataZen, you can configure rate limiting to stay within those limits and reduce throttling errors.

Two rate limiting options are available and can be used together in DataZen Manager:

• Call Rate — limit the number of calls made within a time window
• KB Rate — limit the amount of data transmitted within a time window

Call Rate and Bursting

Configure how many calls are allowed in a time window (for example, 20 calls every 5 minutes). Choose a time unit of Second, Minute, Hour, or Day.

When Allow Bursting is disabled, calls are spread evenly across the time window. For example, if 10 calls per minute are specified, a call is made about every 6 seconds. When Allow Bursting is enabled, calls are made as quickly as possible until the limit is reached, then the job pauses until the next window starts. Using the same example, up to 10 calls would run immediately; additional calls wait up to 60 seconds before another batch of 10 can be sent.

KB Rate (DataZen Manager)

In DataZen Manager, you can also limit throughput by data volume using KB Rate. That option is not available in the Web Portal; use Manager when you need both Call Rate and KB Rate on the same connection.

When a rate limit is reached for a job, a warning appears in the output window. The job pauses so it remains within the configured limit.

Rate limiting applies per job. If two jobs share the same connection and that connection has a call rate limit, each job is limited independently. The combined call volume across both jobs is not rate-limited as a single aggregate.