5.3 Workflow Orchestration Limitations
Temporal limits
OLIVE workflows are executed by Temporal. Besides the datapoint result size cap, you may also hit Temporal limits on activity timeouts and on how much data is stored in the workflow history.
Examples below reflect typical Temporal defaults.
Activity Timeouts (Temporal Execution Layer)
In addition to workflow- and task-level timeouts defined in the DSL, OLIVE workflows are executed on Temporal, which introduces Activity-level timeouts.
These timeouts control how long individual datapoint calls (Activities) are allowed to run and are configured at deployment time, not inside the workflow definition.
This section explains what these timeouts mean and how to configure them.
Activity Timeout Types
Every call task in a workflow is executed as a Temporal Activity.
Temporal provides multiple timeout types to control different phases of execution. If not explicitly configured, OLIVE uses the following defaults.
| Timeout | Default Value | Description |
|---|---|---|
| startToCloseTimeout | 12 hours | Maximum time a single Activity execution is allowed to run after a worker has started it. If this timeout is exceeded, the Activity is immediately failed. Recommended to always set this timeout. |
| scheduleToCloseTimeout | unlimited | Maximum total time allowed for the Activity from scheduling to completion, including all retries. Acts as the overall upper bound for the Activity lifecycle. |
| scheduleToStartTimeout | 24 hours | Maximum time an Activity may remain in the task queue before being picked up by a worker. If exceeded, the Activity fails without ever starting. |
| heartbeatTimeout | 2 minutes | Maximum time allowed between heartbeat signals for long-running Activities. If no heartbeat is received within this interval, the Activity is considered failed. Only applies if the Activity emits heartbeats. |
The Constraint Rules
When you set a timeout value, all other timeouts must respect these rules:
- heartbeatTimeout ≤ startToCloseTimeout — You cannot set a heartbeat interval longer than the execution timeout
- startToCloseTimeout ≤ scheduleToCloseTimeout — You cannot set an execution timeout longer than the total time limit
- scheduleToStartTimeout ≤ scheduleToCloseTimeout — You cannot set a queue timeout longer than the total time limit
More information can be found here: https://temporal.io/blog/activity-timeouts
Practical Impact
If you increase startToCloseTimeout, you may need to increase scheduleToCloseTimeout as well to maintain valid constraints. For example, if you change startToCloseTimeout from 15 minutes to 30 minutes, but scheduleToCloseTimeout is still 30 minutes, you have violated the hierarchy (the total time equals just the execution time with no room for queue wait). Similarly, if you set heartbeatTimeout to 5 minutes but startToCloseTimeout is only 3 minutes, the configuration is invalid because the activity would fail on heartbeat timeout before it could run out of execution time.
How to configure
Activity timeouts are configured via environment variables at deploy time in the values.yaml of the temporal-caller service.
The values are passed through the express.env section of values.yaml. Below you can see an example configuration:
values.yaml
express:
env:
ACTIVITY_START_TO_CLOSE_TIMEOUT: "900000" # 15 min
ACTIVITY_SCHEDULE_TO_START_TIMEOUT: "120000" # 2 min
ACTIVITY_SCHEDULE_TO_CLOSE_TIMEOUT: "1800000" # 30 min
ACTIVITY_HEARTBEAT_TIMEOUT: "10000" # 10 sec
All values are interpreted as milliseconds.
If no values are provided, the default configuration is used instead.
Workflow history size exceeded
Limit description
Workflow history is an append-only log Temporal keeps for a run: every datapoint call, timer, wait and completion adds a record to it.
A terminated execution may include a history event like:
{
"type": "workflowExecutionTerminatedEventAttributes",
"reason": "Workflow history size exceeds limit.",
"identity": "history-service"
}
How it grows
The more steps and the larger the data written into history (especially big inputs or results), the bigger the history gets.
How to handle it
Temporal enforces a maximum on that history. If the limit is exceeded, the workflow execution is terminated.
This table shows the threshold values for the limit as both an approximate number of history records and an approximate total size.
| Threshold | History records (approx.) | History size (approx.) |
|---|---|---|
| Warning | 10,240 | 10 MB |
| Hard limit | 51,200 | 50 MB |
Common causes
- A long-running workflow with many calls, timers or waits — each one adds history records.
- Tight loops: a
forover a large collection, where every iteration schedules a call or a timer. - Large datapoint inputs or results recorded in history, which inflates the serialized size rather than the record count (see the next section).
- Very wide fan-out: a
forkwith many branches, or a large batch of calls scheduled in one turn.
To stay under the limit, you can:
- Do fewer steps per run: tighter loops mean fewer history records.
- Keep datapoint payloads small: less data serialized into history on each
call. - Use references for bulk data — a file key rather than the content, so heavy data is loaded inside the datapoint instead of travelling through the workflow. See File Handling in Connectivity.
- Avoid huge command batches in a single workflow task; spread the work across turns if needed.
- Split the work into separate, smaller workflows. A
calltask takes theidof any connectable element, so a parent workflow can invoke other workflows and connectors. Each one runs as its own job with its own history, and the parent records only the call's input and result — see below.
Activity input exceeds schedule / payload limit
Limit description
The input for a datapoint (the activity schedule payload) is the argument object Temporal sends when a call runs as an Activity.
Temporal stores that input in workflow history. There is a maximum size for one payload (blob limit); that is different from the total workflow history size limit in the previous subsection.
A termination or failure may include a history event like:
{
"type": "workflowExecutionTerminatedEventAttributes",
"reason": "BadScheduleActivityAttributes: ScheduleActivityTaskCommandAttributes.Input exceeds size limit.",
"identity": "history-service"
}
How it grows
Each datapoint schedule adds bytes to history. Schedules from the same workflow turn are applied together, so many small inputs in one turn can still exceed the workflow task limit.
How to handle it
If one payload or the whole batch is too large, Temporal may reject it or terminate the run (see the example above).
| Scope | Approx. limit |
|---|---|
| One datapoint input (single payload) | ~2 MB |
| One workflow task completion (all commands in that turn together) | ~4 MB |
Common causes
- The serialized input object is simply too large: a big JSON document, an entire file inlined, a long list.
- Passing a whole document or blob into the datapoint instead of a reference to it.
- Scheduling too many calls in one turn, so that the combined commands exceed the workflow-task limit even though no single input does.
To stay under these limits, you can:
- Shrink inputs: pagination, chunking, or smaller argument objects.
- Pass a reference instead of the content — a file key, an id, a URL — and load the data inside the datapoint. This is the one approach that does not depend on how close you are to the limit.
- Spread the calls across more workflow turns rather than one burst of schedules.
- Avoid pushing megabyte-scale workflow state straight into the next datapoint input.
A datapoint that is too large to use inside a workflow is usually fine when invoked directly or from a connector, because nothing records its input and result in a history. If a step genuinely has to move a large payload, that is an argument for taking it out of the workflow — or for moving the payload through the file storage and keeping only the key in the workflow.
Splitting a large orchestration
Because with.datapoint accepts the id of any connectable element, a workflow that has grown too large
can be broken into a parent workflow that calls several smaller ones:
{
"runPhaseOne": {
"call": "olive",
"with": { "datapoint": "mysolution-wf-phase-one" }
}
}
Each nested workflow is executed as its own job, so it gets its own history budget rather than adding records to the parent's. The parent's history grows by one call per phase instead of by every step inside it.
The per-payload limit still applies at the boundary: whatever a nested workflow returns travels back through the parent's history as that call's result. Splitting the work helps with the number of history records and with total history size; it does not let a single large value through. For that, keep the payload in the file storage and return the key.