Skip to main content

6.12 File Handling in Connectivity

General​

Files can be both received as input for connectivity elements as well as returned as output from them.

They can also be passed as parameters from one element to another in a Connector.

Files as input​

Files can be sent as input for a Datapoint or a Connector by sending a multipart form data request to invoke the data-orchestrator.

Example Datapoint using a file input:

{
"id": "dp-with-file-input",
"type": "datapoint",
"version": "0.0.1",
"deprecated": false,
"datapoint": {
"type": "http",
"http": {
"url": "https://<myfileservice>/upload",
"method": "POST",
"body": {
"file": "${input.file}",
"otherparam1": "value1",
"otherparam2": "value2"
}
}
}
}

Example Connector using a file input:

{
"id": "connector-with-file-input",
"type": "connector",
"version": "0.0.1",
"deprecated": false,
"connector": {
"elements": [
{
"ref": "firstDp",
"element": "some-dp-with-no-file-input"
},
{
"element": "dp-with-file-input",
"parameters": {
"file": "${root.file}",
"otherparam1": "${ref.firstDp}"
}
}
]
}
}

Example Bruno call:

post {
url: {{envURL}}data-orchestrator/v1/invoke
body: multipartForm
auth: inherit
}

body:multipart-form {
id: dp-with-file-input
file: @file(C:\test.txt)
}

Files as output​

Datapoints and Connectors can return file contents - either by converting the file content to string and wrapping it in the poll response JSON or by downloading it directly

The response of those Datapoints and Connectors can be retrieved in two ways:

  • as normal text response by calling the data-orchestrator/v1/poll endpoint
  • as actual file download with the original mime type by calling the data-orchestrator/v1/result endpoint

For more details on those endpoints see the Technical documentation for OLIVE

File categories​

New in OLIVE 1.3

Before 1.3 every uploaded file went into one undifferentiated store. From 1.3 a file belongs to a category, and the category decides who may overwrite it. An upload that passes no category is stored as a temp file.

There are three categories:

CategoryWho may uploadWho may readMeant for
coreadmin roleeveryoneFiles the OLIVE runtime itself needs, or that influence system or client behaviour. Specially protected so they cannot easily be manipulated
staticeditor roleeveryoneFiles used by a connectivity solution that stay stable — assets, reference data, templates for output
tempeveryoneeveryoneFiles created while connectivity runs: caching, intermediate storage, hand-over between elements

The category is passed with the upload request. Downloads do not take one — the file is found by its key regardless of category, and reading is not role-restricted.

An unrecognised category is rejected rather than falling back to a default.

Which category to use​

Two rules cover almost every case:

  • A file produced while connectivity runs is a temp file — upload it without a key. When no key is passed, the service generates one (a random identifier followed by the file name), and that generated key is what you use afterwards. Choosing your own key for a temp file means someone else can guess it, and because anyone may write to temp, they can also overwrite it.
  • Anything that is meant to stay put gets an explicit category. A file your solution reads later — an asset, a lookup table, a document template — belongs in static, and you have to say so. core is for platform files and needs the admin role; in normal connectivity work you do not upload into it.
The default is temp, and temp is writable by everyone

An upload without a category is a temp file. If that file matters — if something reads it later and would break or misbehave were its content replaced — then leaving the category out is the wrong default: any user of the tenant may overwrite it at the same key.

Pass the category explicitly for anything except genuinely throwaway runtime data.

Keys and folders​

  • Passing no key is the recommended form for temp files: the service composes one as <random-id>_file_<file-name>.
  • Passing a key uses it verbatim, optionally under a folder prefix.
  • A key accepts letters, digits and - _ . ~ @ /. A folder accepts the same except /, which is the storage hierarchy separator.
  • Everything else is rejected with key contains unsupported characters — including spaces. Percent-encoded values are rejected too: pass the decoded string rather than %20.
  • A key segment is limited to 255 bytes and the whole composed key to 1024 bytes, so keys stay portable across storage backends.
A space in a file name will fail the upload

This is the most common upload error, and it appears late — while uploading, not while building:

Uploading (172/254): legal@@01 Impressum@@index.html
Upload failed for legal@@01 Impressum@@index.html: 400 {"error":"key contains unsupported characters"}

Use _ instead of a space in file and folder names. If you are uploading a built artifact — a microfrontend, a documentation bundle — the key is derived from the source path, so a single directory or file with a space in it anywhere in the tree breaks that one upload while the rest succeed.

Uploading a whole directory tree​

The file storage has no directories. When a set of files has to keep its structure — the output of a build, for example — the convention is to flatten the path into the key and separate the segments with @@:

dist/legal/impressum/index.html -> legal@@impressum@@index.html
dist/assets/images/diagram.png -> assets@@images@@diagram.png

@ is allowed in a key precisely so that this works, and it is also what lets a versioned artifact name such as olive-dashboard-app@1.1.7 be used as a folder. Whatever produces the keys — the deployment script for a UI component, or your own tooling — every segment of the original path has to satisfy the character rules above.

Development deployments

A development deployment has no authentication and a single user holding every role, so the static and core write restrictions do not apply there. They start applying when the same connectivity runs in production.

It also means an HTTP datapoint calling the public file-storage-service API needs no credentials there. In an authenticated deployment it does — the datapoint has to reference an Auth entry of type oidc, otherwise the upload or download fails with 401. See Calling OLIVE's own APIs from connectivity.

Restrictions & Limits​

Restrictions on usage​

In the current version file handling in OLIVE has some limitations where and how files can be used:

  • Files can not be used as input for or as parameter within workflows
  • Files can not be used in or transformed with JSONata expressions
  • Files can only be sent as input in multipart form data, direct file streams are not supported
  • Files must be delivered with the key "file" in the multipart form data
  • Passing multiple files as input is not supported
  • There is currently no way to upload files and guarantee that other users can not read it, so sensitive information in files should be treated with care

Limits​

The file storage itself is built for large files. There is no small ceiling on what you can store in the file-storage-service, and storing a large payload there and passing a reference to it is the recommended way to move bulk data through a connectivity scenario.

What is limited is how much data can travel through an execution, and that limit depends on how the scenario is orchestrated:

How the scenario runsWhat limits the payload
A datapoint on its own, or a connectorThe HTTP request and response size of the services involved. Comfortably more than a few MB in a normal deployment
A workflowThe limits of the Temporal workflow engine, which records every call's input and result in the workflow history. A single value above roughly 2 MB is not carried
There is no single supported maximum

The effective ceiling is not a fixed, documented number. It depends on the deployment — how the platform is configured, how much runs in parallel, and which services a request passes through. Treat the figures in this handbook as the order of magnitude at which to change approach, not as a contract.

The safe design does not depend on the limit at all: put the payload in the file storage and pass the file key. That works the same way in a datapoint, a connector and a workflow, and it does not change when the deployment does.

There is also an overall maximum limit of 32GB the file store can grow to.

For the workflow-specific limits and the errors they produce, see Workflow Orchestration Limitations.

Calling the file storage from a microfrontend

A pilet rendered by the microfrontend-interpreter reaches the file storage through the public file-storage-service API, and may use the GET method only — enough to render an image, for example. Every other method, uploads included, must go through the data-orchestrator. See Client Microfrontends.