Large File Streaming

Large file streaming lets a piece process a big file by passing it through as a stream — transferring the bytes a chunk at a time — instead of loading the whole file into memory first. It is useful when a file is too large to hold in RAM at either the source or the destination. Because the file is never fully buffered, a flow can move files far larger than the worker’s memory budget.

Why it matters

A worker runs your piece code inside a sandbox with a bounded memory budget (about 1 GB, minus overhead — see Limits). Reading a large file into a Buffer holds the entire file in that budget at once, so a big enough file exhausts the memory and the worker is OOM-killed. Streaming avoids this: the bytes flow through as a Node Readable, roughly 5 MB at a time, so no process ever holds the whole file. The transfer is transparent — there are no extra “chunk” steps in your flow; a streaming action looks and behaves like any other.

Requires S3 file storage. Full streaming only works when file storage is set to S3 (AP_FILE_STORAGE_LOCATION=S3). With the default database (DB) storage a stream can’t be written into a column incrementally, so the server buffers the whole file in memory before saving it — which defeats the memory savings and means very large files can still fail. Self-hosted installs default to DB; set S3 to get the benefit. See Set up S3.

When to use it

Which approach should you use?

Buffer (a Buffer, the default) — small files of known size where holding the whole file in memory is cheap and simple.
Stream (a Readable) — large files, files of unknown size, or app-to-app transfers (e.g. downloading a large object from one service and uploading it to another) where buffering would risk exhausting memory.

Typical cases: large media files, multi-hundred-MB CSV or log exports, database dumps, and storage-to-storage transfers.

Pieces that support streaming

Streaming is enabled per action. “In” means the action reads its input file as a stream; “out” means it writes the file it produces out as a stream.

Piece	Streams in	Streams out
Amazon S3	Upload File	Read File
Azure Blob Storage	Create Blob	Read Blob
Dropbox	Upload file	Download File
Google Drive	Upload file	Read File Content, List files, Set public access, New File (trigger)
Microsoft OneDrive	Upload file	Get File
Microsoft SharePoint	Upload File	—
FTP/SFTP	Upload File	Read File Content
Subflows	Stream CSV to Subflows	—

More pieces are being enabled over time. Actions not listed here still work — they buffer the file in memory, which is fine within the size limit.

Stream CSV to Subflows streams a CSV straight from the URL you give it and never writes the file into storage, so it is the one entry above that does not need S3 file storage.

Per-service upload ceilings

Streaming removes the memory ceiling, not the destination API’s own limit. Where a service caps what a single request can carry, the action switches to a chunked upload session above that cap:

Piece	One-request cap	Above it
Dropbox	150 MB (`/2/files/upload` answers `409 payload_too_large`)	`/2/files/upload_session/*`, 8 MiB chunks
Microsoft SharePoint	250 MB (Graph’s simple `PUT …/content`)	Graph upload session, 10 MiB chunks
Microsoft OneDrive	4 MiB	Graph upload session, 10 MiB chunks

These are pre-existing API limits rather than streaming limits, and each action now handles its own. One difference matters if your source doesn’t report a size: Dropbox’s session is offset-based, so it just streams the chunks as they arrive, while Graph wants the file’s total length in every fragment’s Content-Range header — so the two Microsoft actions buffer once to learn the length before they can chunk. Give those a source that reports Content-Length when you can.

Building streaming actions

If you are building a piece, you can stream on both sides: read an input file as a stream, and write an output file as a stream.

Writing a file as a stream

ctx.files.write accepts a Buffer or a Readable. Pass a Readable — such as an S3 object body or a streaming HTTP response — and it streams straight to storage instead of being buffered. It returns a file reference string you return from the action, exactly like the buffered form (see Files).

async run(context) {
  const s3 = await resolveS3Client({ authProps, server: context.server });

  const { Body } = await s3.getObject({ Bucket: bucket, Key: key });

  // Body is a Readable — hand it straight to files.write, no Buffer in between
  return context.files.write({
    fileName: key,
    data: Body,
  });
}

Reading a file as a stream

Add streaming: true to a Property.File. The property then resolves to an ApStreamingFile instead of an ApFile:

type ApStreamingFile = {
  filename: string;
  extension?: string;
  size?: number;      // may be undefined when the source doesn't report a length
  body: Readable;
};

Consume body directly. How you hand it to the destination depends on what that destination’s client accepts — in order of preference: 1. A chunking uploader (best). Accepts a stream of unknown length and buffers each part before sending it, so no content length is needed and parts are individually replayable. For S3 that is Upload from @aws-sdk/lib-storage; for Azure Blob Storage, blockBlobClient.uploadStream.

props: {
  file: Property.File({
    displayName: 'File',
    required: true,
    streaming: true,
  }),
},
async run(context) {
  const { file } = context.propsValue;
  const s3 = await resolveS3Client({ authProps, server: context.server });

  await new Upload({
    client: s3,
    params: {
      Bucket: bucket,
      Key: finalFileName,
      Body: file.body,
    },
  }).done();
}

2. An SDK that takes a stream directly. Some clients accept a Readable as-is — Google Drive’s media.body, SFTP’s client.put. Just pass file.body. 3. A single-request HTTP upload. If the destination is a plain PUT/POST that needs an explicit Content-Length, you have to use file.size — and size is best-effort, so this path needs a buffered fallback for when it is missing. Dropbox, SharePoint and OneDrive all look like this:

import { buffer as readableToBuffer } from 'node:stream/consumers';

const headers: Record<string, string> = { 'Content-Type': 'application/octet-stream' };
let body;
if (file.size != null) {
  headers['Content-Length'] = String(file.size);
  body = file.body;
} else {
  body = await readableToBuffer(file.body);
}

size is informational and best-effort — it is undefined when the source reports no Content-Length, and it is also dropped when the response is compressed (Content-Encoding: gzip/br/deflate), because the decompressed body no longer matches the advertised length. Don’t require it: prefer pattern 1 or 2, which never need it.

Property.File() without streaming is unchanged — it still resolves to an ApFile with a data buffer, so existing actions keep working.

Limits & storage

Writing into storage is capped. A stream passed to ctx.files.write is counted against AP_MAX_FILE_SIZE_MB (Cloud: 10 MB, self-hosted default: 25 MB) while the bytes flow; exceeding it aborts the transfer and fails the step. See Limits.
Reading a streamed input is not capped. A streaming: true file input has no AP_MAX_FILE_SIZE_MB ceiling — that is deliberate, since the point of the feature is to move files larger than the cap out to an external service.
Storage backend. Streaming end-to-end requires S3 file storage — see the callout at the top of this page.

Individual parts retry, the whole transfer doesn’t. A multipart uploader buffers each ~5 MB part before sending it, so it can replay that part on a transient error. But the source Readable can be read only once, so there is no retry of the transfer as a whole: a streamed ctx.files.write gets no S3-error fallback to database storage, and a step that fails after its stream is drained cannot simply be re-run against the same stream.
httpClient does not retry a stream body at all. The retry loop in pieces-common reuses the body it serialized before the first attempt, and a stream is one-shot — retrying would replay a drained stream and send a truncated body. So a request whose body is a Readable (or a form-data payload, which is streamed too) runs its retries setting as 0. This applies to any piece sending a stream through httpClient, not just file actions. If you need real retries, buffer the body instead.
Multipart webhook signatures aren’t verified. Verifying an HMAC signature over an uploaded file needs the raw bytes held in memory, which is exactly what streaming avoids. Streamed multipart webhook uploads therefore skip signature verification. JSON, XML, form, and text webhook bodies are unaffected.

Build a Piece

Piece Reference

Misc

Why it matters

When to use it

Pieces that support streaming

Per-service upload ceilings

Building streaming actions

Writing a file as a stream

Reading a file as a stream

Limits & storage

​Why it matters

​When to use it

​Pieces that support streaming

​Per-service upload ceilings

​Building streaming actions

​Writing a file as a stream

​Reading a file as a stream

​Limits & storage

Why it matters

When to use it

Pieces that support streaming

Per-service upload ceilings

Building streaming actions

Writing a file as a stream

Reading a file as a stream

Limits & storage