Large File Streaming

Large file streaming lets a piece process a big file by passing it through as a stream — transferring the bytes a chunk at a time — instead of loading the whole file into memory first. It is useful when a file is too large to hold in RAM at either the source or the destination. Because the file is never fully buffered, a flow can move files far larger than the worker’s memory budget.

Why it matters

A worker runs your piece code inside a sandbox with a bounded memory budget (about 1 GB, minus overhead — see Limits). Reading a large file into a Buffer holds the entire file in that budget at once, so a big enough file exhausts the memory and the worker is OOM-killed. Streaming avoids this: the bytes flow through as a Node Readable, roughly 5 MB at a time, so no process ever holds the whole file. The transfer is transparent — there are no extra “chunk” steps in your flow; a streaming action looks and behaves like any other.

Requires S3 file storage. Full streaming only works when file storage is set to S3 (AP_FILE_STORAGE_LOCATION=S3). With the default database (DB) storage a stream can’t be written into a column incrementally, so the server buffers the whole file in memory before saving it — which defeats the memory savings and means very large files can still fail. Self-hosted installs default to DB; set S3 to get the benefit. See Set up S3.

When to use it

Which approach should you use?

Buffer (a Buffer, the default) — small files of known size where holding the whole file in memory is cheap and simple.
Stream (a Readable) — large files, files of unknown size, or app-to-app transfers (e.g. downloading a large object from one service and uploading it to another) where buffering would risk exhausting memory.

Typical cases: large media files, multi-hundred-MB CSV or log exports, database dumps, and storage-to-storage transfers.

Pieces that support streaming

Streaming is enabled per action. Today:

Piece	Action	Streams
Amazon S3	Upload File	The file it reads in
Amazon S3	Read File	The object it writes out

More pieces are being enabled over time. Actions not listed here still work — they buffer the file in memory, which is fine within the size limit.

Building streaming actions

If you are building a piece, you can stream on both sides: read an input file as a stream, and write an output file as a stream.

Writing a file as a stream

ctx.files.write accepts a Buffer or a Readable. Pass a Readable — such as an S3 object body or a streaming HTTP response — and it streams straight to storage instead of being buffered. It returns a file reference string you return from the action, exactly like the buffered form (see Files).

async run(context) {
  const s3 = await resolveS3Client({ authProps, server: context.server });

  const { Body } = await s3.getObject({ Bucket: bucket, Key: key });

  // Body is a Readable — hand it straight to files.write, no Buffer in between
  return context.files.write({
    fileName: key,
    data: Body,
  });
}

Reading a file as a stream

Add streaming: true to a Property.File. The property then resolves to an ApStreamingFile instead of an ApFile:

type ApStreamingFile = {
  filename: string;
  extension?: string;
  size?: number;      // may be undefined when the source doesn't report a length
  body: Readable;
};

Consume body directly. Prefer an uploader that accepts a stream of unknown length — for S3 that is Upload from @aws-sdk/lib-storage, which splits the body into ~5 MB parts and needs no content length up front.

props: {
  file: Property.File({
    displayName: 'File',
    required: true,
    streaming: true,
  }),
},
async run(context) {
  const { file } = context.propsValue;
  const s3 = await resolveS3Client({ authProps, server: context.server });

  await new Upload({
    client: s3,
    params: {
      Bucket: bucket,
      Key: finalFileName,
      Body: file.body,
    },
  }).done();
}

size is informational and best-effort — it is undefined when the source reports no Content-Length, and it is also dropped when the response is compressed (Content-Encoding: gzip/br/deflate), because the decompressed body no longer matches the advertised length. Don’t require it: only pass it to an API that needs an explicit content length, and make sure that path has a buffered fallback for when it is missing.

Property.File() without streaming is unchanged — it still resolves to an ApFile with a data buffer, so existing actions keep working.

Limits & storage

Writing into storage is capped. A stream passed to ctx.files.write is counted against AP_MAX_FILE_SIZE_MB (Cloud: 10 MB, self-hosted default: 25 MB) while the bytes flow; exceeding it aborts the transfer and fails the step. See Limits.
Reading a streamed input is not capped. A streaming: true file input has no AP_MAX_FILE_SIZE_MB ceiling — that is deliberate, since the point of the feature is to move files larger than the cap out to an external service.
Storage backend. Streaming end-to-end requires S3 file storage — see the callout at the top of this page.

Individual parts retry, the whole transfer doesn’t. A multipart uploader buffers each ~5 MB part before sending it, so it can replay that part on a transient error. But the source Readable can be read only once, so there is no retry of the transfer as a whole: a streamed ctx.files.write gets no S3-error fallback to database storage, and a step that fails after its stream is drained cannot simply be re-run against the same stream.
Multipart webhook signatures aren’t verified. Verifying an HMAC signature over an uploaded file needs the raw bytes held in memory, which is exactly what streaming avoids. Streamed multipart webhook uploads therefore skip signature verification. JSON, XML, form, and text webhook bodies are unaffected.

Build a Piece

Piece Reference

Misc

Why it matters

When to use it

Pieces that support streaming

Building streaming actions

Writing a file as a stream

Reading a file as a stream

Limits & storage

​Why it matters

​When to use it

​Pieces that support streaming

​Building streaming actions

​Writing a file as a stream

​Reading a file as a stream

​Limits & storage

Why it matters

When to use it

Pieces that support streaming

Building streaming actions

Writing a file as a stream

Reading a file as a stream

Limits & storage