> ## Documentation Index
> Fetch the complete documentation index at: https://www.activepieces.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Large File Streaming

> Move large files through a flow without buffering them in memory

Large file streaming lets a piece process a big file by passing it through as a stream —
transferring the bytes a chunk at a time — instead of loading the whole file into memory
first. It is useful when a file is too large to hold in RAM at either the source or the
destination. Because the file is never fully buffered, a flow can move files far larger
than the worker's memory budget.

## Why it matters

A worker runs your piece code inside a sandbox with a bounded memory budget (about 1 GB,
minus overhead — see [Limits](/docs/install/reference/limits)). Reading a large file into a
`Buffer` holds the entire file in that budget at once, so a big enough file exhausts the
memory and the worker is OOM-killed.

Streaming avoids this: the bytes flow through as a Node `Readable`, roughly 5 MB at a
time, so no process ever holds the whole file. The transfer is transparent — there are no
extra "chunk" steps in your flow; a streaming action looks and behaves like any other.

<Warning>
  **Requires S3 file storage.** Full streaming only works when file storage is set to S3
  (`AP_FILE_STORAGE_LOCATION=S3`). With the default database (`DB`) storage a stream **can't**
  be written into a column incrementally, so the server buffers the whole file in memory
  before saving it — which defeats the memory savings and means very large files can still
  fail. Self-hosted installs default to `DB`; set S3 to get the benefit. See
  [Set up S3](/docs/install/configure-operate/setup-s3).
</Warning>

## When to use it

<Note>
  **Which approach should you use?**

  * **Buffer** (a `Buffer`, the default) — small files of known size where holding the whole
    file in memory is cheap and simple.
  * **Stream** (a `Readable`) — large files, files of unknown size, or app-to-app transfers
    (e.g. downloading a large object from one service and uploading it to another) where
    buffering would risk exhausting memory.
</Note>

Typical cases: large media files, multi-hundred-MB CSV or log exports, database dumps, and
storage-to-storage transfers.

## Pieces that support streaming

Streaming is enabled per action. Today:

| Piece     | Action      | Streams                      |
| --------- | ----------- | ---------------------------- |
| Amazon S3 | Upload File | The file it reads **in**     |
| Amazon S3 | Read File   | The object it writes **out** |

More pieces are being enabled over time. Actions not listed here still work — they buffer
the file in memory, which is fine within the [size limit](/docs/install/reference/limits).

## Building streaming actions

If you are [building a piece](/docs/build-pieces/building-pieces/overview), you can stream on both sides: read an
input file as a stream, and write an output file as a stream.

### Writing a file as a stream

`ctx.files.write` accepts a `Buffer` **or** a `Readable`. Pass a `Readable` — such as an S3
object body or a streaming HTTP response — and it streams straight to storage instead of
being buffered. It returns a file reference string you return from the action, exactly like
the buffered form (see [Files](./files)).

```ts theme={null}
async run(context) {
  const s3 = await resolveS3Client({ authProps, server: context.server });

  const { Body } = await s3.getObject({ Bucket: bucket, Key: key });

  // Body is a Readable — hand it straight to files.write, no Buffer in between
  return context.files.write({
    fileName: key,
    data: Body,
  });
}
```

### Reading a file as a stream

Add `streaming: true` to a `Property.File`. The property then resolves to an
`ApStreamingFile` instead of an `ApFile`:

```ts theme={null}
type ApStreamingFile = {
  filename: string;
  extension?: string;
  size?: number;      // may be undefined when the source doesn't report a length
  body: Readable;
};
```

Consume `body` directly. Prefer an uploader that accepts a stream of unknown length — for S3
that is `Upload` from `@aws-sdk/lib-storage`, which splits the body into \~5 MB parts and
needs no content length up front.

```ts theme={null}
props: {
  file: Property.File({
    displayName: 'File',
    required: true,
    streaming: true,
  }),
},
async run(context) {
  const { file } = context.propsValue;
  const s3 = await resolveS3Client({ authProps, server: context.server });

  await new Upload({
    client: s3,
    params: {
      Bucket: bucket,
      Key: finalFileName,
      Body: file.body,
    },
  }).done();
}
```

`size` is informational and best-effort — it is `undefined` when the source reports no
`Content-Length`, and it is also dropped when the response is compressed
(`Content-Encoding: gzip`/`br`/`deflate`), because the decompressed body no longer matches the
advertised length. Don't require it: only pass it to an API that needs an explicit content
length, and make sure that path has a buffered fallback for when it is missing.

<Tip>
  `Property.File()` without `streaming` is unchanged — it still resolves to an `ApFile` with a
  `data` buffer, so existing actions keep working.
</Tip>

## Limits & storage

* **Writing into storage is capped.** A stream passed to `ctx.files.write` is counted against
  `AP_MAX_FILE_SIZE_MB` (Cloud: 10 MB, self-hosted default: 25 MB) *while* the bytes
  flow; exceeding it aborts the transfer and fails the step. See
  [Limits](/docs/install/reference/limits).
* **Reading a streamed input is not capped.** A `streaming: true` file input has no
  `AP_MAX_FILE_SIZE_MB` ceiling — that is deliberate, since the point of the feature is to move
  files larger than the cap out to an external service.
* **Storage backend.** Streaming end-to-end requires S3 file storage — see the callout at
  the top of this page.

<Warning>
  - **Individual parts retry, the whole transfer doesn't.** A multipart uploader buffers each
    \~5 MB part before sending it, so it can replay *that part* on a transient error. But the
    source `Readable` can be read only once, so there is no retry of the transfer as a whole: a
    streamed `ctx.files.write` gets no S3-error fallback to database storage, and a step that
    fails after its stream is drained cannot simply be re-run against the same stream.
  - **Multipart webhook signatures aren't verified.** Verifying an HMAC signature over an
    uploaded file needs the raw bytes held in memory, which is exactly what streaming avoids.
    Streamed multipart webhook uploads therefore skip signature verification. JSON, XML, form,
    and text webhook bodies are unaffected.
</Warning>