diff --git a/docs/jobs/limits.md b/docs/jobs/limits.md index 77ecf74aaa8..9d343fa99a0 100644 --- a/docs/jobs/limits.md +++ b/docs/jobs/limits.md @@ -2,75 +2,53 @@ title: Limits --- -Both microservice and OpenFn.org (the iPaaS) have some pre-configured limits to -ensure smooth operation. Read below for a detailed explanation of those defaults -and how to adjust the limits. +OpenFn has a number of configurable limits that help ensure smooth operation. +This page details what those limits are and how they can be adjusted. -:::tip +:::tip Increasing limits -If you don't feel like reading much, here's the tl;dr: - -1. Don't send more than `8MB` to the `/inbox` API in a single request. -2. Make sure your `expression.js` and your `state.json` don't total more than - `10MB` when you _start_ or _finish_ a run. -3. Keep memory usage below `512MB` _during_ the run. +Contact enterprise@openfn.org to increase limits on standard plans, configure +custom limits on dedicated deployments, or get help adjusting limits on your +locally deployment instance. ::: -## NodeVM memory limits - -The default memory limit for a Node process is `512MB` and unless you've -customized it for use on core or have agreed to a high-memory plan on -OpenFn.org, job runs will be killed by Node (`JavaScript heap out of memory`) -when they reach that threshold. - -### Increase memory limit for an individual execution - -For a `1GB` limit for an individual job, use: - -```sh -node --max-old-space-size=1024 core execute ...arguments -``` - -### Increase memory limit for all jobs on that machine - -For a `4GB` limit for all jobs, set an environment variable with: - -```sh -export NODE_OPTIONS=--max_old_space_size=4096 -``` +## Workflow execution duration (1 hour) -## Request body limit on inboxes +Each workflow attempt needs to complete in less than `1 hour`. You can view the +duration of each attempt by clicking on the attempt ID. If an attempt exceeds +this limit, it will be killed by the worker and you'll see a `Killed:Timeout` +badge as your attempt state. -The maximum size of the JSON body accepted on the `/inbox/your-uuid` endpoint is -`8MB`. Note that payloads should be kept below this limit in order to ensure -that [Job state limits](#job-state-limits) are not hit when a subsequent job run -is executed. +> _Instance superusers can control this limit the `MAX_RUN_DURATION` environment +> variable._ -:::note +## Memory Usage (1GB) -If you're using OpenFn for bulk data processing/periodic batch jobs, you can -either reconfigure these limits on your microservice deployment or contact -enterprise@openfn.org to have special provisions made for your OpenFn.org -project. +Each workflow attempt may not use more than `1GB` of memory. You can view the +maximum memory usage of each attempt by clicking on the attempt ID. If an +attempt exceeds this limit, it will be killed by the worker and you'll see a +`Killed:OOM` badge as your attempt state. -::: +> _Instance superusers can control this limit via the `MAX_RUN_MEMORY` +> environment variable._ -## Job state limits +## Dataclip Size (10MB) -When a job is to be sent for processing the total size of all job artifacts -(your job `expression` and the initial `state`) cannot exceed `10MB`. In other -words, your code (`expression.js`) plus your data and configuration -(`state.json`) cannot exceed `10MB` in total. +1. Each **webhook request** to a trigger URL cannot exceed `10MB`. +2. If you are persisting the final state of each **run** as a dataclip, each + dataclip may not exceed `10MB`. +3. If you are persisting the final state of an **attempt** as a dataclip, it may + not exceed `10MB`. -Similarly, when state is saved to cloud storage/disk _after_ execution there is -a `10MB` limit to the size of a job's final `state`. +If you send a payload to a webhook trigger URL which breaches this limit, the +server will respond with a `413` error with a `:request_entity_too_large` +message. If the dataclips produced by the final state of runs and attempts are +too large, you will no be able to access them and instead see: -If you're dealing with large amounts of data you've got to (a) keep your total -memory use below `512MB` during execution and (b) clean up state so that -whatever you're passing to the next job, whether via FLOW or via saved state for -a CRON job, remains below `10MB`. +```json +{ "__lightning": "Run result too large to save" } +``` -Sometimes, this necessitates streaming data from A -> B in the same job. Other -times it will necessitate storing data as local variables and then posting data -to your inbox in chunks before cleaning up state. +> _Instance superusers can control this limit via the `MAX_DATACLIP_SIZE` +> environment variable._