From cc036283bfe5d1274bc04320a38bde0e7cc63667 Mon Sep 17 00:00:00 2001 From: Taylor Downs Date: Wed, 22 Nov 2023 10:26:39 +0000 Subject: [PATCH 1/4] new limits --- docs/jobs/limits.md | 81 ++++++++++++--------------------------------- 1 file changed, 22 insertions(+), 59 deletions(-) diff --git a/docs/jobs/limits.md b/docs/jobs/limits.md index 77ecf74aaa8..fe497991ca1 100644 --- a/docs/jobs/limits.md +++ b/docs/jobs/limits.md @@ -2,75 +2,38 @@ title: Limits --- -Both microservice and OpenFn.org (the iPaaS) have some pre-configured limits to -ensure smooth operation. Read below for a detailed explanation of those defaults -and how to adjust the limits. +OpenFn has a number of configurable limits that help ensure smooth operation. +This page details what those limits are and how they can be adjusted. -:::tip +:::tip Increasing limits -If you don't feel like reading much, here's the tl;dr: - -1. Don't send more than `8MB` to the `/inbox` API in a single request. -2. Make sure your `expression.js` and your `state.json` don't total more than - `10MB` when you _start_ or _finish_ a run. -3. Keep memory usage below `512MB` _during_ the run. +Contact enterprise@openfn.org to increase limits on standard plans, configure +custom limits on dedicated deployments, or get help adjusting limits on your +locally deployment instance. ::: -## NodeVM memory limits - -The default memory limit for a Node process is `512MB` and unless you've -customized it for use on core or have agreed to a high-memory plan on -OpenFn.org, job runs will be killed by Node (`JavaScript heap out of memory`) -when they reach that threshold. - -### Increase memory limit for an individual execution - -For a `1GB` limit for an individual job, use: - -```sh -node --max-old-space-size=1024 core execute ...arguments -``` - -### Increase memory limit for all jobs on that machine +## Workflow execution duration -For a `4GB` limit for all jobs, set an environment variable with: +Each workflow attempt needs to complete in less than 1 hour. -```sh -export NODE_OPTIONS=--max_old_space_size=4096 -``` +This limit can be adjusted by an instance superuser via the `MAX_RUN_DURATION` +environment variable. -## Request body limit on inboxes - -The maximum size of the JSON body accepted on the `/inbox/your-uuid` endpoint is -`8MB`. Note that payloads should be kept below this limit in order to ensure -that [Job state limits](#job-state-limits) are not hit when a subsequent job run -is executed. - -:::note - -If you're using OpenFn for bulk data processing/periodic batch jobs, you can -either reconfigure these limits on your microservice deployment or contact -enterprise@openfn.org to have special provisions made for your OpenFn.org -project. - -::: +## Memory Usage -## Job state limits +Each workflow attempt may not use more than `1GB` of memory. -When a job is to be sent for processing the total size of all job artifacts -(your job `expression` and the initial `state`) cannot exceed `10MB`. In other -words, your code (`expression.js`) plus your data and configuration -(`state.json`) cannot exceed `10MB` in total. +This limit can be adjusted by an instance superuser via the `MAX_RUN_MEMORY` +environment variable. -Similarly, when state is saved to cloud storage/disk _after_ execution there is -a `10MB` limit to the size of a job's final `state`. +## Dataclip Size -If you're dealing with large amounts of data you've got to (a) keep your total -memory use below `512MB` during execution and (b) clean up state so that -whatever you're passing to the next job, whether via FLOW or via saved state for -a CRON job, remains below `10MB`. +1. Each webhook request to a trigger URL cannot exceed `10MB` +2. If you are persisting the final state of each run as a dataclip, each + dataclip may not exceed `10MB` +3. If you are persisting the final state of an attempt as a dataclip, it may not + exceed `10MB` -Sometimes, this necessitates streaming data from A -> B in the same job. Other -times it will necessitate storing data as local variables and then posting data -to your inbox in chunks before cleaning up state. +These limits can be adjusted by an instance superuser via the +`MAX_DATACLIP_SIZE` environment variable. From afad8dd130808eb796aff488e70d910297e75215 Mon Sep 17 00:00:00 2001 From: Taylor Downs Date: Wed, 22 Nov 2023 12:58:18 +0000 Subject: [PATCH 2/4] Update limits.md TODO - how to communicate the dataclip too large errors. --- docs/jobs/limits.md | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/docs/jobs/limits.md b/docs/jobs/limits.md index fe497991ca1..4b4226cb547 100644 --- a/docs/jobs/limits.md +++ b/docs/jobs/limits.md @@ -15,14 +15,24 @@ locally deployment instance. ## Workflow execution duration -Each workflow attempt needs to complete in less than 1 hour. +Each workflow attempt needs to complete in less than `1 hour`. +You can view the duration of each attempt by clicking on the attempt ID. +If an attempt exceeds this limit, it will be killed by the worker and you'll see +a `Killed:Timeout` badge as your attempt state. + +:::info This limit can be adjusted by an instance superuser via the `MAX_RUN_DURATION` environment variable. +::: + ## Memory Usage -Each workflow attempt may not use more than `1GB` of memory. +Each workflow attempt may not use more than `1GB` of memory. You can view the maximum +memory usage of each attempt by clicking on the attempt ID. If an attempt exceeds this +limit, it will be killed by the worker and you'll see a `Killed:OOM` badge as your +attempt state. This limit can be adjusted by an instance superuser via the `MAX_RUN_MEMORY` environment variable. @@ -35,5 +45,9 @@ environment variable. 3. If you are persisting the final state of an attempt as a dataclip, it may not exceed `10MB` +If you send a larger payload to a webhook URL, the server will respond with a +`413: request_entity_too_large` error. If the dataclips produced by the final state of +runs and attempts are too large, you will see _____________________________________. + These limits can be adjusted by an instance superuser via the `MAX_DATACLIP_SIZE` environment variable. From 8355cde167aa64d00559e006216004f5e205fb40 Mon Sep 17 00:00:00 2001 From: Taylor Downs Date: Wed, 22 Nov 2023 13:52:39 +0000 Subject: [PATCH 3/4] update limits page --- docs/jobs/limits.md | 55 +++++++++++++++++++++------------------------ 1 file changed, 26 insertions(+), 29 deletions(-) diff --git a/docs/jobs/limits.md b/docs/jobs/limits.md index 4b4226cb547..84f32516d0c 100644 --- a/docs/jobs/limits.md +++ b/docs/jobs/limits.md @@ -13,41 +13,38 @@ locally deployment instance. ::: -## Workflow execution duration +## Workflow execution duration (1hr) -Each workflow attempt needs to complete in less than `1 hour`. -You can view the duration of each attempt by clicking on the attempt ID. -If an attempt exceeds this limit, it will be killed by the worker and you'll see -a `Killed:Timeout` badge as your attempt state. +Each workflow attempt needs to complete in less than `1 hour`. You can view the +duration of each attempt by clicking on the attempt ID. If an attempt exceeds +this limit, it will be killed by the worker and you'll see a `Killed:Timeout` +badge as your attempt state. -:::info +> _Instance superusers can control this limit the `MAX_RUN_DURATION` environment +> variable._ -This limit can be adjusted by an instance superuser via the `MAX_RUN_DURATION` -environment variable. +## Memory Usage (1GB) -::: - -## Memory Usage - -Each workflow attempt may not use more than `1GB` of memory. You can view the maximum -memory usage of each attempt by clicking on the attempt ID. If an attempt exceeds this -limit, it will be killed by the worker and you'll see a `Killed:OOM` badge as your -attempt state. +Each workflow attempt may not use more than `1GB` of memory. You can view the +maximum memory usage of each attempt by clicking on the attempt ID. If an +attempt exceeds this limit, it will be killed by the worker and you'll see a +`Killed:OOM` badge as your attempt state. -This limit can be adjusted by an instance superuser via the `MAX_RUN_MEMORY` -environment variable. +> _Instance superusers can control this limit via the `MAX_RUN_MEMORY` +> environment variable._ -## Dataclip Size +## Dataclip Size (10MB) -1. Each webhook request to a trigger URL cannot exceed `10MB` -2. If you are persisting the final state of each run as a dataclip, each - dataclip may not exceed `10MB` -3. If you are persisting the final state of an attempt as a dataclip, it may not - exceed `10MB` +1. Each **webhook request** to a trigger URL cannot exceed `10MB`. +2. If you are persisting the final state of each **run** as a dataclip, each + dataclip may not exceed `10MB`. +3. If you are persisting the final state of an **attempt** as a dataclip, it may + not exceed `10MB`. -If you send a larger payload to a webhook URL, the server will respond with a -`413: request_entity_too_large` error. If the dataclips produced by the final state of -runs and attempts are too large, you will see _____________________________________. +If you send a payload to a webhook trigger URL which breaches this limit, the +server will respond with a `413` error with a `:request_entity_too_large` +message. If the dataclips produced by the final state of runs and attempts are +too large, you will see {TODO: Set text for dataclip size warning}. -These limits can be adjusted by an instance superuser via the -`MAX_DATACLIP_SIZE` environment variable. +> _Instance superusers can control this limit via the `MAX_DATACLIP_SIZE` +> environment variable._ From 9908c7b05755550a9c9c610e94d0a04e94726910 Mon Sep 17 00:00:00 2001 From: Taylor Downs Date: Tue, 28 Nov 2023 08:25:29 +0000 Subject: [PATCH 4/4] more --- docs/jobs/limits.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/jobs/limits.md b/docs/jobs/limits.md index 84f32516d0c..9d343fa99a0 100644 --- a/docs/jobs/limits.md +++ b/docs/jobs/limits.md @@ -13,7 +13,7 @@ locally deployment instance. ::: -## Workflow execution duration (1hr) +## Workflow execution duration (1 hour) Each workflow attempt needs to complete in less than `1 hour`. You can view the duration of each attempt by clicking on the attempt ID. If an attempt exceeds @@ -44,7 +44,11 @@ attempt exceeds this limit, it will be killed by the worker and you'll see a If you send a payload to a webhook trigger URL which breaches this limit, the server will respond with a `413` error with a `:request_entity_too_large` message. If the dataclips produced by the final state of runs and attempts are -too large, you will see {TODO: Set text for dataclip size warning}. +too large, you will no be able to access them and instead see: + +```json +{ "__lightning": "Run result too large to save" } +``` > _Instance superusers can control this limit via the `MAX_DATACLIP_SIZE` > environment variable._