The OpenFn Runtime used by Lightning is a Node.js application that runs on a server. It is responsible for executing the code that you write in your OpenFn jobs.
A Runtime Manager is a module or application that processes Runs, which contains reference to the Workflow, the starting point and the initial data.
Runs are enqueued, and Runtime Managers request work to be performed when they are ready.
In previous versions of OpenFn products, the server would invoke a NodeJS child process for each Run that needs to be executed. Deciding which Job to run in a workflow is decided after each run is completed.
The current approach to executing Runs, is that a worker checks out the entire run and executes it and all the jobs required.
The advantage of this approach is a significant reduction in latency of launching new workers for every Run to be processed.
Our standard Runtime Manager uses WebSockets to communicate with Lightning.
However, since the workers are treated no differently from any other incoming traffic; there are two layers of security applied to anything that can access your data or process work.
We use two different JWTs to facilitate the follow:
This token is created by the worker and signed with a shared HS256 secret that Lightning uses to verify.
This token is created by Lightning for every Run that a worker receives. The token is signed with a private key, and therefore cannot be generated by 3rd parties.
The token is scoped for a single Run and cannot be used to access anything outside of the scope of that Run.
Lightning comes with a mix task to help generate keys:
mix lightning.gen_worker_keys
There are three environment variables required to make everything work.
WORKER_RUNS_PRIVATE_KEY
Lightning only
A Base64 encoded private key in PEM format, used to generate JWTs granting workers permission to access a specific Run, the related Workflow and credentials.
[!IMPORTANT] This key should never be shared. If you suspect it has been compromised, generate a new one and reconfigure both Lightning and your workers.
WORKER_SECRET
Lightning and Workers
A 256bit long shared secret used by the worker to sign JWTs for authentication with the Lightning API.
WORKER_LIGHTNING_PUBLIC_KEY
Workers only
A Base64 encoded public key in PEM format derived from the private key. This is used by workers to verify Run Tokens coming from Lightning.
NodeJS v17 changed how DNS name
resolution is handled. In earlier versions, Node would reorder DNS lookup
results so IPv4 addresses came before IPv6 addresses. In v17, Node started
returning addresses in the same order provided by the resolver. The net effect
is that your workers might fail to connect to Lightning if the native DNS
resolver on your system is returning the loopback IPv6 address (::1) before
the IPv4 address (127.0.0.1). In these cases, your worker might report an
error similar to the following:
CRITICAL ERROR: could not connect to lightning at ws://localhost:4000/worker
If you experience this, it can be helpful to set the NODE_OPTIONS environment
variable as shown below. This will force pre-v17 behavior of returning IPv4
addresses first.
NODE_OPTIONS=--dns-result-order=ipv4first
You can directly inspect what IP address Node returns for localhost by running
the following command:
node -e 'dns.lookup("localhost", console.log)'