Allow a workflow to support multiple adaptor versions #610
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Basically instead of looking up the adaptor version from the global list of options, we calculate the version for each step and pass that through to execution
This lets the runtime use the repo to install the correct versions of adaptors. It might run a little slower though
|
Likely issue: I think the module-overiding thing is removing path information. So if you specify an adaptor path in the linker options, it'll get blatted out and ignored. This broke the engine and will be a problem in some CLI edge cases. I've also preferred repoDir to adaptorPaths in the engine, but I wonder if this is really the right thing to do. It might be a bit slower. |
josephjclark
commented
Feb 23, 2024
josephjclark
commented
Feb 23, 2024
josephjclark
commented
Feb 23, 2024
|
So this is good to go... but we can't release it until we do SOMETHING about version reporting. At the moment the adaptor version, as currently reported, is basically a lie. So I'm going to branch off and fix #574 |
josephjclark
added a commit
that referenced
this pull request
Feb 23, 2024
* runtime: allow each job in a workflow to use a different adaptor version Basically instead of looking up the adaptor version from the global list of options, we calculate the version for each step and pass that through to execution * tests: add test of multiple versions * engine: prefer repoDir to adaptorPaths This lets the runtime use the repo to install the correct versions of adaptors. It might run a little slower though * runtime: allow linker options to be passed on each job * runtime: better mreging of linker options * engine: update autoinstall to write paths to the plan
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #268
Bit tricky this one! But I think I've got it.
Every step in a workflow gets compiled by the runtime manager (the CLI or the Engine) and is given an import statement
import { fn } from '@openfn/language-common'.It's up to the linker, deep inside the runtime, to decide how how resolve this import. Should we look in the repo? Were we given an explicit mapping? What version do we want?
What usually happens is that the runtime manager (the CLI or worker) will pre-parse the workflow and make a bunch of decisions about these imports. In the CLI, for example, the user might add an explicit path to
language-common. Or if someone asked for@latest, the CLI will maybe autoinstall and replace@latestwith an actual version (I think).The problem is that one list of adaptor-version mappings is compiled for the whole workflow, and that one list is shared by the linker. So if we decide that
language-commonis 1.0, all steps will load 1.0 regardless of what was asked for.What this PR does is calculate this module map for each step inside the runtime based on the original input, and passes this "localised" map into the linker for each step.
It seems to work.
Sadly I can't get unit tests very close to the code without making major changes. It's actually very small tweak in a glue layer and I can't intercept in the right place.
TODO