Use local adaptors in a local Lightning instance

[amberrignell]

User story

As a user that has built a new adaptor locally, I would like to spin up Lightning and use that adaptor, so that I can see it in action and properly test it.

Details

Implementation notes

Release notes

User acceptance criteria

[josephjclark]

Here is a design for this:

When Lightning populates the adaptor list, it should do some from two possible sources: npm or the monorepo.

If Lightning detects the OPENFN_ADAPTORS_REPO env var (which is is a path to the repo and is used by the CLI to interface with the monorepo), it should enable the monorepo source.

Any adaptor version found in the monorepo should be displayed at the TOP of the picklist for new adaptors, with version set to local. This internally needs to set the version on the Run to @local (ie, @openfn/language-common@local). That is, the pick list label should be local and the value should be @openfn/language-common@local.

The monorepo source needs to be populated by looking in the packages/ folder of the adaptors monorepo. Every folder inside packages is an adaptor which should be added to the list of options.

In order to support this, the Worker will need to have special handling for the @local version. Not a terribly big deal, see OpenFn/kit#806

[josephjclark]

@stuartc I have (almost) implemented support for local/monorepo adaptors in the worker in #808

tl;dr: if lightning sets the adaptor version to @local on a given step, the worker will try to load it from the monorepo

This has been useful for collections so I just ploughed on with it

Edit: that's merged. So all we need to do is update the picklist with an @local option and we can support local adaptor dev in lightning

[christad92]

• Update the adaptor picklist with an @local option and we can support local adaptor dev in lightning

[josephjclark]

Hey, I've been thinking about a different (cheaper) design for this

Maybe we can start the worker in a mode that ONLY uses the monorepo to load adaptors. Driven by an env var maybe.

So if you do:

LOCAL_ADAPTORS=true mix phx.server

Lightning will start the worker with a flag that forces all adaptors to be local.

The advantage is that the dev effort is much less. No need to faff around with this @local stuff. It is ARGUABLY better for users to - suddenly everything runs out of your monorepo no matter what version is set in your workflow. You can argue the toss but maybe it's a cleaner experience. More predictable.

Counter-argument to that: if you're running with the monorepo you have to remember to keep your local build up to date. It's very easy to forget to build and run an old version of an adaptor by accident. That's why I like the @Local approach - it's more faff (for us and for users) but it's very explicit.

The disadvantage is that Lightning is lying to users. The workflow will say like "this step uses http 6.0", but actually the runtime log will say "I'm using whatever is in your monorepo". That's kind of confusing and maybe a bit tricky. And compounded by the problem of forgetting to rebuild. Suddenly it's not actually clear what version of the adaptor you're even running.

I suppose we could do something like: if the LOCAL_ADAPTORS env var is set, we don't show a version picklist in the UI. That's a bit more honest.

Maybe the question is: for a developer using lightning to test adaptors, do I want to set an individual job to use a local adaptor, or do I just want to always use local adaptors?