Backend renaming (1.0 version bumps plus the lexicon) #585
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
there are cases where job is more accurate and useful
Done a big chunk of execute but still a way to go
This is 90% of the basic rename done. Tests may even pass
…ow specific If a workflow wants to remove props, it'll add an fn bock
Merged
|
Hi @mtuchi when you have a spare half an hour would you mind taking a look at the CLI on this branch? There's some stuff about terminology in the CLI readme - probably a good idea to look at that first. You should look closely at the log output and make sure that it aligns with your documentation. Also if workflows and steps have names, they should be used in the log output now (instead of UUIDS), although that likely affects the worker more than the CLI |
10 tasks
|
@taylordowns2000 @stuartc can I ask you to take a quick look at the core typings on this? I've created a central types and definitions repo called You may want to check the actual typings too. No changes to the Lightning-Worker contract are introduced in this PR - the former "attempt" model is still what the Worker expects to receive from Lightning. But now we convert the Run into an Execution plan, and everything down the stack expects to use that data structure. |
josephjclark
commented
Feb 8, 2024
| @@ -0,0 +1,42 @@ | |||
| The lexicon (aka the OpenFunctionicon) is a central repositoty of key type and word definitions. | |||
There was a problem hiding this comment.
@taylordowns2000 @stuartc It's a big diff so here's a shortcut to the lexicon - really I'd just like you to approve this bit. All other diffs are just alignments to core.d.ts and lightning.d.ts
* worker: send worker and API versions to Lightning * lexicon: fix API_VERSION export
stuartc
approved these changes
Feb 9, 2024
|
Thanks @stuartc ! |
josephjclark
added a commit
that referenced
this pull request
Feb 15, 2024
* engine: spike mapping console logs to an adaptor logger
* runtime: messy tweak to module loading
* engine,runtime: revert linker change and fix tests
* engine: track test file
* logger: dont stringify json output AND serialize errors
This cause problems with the worker because errors get flattened to {}, and also we have to double parse.
Now the logger will just emit whatever it logged to whatever the log emmiter is, so JSON stays as JSON. Which is good, but it no longer guarantees it'll be serializable
* logger: tidy
* engine: don't parse json logs coming out of the logger
* engine, worker: better handling of objects coming from the logger
The logger always sends raw json, but the log message is stringified by the engine, and rebuilt by the worker before sending to lightning
this last bit needs work but its better
* engine: fix tests
* logger: tests and types
* cli: update test
* engine: types
* worker: update tests
* logger: set a special json emitter so that json logs get nicely printed in the CLI
* logger: fix types
* logger: log all json to .log
* tests: fixes
* logger: fix tests
* logger: serialise print() properly
* logger: types
* engine: fix logs to gcp
They were neglecting to parse the strings sent out by the new json logger
* test: update log handling
* engine: fix passing test
It was secretly failing under the hood
* runtime: add tests on job logger and errors
* logger: improve detection of error objects
* engine: tests on error logging
* engine: restore adaptor logger
* changesets
* Tidy ups
* engine: refactor log messages (and be a bit more lenient about structure)
* worker: simplify logging
* tiny tidyups
* remove old docs
* lexicon: start building a central lexicon of definitions
* runtime: huge refactor of runtime core API
* runtime: more refactoring
* runtime: take initial state out of the execution plan
* fix tests
* runtime: changeset
* runtime: extra type tweakings
* runtime: readme
* runtime: jobs -> steps (mostly)
there are cases where job is more accurate and useful
* cli: start refactoring towrads new runtime API
Done a big chunk of execute but still a way to go
* cli: basically get the CLI working again
* cli: types
* cli: fix a bunch of tests, update workflow parsing
* cli: fix execute and compile tests
* cli: more test fixes
* fix more cli tests
* cli: fix integration tests
* cli: tidy
* runtime: remove strict mode
* remove strict mode
* cli: default workflow name to the file name
* runtime: tweak log output
* cli: remove log
* cli: types
* docs
* deploy: adjust logging
* engine: update types
* engine: update names and types
This is 90% of the basic rename done. Tests may even pass
* runtime: male statePropsToRemove a system options, rather than workflow specific
If a workflow wants to remove props, it'll add an fn bock
* engine: restore statePropsToRemove tests
* mock: update to lexicon
* worker: start mapping to lexicon. Handled run-> plan conversion
* worker: typings
* worker: fix all tests
* engine: types
* worker: fix cheeky test
somehow missed it last time
* tests: fix cli tests
* worker: update test
* package lock
* tests: update test
* changesets and housekeeping
* more housekeeping
* engine: tweak test
* runtime: tweak error messages
* worker: stricter type checkign on tests
* fix test
* typing in worker tests
* worker: update channel mock
* lexicon: docs
* Run -> LightningPlan
* version bumps for logger and mock
* mock: return error if dataclip not found
* worker: better handling of dataclip errors
* lightning-mock: fix test
* worker: changeset
* worker: fix test
Don't return the loaded dataclip after the refactor
* worker: fix test again
* Backend renaming (1.0 version bumps plus the lexicon) (#585)
* lexicon: start building a central lexicon of definitions
* runtime: huge refactor of runtime core API
* runtime: more refactoring
* runtime: take initial state out of the execution plan
* fix tests
* runtime: changeset
* runtime: extra type tweakings
* runtime: readme
* runtime: jobs -> steps (mostly)
there are cases where job is more accurate and useful
* cli: start refactoring towrads new runtime API
Done a big chunk of execute but still a way to go
* cli: basically get the CLI working again
* cli: types
* cli: fix a bunch of tests, update workflow parsing
* cli: fix execute and compile tests
* cli: more test fixes
* fix more cli tests
* cli: fix integration tests
* cli: tidy
* runtime: remove strict mode
* remove strict mode
* cli: default workflow name to the file name
* runtime: tweak log output
* cli: remove log
* cli: types
* docs
* deploy: adjust logging
* engine: update types
* engine: update names and types
This is 90% of the basic rename done. Tests may even pass
* runtime: male statePropsToRemove a system options, rather than workflow specific
If a workflow wants to remove props, it'll add an fn bock
* engine: restore statePropsToRemove tests
* mock: update to lexicon
* worker: start mapping to lexicon. Handled run-> plan conversion
* worker: typings
* worker: fix all tests
* engine: types
* worker: fix cheeky test
somehow missed it last time
* tests: fix cli tests
* worker: update test
* package lock
* tests: update test
* changesets and housekeeping
* more housekeeping
* engine: tweak test
* runtime: tweak error messages
* worker: stricter type checkign on tests
* fix test
* typing in worker tests
* worker: update channel mock
* lexicon: docs
* Run -> LightningPlan
* version bumps for logger and mock
* Send worker versions (#593)
* worker: send worker and API versions to Lightning
* lexicon: fix API_VERSION export
* cli: dont print compiler,runtime versions, also show monorepo for adaptor
* cli tweak output to optionally show components
* worker: simplify version output
* mock: resolve conflict
* Autoinstall by default (#594)
* lexicon: start building a central lexicon of definitions
* runtime: huge refactor of runtime core API
* runtime: more refactoring
* runtime: take initial state out of the execution plan
* fix tests
* runtime: changeset
* runtime: extra type tweakings
* runtime: readme
* runtime: jobs -> steps (mostly)
there are cases where job is more accurate and useful
* cli: start refactoring towrads new runtime API
Done a big chunk of execute but still a way to go
* cli: basically get the CLI working again
* cli: types
* cli: fix a bunch of tests, update workflow parsing
* cli: fix execute and compile tests
* cli: more test fixes
* fix more cli tests
* cli: fix integration tests
* cli: tidy
* runtime: remove strict mode
* remove strict mode
* cli: default workflow name to the file name
* runtime: tweak log output
* cli: remove log
* cli: types
* docs
* deploy: adjust logging
* engine: update types
* engine: update names and types
This is 90% of the basic rename done. Tests may even pass
* runtime: male statePropsToRemove a system options, rather than workflow specific
If a workflow wants to remove props, it'll add an fn bock
* engine: restore statePropsToRemove tests
* mock: update to lexicon
* worker: start mapping to lexicon. Handled run-> plan conversion
* worker: typings
* worker: fix all tests
* engine: types
* worker: fix cheeky test
somehow missed it last time
* tests: fix cli tests
* worker: update test
* package lock
* tests: update test
* changesets and housekeeping
* more housekeeping
* engine: tweak test
* runtime: tweak error messages
* worker: stricter type checkign on tests
* fix test
* typing in worker tests
* worker: update channel mock
* lexicon: docs
* Run -> LightningPlan
* version bumps for logger and mock
* cli: autoinstall by default
* cli: docs
* changeset
* cli: fix tests
Need to disable autoinstall now or some tests will blow up!
* openfnx: update console output
* runtime: fix tests
* worker: support output_dataclips on run options
* worker: additioonal test of output_dataclips
* types
* mock: error if a credential does not exist
* engine: throw nice exception if credentials fail to load
* tests: add tset for bad credential
* worker: bad credential test
* mock: update dev endpoint to allow invalid credentials
* changeset
* worker: move tesdt into reasons
* worker: tweak logs
* Verify run token (#598)
* worker: start trying to verify the attempt token
* worker: roughly verify the run token
* mock: generate a real jwt for runs
* mock: tweak key handling
* worker: verify the run token
* changesets
* todo
* worker: support public key from env
* worker: better cli handling
* error handling
* worker: destroy server if run token is invalid
* test: add integration test for errors
* tests: add keys to more tests
* test: fix privateKey
* tidyups
* more tidyups
* version lock pheonix to 1.7.10
1.7.11 introduces a compatability issue
* logger: add proxy function to the mock
* engine: don't send adaptor logs to stdout
* tests: add test for adaptor logs
* changeset
* tests: remove logging
* types
* logger: rethink mock proxy. It's still not working.
* logger: fix mock proxy function
* engine: fix tests
* tests: update tests
* worker: fixed a tricky issue with server shutdown
If a server is destroyed before the lightning connection returned, the workloop will still fire even if the server is technically destroyed
* package lock
* package lock
* tests: tweak output
* tests: run serially
* tests: reorganise
* version: [email protected] [email protected]
---------
Co-authored-by: Taylor Downs <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR bumps the runtime up to 1.0 and introduces the Lexicon, a central repo for typings and namings.
I have started to enforce stronger type checking on the tests as part of this PR, which has helped to catch any issues in the change of formats but also resulted in a large diff!
Closes #483 #556 #409 #233
Runtime changes:
CLI changes
Engine Changes
Worker Changes