-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Navigate Pipeline via Pipes instead of Blocks #491
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I left some comments for barely relevant things outside of the scope of this PR (e.g. GTFSExtractor
and I am happy if you ignore them.
I am however happy to die on the hill of using DAG naming (parents
, children
etc) so I'd ask for reverting the renaming to followIngoing/OutgoingPipes
.
-> TripsFilePicker | ||
-> TripsTextFileInterpreter | ||
-> TripsCSVInterpreter | ||
-> TripsTableInterpreter | ||
-> TripsLoader; | ||
|
||
// 3. As a first step, we download the zip file and interpret it. | ||
block GTFSSampleFeedExtractor oftype HttpExtractor { | ||
block GTFSSampleFeedExtractor oftype GTFSExtractor { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I know this is not part of the PR but this seems very boring. We could enhance the GTFSExtractor
in the std lib to also include picking a file and parsing it as CSV, maybe as a composite block inside a composite block.
The new block could be called GTFSFileExtractor
and take an url
and a path
as property, then use the existing GTFSExtractor
with the url and the FilePicker
with the file, then parse that file as CSV file. This would reduce this example by 3 blocks and 3 lines for every file (so 33 less blocks and 33 less lines in the pipeline itself!).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you don't mind I'd leave it as it is for now.
But your example looks interesting, also in the face of optimizations. In the best case, we would also reuse executed blocks multiple times. I'm not sure if that would be the case in the current implementation, so I'd leave it open for a follow-up issue.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I agree on all counts, it's not in the scope of this PR anyway.
libs/language-server/src/lib/validation/checks/block-definition.ts
Outdated
Show resolved
Hide resolved
33f095d
to
cda9e9a
Compare
export class PipelineWrapper | ||
implements AstNodeWrapper<PipelineDefinition | CompositeBlocktypeDefinition> | ||
{ | ||
public readonly astNode: PipelineDefinition | CompositeBlocktypeDefinition; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought about transforming it into the following so that astNode is typed better. I'm not sure if we have a benefit from that though.
export class PipelineWrapper | |
implements AstNodeWrapper<PipelineDefinition | CompositeBlocktypeDefinition> | |
{ | |
public readonly astNode: PipelineDefinition | CompositeBlocktypeDefinition; | |
export class PipelineWrapper< | |
T extends PipelineDefinition | CompositeBlocktypeDefinition, | |
> implements AstNodeWrapper<T> | |
{ | |
public readonly astNode: T; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome, seems to work now after removing the SimplePipe
syntax in #487
Until now, we have navigated over all blocks within a pipeline and connected them via pipes to get our execution order.
However, #485 will change that by allowing block definitions outside of pipelines, raising the necessity to iterate purely over the pipes with a pipeline.
This PR introduces a
PipelineWrapper
in the already existing ofAstNodeWrapper
s that inhibits the functionality to navigate over a pipeline and replaces the existing helper methods in themodel-util
.The PR should not change the behavior of Jayvee besides:
TODOs:
Pipelines
CompositeBlockTypes
(=> showcased by gtfs-static example)