Drop SqlJob and FlinkSqlJobReconciler #35
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Removed the SqlJob CRD, models, and related controller.
The idea behind the SqlJob CRD was to have non-functional metadata in Kubernetes for generic jobs, irrespective of an underlying implementation and runtime (e.g. Flink vs Spark). Then, a controller or external operator could translate SqlJobs into something concrete and deployable, e.g. FlinkDeployments, as was done by
FlinkSqlJobReconciler
.However, this meant that every runtime (e.g. Flink vs Spark) would need to implement such a controller, even if they simply translate SqlJobs into something else. To avoid a lot of duplicate controller logic between runtimes, I'm dropping this intermediate CRD. Instead, the planner can generate concrete, deployable objects (e.g. FlinkDeployments) directly.
This simplifies several things without loss of power. I've opened #34 to track a newly relevant limitation: the planner can currently only generate one YAML object for each Resource, whereas a custom controller could create any number of downstream objects. But technically, an adapter could still implement a custom controller if needed, so we don't lose anything in the interim. And once #34 is fixed, we will be able to generate any number of downstream objects trivially -- without writing an entire controller.
Notice how simple the new
hoptimator-flink-adapter
is! It merely:SqlJob.yaml.template
which generatesFlinkDeployments
.flinkdeployments
).I've added the flink adapter to the CLI s.t.
!yaml
will showFlinkDeployments
instead of some genericSqlJob
.Testing
The
!yaml
command now generatesFlinkDeployments
instead of a non-functional placeholderSqlJob
object: