Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop SqlJob and FlinkSqlJobReconciler #35

Merged
merged 4 commits into from
Jun 23, 2023
Merged

Drop SqlJob and FlinkSqlJobReconciler #35

merged 4 commits into from
Jun 23, 2023

Conversation

ryannedolan
Copy link
Collaborator

Removed the SqlJob CRD, models, and related controller.

The idea behind the SqlJob CRD was to have non-functional metadata in Kubernetes for generic jobs, irrespective of an underlying implementation and runtime (e.g. Flink vs Spark). Then, a controller or external operator could translate SqlJobs into something concrete and deployable, e.g. FlinkDeployments, as was done by FlinkSqlJobReconciler.

However, this meant that every runtime (e.g. Flink vs Spark) would need to implement such a controller, even if they simply translate SqlJobs into something else. To avoid a lot of duplicate controller logic between runtimes, I'm dropping this intermediate CRD. Instead, the planner can generate concrete, deployable objects (e.g. FlinkDeployments) directly.

This simplifies several things without loss of power. I've opened #34 to track a newly relevant limitation: the planner can currently only generate one YAML object for each Resource, whereas a custom controller could create any number of downstream objects. But technically, an adapter could still implement a custom controller if needed, so we don't lose anything in the interim. And once #34 is fixed, we will be able to generate any number of downstream objects trivially -- without writing an entire controller.

Notice how simple the new hoptimator-flink-adapter is! It merely:

  1. includes a SqlJob.yaml.template which generates FlinkDeployments.
  2. registers the corresponding API endpoint (flinkdeployments).

I've added the flink adapter to the CLI s.t. !yaml will show FlinkDeployments instead of some generic SqlJob.

Testing

The !yaml command now generates FlinkDeployments instead of a non-functional placeholder SqlJob object:

0: hoptimator> !yaml SELECT * FROM RAWKAFKA."products"
...
---
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  namespace: {{namespace}}
  name: {{name}}-flink-job
...

@ryannedolan ryannedolan enabled auto-merge (squash) June 22, 2023 15:01
@ryannedolan ryannedolan merged commit d5c4edd into main Jun 23, 2023
@ryannedolan ryannedolan deleted the drop-sqljob branch June 23, 2023 17:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants