Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add deploying ingestion specs from custom resources #226

Open
soenkeliebau opened this issue May 3, 2022 · 0 comments
Open

Add deploying ingestion specs from custom resources #226

soenkeliebau opened this issue May 3, 2022 · 0 comments

Comments

@soenkeliebau
Copy link
Member

soenkeliebau commented May 3, 2022

Implementation ticket for #168

The Druid operator should be extended to be able to deploy ingestion specs from definitions provided in CRDs.

Ingestion specs can be defined by the user via a customresource, which is watched by a controller in the Druid operator that then provisions these specs.

The crd will contain at least the following:

  • type of source (stream, flat file, ...)
  • failure behavior
    • none
    • retry
    • delete and retry
  • specification of the ingest job
    • inline
    • reference to configmap
    • reference to pvc (or file therein)

These objects will initially be considered read-only, so changes to them will not be propagated to Druid by the controller.

The initial implementation will not be a perfect ingestion task management solution, but rather a first attempt to offer something useful to our users.
The user needs to decide themselves what the appropriate failure behavior is for the spec they provide to the operator, whether duplicate data might be created by retrying etc.
The defined failure options should offer simple solutions for all scenarios:

  • task is too complex -> none , the user will investigate themselves
  • no duplicates expected -> retry , task is idempotent and can be retried
  • duplicates possible -> delete and retry , not idempotent, to be on the safe side delete the target before retrying
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant