Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Katalogus caching in the scheduler #3357

Open
jpbruinsslot opened this issue Aug 13, 2024 · 1 comment · May be fixed by #4108
Open

Katalogus caching in the scheduler #3357

jpbruinsslot opened this issue Aug 13, 2024 · 1 comment · May be fixed by #4108
Assignees
@jpbruinsslot @Donnype
Labels
katalogus mula Issues related to the scheduler tech-debt

Comments

@jpbruinsslot
Copy link
Contributor

jpbruinsslot commented Aug 13, 2024 •
edited
Loading

The sophisticated caching and filtering in the scheduler can perhaps be optimized by:

  • Adding more API filtering options in the Katalogus that would resolve creating 3 separate caches in the scheduler
  • Adding ETag (Entity Tag) or Last-Modified headers to katalogus to invalidate internal katalogus state in the scheduler instead of relying on the ExpiringDict functionality
  • Potentially resolve Scheduler katalogus freshness #856 since the katalogus state in the scheduler will always be updated to the latest modified timestamp
  • Retrieval of new boefjes is now a call per organisation to diff internal cache of present boefjes to enabled boefjes. Could be optimized by a signal to the scheduler when a boefje gets enabled/added

Related: #856
@jpbruinsslot jpbruinsslot added mula Issues related to the scheduler tech-debt katalogus labels Aug 13, 2024
@jpbruinsslot jpbruinsslot self-assigned this Aug 13, 2024
This was referenced Oct 29, 2024
Merged
Open
@underdarknl
Copy link
Contributor

In general, I want to move in the direction of storing the enabled/configured plugins in XTDB, instead of the katalogus.
For the following point:

* Retrieval of new boefjes is now a call per organisation to diff internal cache of present boefjes to enabled boefjes. Could be optimized by a signal to the scheduler when a boefje gets enabled/added

When we move the plugin config to XTDB, octopoes should already send out signals to the Scheduler for changes to OOI's. We can use the same mechanism to pick up changes on plugins.

@jpbruinsslot jpbruinsslot mentioned this issue Dec 12, 2024
Merged
9 tasks
@Donnype Donnype self-assigned this Feb 6, 2025
@Donnype Donnype added this to KAT Feb 12, 2025
@github-project-automation github-project-automation bot moved this to Incoming features / Need assessment in KAT Feb 12, 2025
@Donnype Donnype moved this from Incoming features / Need assessment to In Progress in KAT Feb 12, 2025
@jpbruinsslot jpbruinsslot linked a pull request Feb 19, 2025 that will close this issue
Remove caches for the KATalogus in the scheduler #4108
Open
9 tasks
@jpbruinsslot jpbruinsslot moved this from In Progress to Backlog / To do in KAT Feb 20, 2025
@jpbruinsslot jpbruinsslot linked a pull request Feb 20, 2025 that will close this issue
Open
9 tasks
@underdarknl underdarknl moved this from Backlog / To do to Review in KAT Feb 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jpbruinsslot jpbruinsslot

@Donnype Donnype

Labels
katalogus mula Issues related to the scheduler tech-debt
Projects
KAT
Status: Review
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove caches for the KATalogus in the scheduler minvws/nl-kat-coordination
3 participants
@jpbruinsslot @underdarknl @Donnype