Add concurency configuration option for keycloak backend

[AndrienkoAleksandr]

Hey, I just made a Pull Request!

What does this pr fix:

janus-idp/backstage-plugins#2471
https://issues.redhat.com/browse/RHIDP-4793

Demo:

https://youtu.be/miQ9DgaOYcg

✔️ Checklist

• A changeset describing the change and affected packages. (more info)
• Added or updated documentation
• Tests for new functionality and regression tests for bug fixes
• Screenshots attached (for UI changes)
• All your commits have a Signed-off-by line in the message. (more info)

[backstage-goalie]

Changed Packages

Package Name	Package Path	Changeset Bump	Current Version
@backstage-community/plugin-catalog-backend-module-keycloak	workspaces/keycloak/plugins/catalog-backend-module-keycloak	patch	v3.4.0

[dzemanov]

@AndrienkoAleksandr Can you please generate changeset when you will have time?

[dzemanov]

Tested with 1000 groups and 500000 users, works as expected.
Interestingly, each scheduled run more users are commited, e.g:

Committed 27173 Keycloak users and 1000 Keycloak groups
...
Committed 31119 Keycloak users and 1000 Keycloak groups
...
Committed 42271 Keycloak users and 1000 Keycloak groups
...
Committed 43885 Keycloak users and 1000 Keycloak groups
...
Committed 44735 Keycloak users and 1000 Keycloak groups
...

So it will take more scheduled runs to commit all users in Keycloak.

Tested also locally as dynamic plugin in backstage-showcase.

[AndrienkoAleksandr]

Tested with 1000 groups and 500000 users, works as expected.
Interestingly, each scheduled run more users are commited, e.g:
Committed 27173 Keycloak users and 1000 Keycloak groups
...
Committed 31119 Keycloak users and 1000 Keycloak groups
...
Committed 42271 Keycloak users and 1000 Keycloak groups
...
Committed 43885 Keycloak users and 1000 Keycloak groups
...
Committed 44735 Keycloak users and 1000 Keycloak groups
...
So it will take more scheduled runs to commit all users in Keycloak.

@dzemanov, Yes, I agree that this could happen if you set a very high maxConcurrency value. I recorded a demo where I demonstrated that, with a reasonable maxConcurrency value, you can fetch 5,000,000 users and 1,000 groups in the first scheduled run iteration and commit all the data at once. Subsequent scheduled runs should also succeed without losing users or groups. Lost users and groups could occur, but only if you encounter actual network issues.

P.S.: Thank you for your comment. I have changed the default maxConcurrency value; it is no longer set to infinity. I have provided a default value of 20. I'm not sure if this is the optimal default, but it is much better than infinity, which could definitely lead to a DoS attack on the Keycloak server.

[awanlin]

Thanks for the contribution @AndrienkoAleksandr, I can't confidently review the actual code changes as I don't know this plugin very well and will leave that to other reviewers but I did notice some more general items that need to be changed. 👍

[AndrienkoAleksandr]

@awanlin, I handled your code review feedback.

[AndrienkoAleksandr]

@AndrienkoAleksandr Can you please generate changeset when you will have time?

Thanks @dzemanov, done.

[awanlin]

Thanks for following up @AndrienkoAleksandr, there's a few structural items the need to be addressed still, sorry.

[awanlin]

This looks to be in a good state now, would still like someone with more knowledge about the plugin itself to review so not giving an explicit approval till that happens.

@schultzp2020, can you look at this? Just asking as it looks like you got tagged to review 👍

[04kash]

This looks to be in a good state now, would still like someone with more knowledge about the plugin itself to review so not giving an explicit approval till that happens.

Maybe @PatAKnight or @dzemanov?

[dzemanov]

Other then minor comment, looks good to me.

[awanlin]

Hi @AndrienkoAleksandr, looks like we need to resolve the conflict and address the comment from @dzemanov 👍

[AndrienkoAleksandr]

Hi @AndrienkoAleksandr, looks like we need to resolve the conflict and address the comment from @dzemanov

Done.

[PatAKnight]

A simple and quick question while I prepare to test

[dzemanov]

Tested again and works fine. Great work @AndrienkoAleksandr 🪄

[dzemanov]

lgtm!

[dzemanov]

@awanlin can you approve the PR when you get a chance?

[04kash]

All review comments were addressed, lgtm!

[AndrienkoAleksandr]

@awanlin can you remove block from this pr, please ?

[awanlin]

Cleared out on my end! 🚀

@04kash, will leave this to you to merge 👍

resolved