forked from adorsys/keycloak-config-cli
-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
This is a squash of the commits from multiple repositories: Original PR by sonOfRa: adorsys#818 Adaptations by thomasdarimont: https://github.com/thomasdarimont/keycloak-config-cli/tree/feature/realm-normalization Fixes adorsys#799 Co-authored-by: Simon Levermann <[email protected]> Signed-off-by: Thomas Darimont <[email protected]>
- Loading branch information
1 parent
292a4f0
commit f53a6be
Showing
99 changed files
with
12,499 additions
and
508 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,3 +19,4 @@ release.properties | |
|
|
||
| /*.json | ||
| /test* | ||
| /exports | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| # Realm normalization | ||
|
|
||
| Realm normalization is a feature that is supposed to aid users in migrating from an "unmanaged" Keycloak installation, | ||
| to an installation managed by keycloak-config-cli. | ||
| To achieve this, it uses a full [realm export](https://www.keycloak.org/server/importExport#_exporting_a_specific_realm) | ||
| as an input, and only retains things that deviate from the default | ||
|
|
||
| ## Usage | ||
|
|
||
| To run the normalization, run keycloak-config-cli with the CLI option `--run.operation=NORMALIZE`. | ||
| The default value for this option is `IMPORT`, which will run the regular keycloak-config-cli import. | ||
|
|
||
| ### Configuration options | ||
|
|
||
| | Configuration key | Purpose | Example | | ||
| |--------------------------------------|-----------------------------------------------------------------------------------------------------------------|---------------| | ||
| | run.operation | Tell keycloak-config-cli to normalize, rather than import | NORMALIZE | | ||
| | normalization.files.input-locations | Which realm files to import | See IMPORT.md | | ||
| | normalization.files.output-directory | Where to save output realm files | ./exports/out | | ||
| | normalization.output-format | Whether to output JSON or YAML. Default value is YAML | YAML | | ||
| | normalization.fallback-version | Use this version as a baseline of keycloak version in realm is not available as baseline in keycloak-config-cli | 19.0.3 | | ||
|
|
||
| ### Unimplemented Features | ||
| - Components: | ||
| - Currently, keycloak-config-cli will not yet look at the `components` section of the exported JSON | ||
| - Therefore, some things (like LDAP federation configs and Key providers) are missing from the normalized YAML | ||
| - Users | ||
| - Users are not currently considered by normalization. | ||
|
|
||
| ## Missing entries | ||
| keycloak-config-cli will WARN if components that are present in a realm by default are missing from an exported realm. | ||
| An example of such a message is: | ||
| ``` | ||
| Default realm requiredAction 'webauthn-register-passwordless' was deleted in exported realm. It may be reintroduced during import | ||
| ``` | ||
| Messages like these will often show up when using keycloak-config-cli to normalize an import from an older version of Keycloak, and compared to a newer baseline. | ||
| In the above case, the Keycloak version is 18.0.3, and the baseline for comparison was 19.0.3. | ||
| Since the webauthn feature was not present (or enabled by default) in the older version, this message is mostly informative. | ||
| If a message like this appears on a component that was *not* deleted or should generally be present, this may indicate a bug in keycloak-config-cli. | ||
|
|
||
| ## Cleaning of invalid data | ||
| Sometimes, realms of existing installations may contain invalid data, due to faulty migrations, or due to direct interaction with the database, | ||
| rather than the Keycloak API. | ||
| When such problems are found, we attempt to handle them in keycloak-config-cli, or at least notify the user about the existence of these problems. | ||
|
|
||
| ### SAML Attributes on clients | ||
| While not necessarily invalid, openid-connect clients that were created on older Keycloak versions, will sometimes contain | ||
| SAML-related attributes. These are filtered out by keycloak-config-cli. | ||
|
|
||
| ### Unused non-top-level Authentication Flows | ||
| Authentication flows in Keycloak are marked as top-level if they are supposed to be available for binding or overrides. | ||
| Authentication flows that are not marked as top-level are used as sub-flows in other authentication flows. | ||
| The normalization process recognizes recursively whether there are authentication flows that are not top level and not used | ||
| by any top level flow, and does not include them in the final result. | ||
|
|
||
| A warning message is logged, and you can use the following SQL query to find any authentication flows that are not referenced. | ||
| Note that this query, unlike keycloak-config-cli, is not recursive. | ||
| That means that after deleting an unused flow, additional unused flows may appear after the query is performed again. | ||
|
|
||
| ```sql | ||
| select flow.alias | ||
| from authentication_flow flow | ||
| join realm r on flow.realm_id = r.id | ||
| left join authentication_execution execution on flow.id = execution.auth_flow_id | ||
| where r.name = 'mytest' | ||
| and execution.id is null | ||
| and not flow.top_level | ||
| ``` | ||
|
|
||
| ### Unused and duplicate Authenticator Configs | ||
| Authenticator Configs are not useful if they are not referenced by at least one authentication execution. | ||
| Therefore, keycloak-config-cli detects unused configurations and does not include them in the resulting output. | ||
| Note that the check for unused configs runs *after* the check for unused flows. | ||
| That means a config will be detected as unused if it is referenced by an execution that is part of a flow that is unused. | ||
|
|
||
| A warning message is logged on duplicate or unused configs, and you can use the following SQL query to find any configs | ||
| that are unused: | ||
|
|
||
| ```sql | ||
| select ac.alias, ac.id | ||
| from authenticator_config ac | ||
| left join authentication_execution ae on ac.id = ae.auth_config | ||
| left join authentication_flow af on ae.flow_id = af.id | ||
| join realm r on ac.realm_id = r.id | ||
| where r.name = 'master' and af.alias is null | ||
| order by ac.alias | ||
| ``` | ||
|
|
||
| And the following query to find duplicates: | ||
|
|
||
| ```sql | ||
| select alias, count(alias), r.name as realm_name | ||
| from authenticator_config | ||
| join realm r on realm_id = r.id | ||
| group by alias, r.name | ||
| having count(alias) > 1 | ||
| ``` | ||
|
|
||
| If the `af.id` and `af.alias` fields are `null`, the config in question is not in use. | ||
| Note that configs used by unused flows are not marked as unused in the SQL result, as these need to be deleted first | ||
| to become unused. | ||
| After the unused flows (and executions) are deleted, the configs will be marked as unused and can also be deleted. | ||
|
|
||
| ### Authentication Executions with invalid subflows | ||
| Some keycloak exports have invalid authentication executions that reference a subflow, while also setting an authenticator. | ||
| This is only a valid configuration if the subflow's type is `form-flow`. | ||
| If it is not, then keycloak-config-cli will not import the configuration. | ||
| This will be marked by an ERROR severity message in the log output. | ||
| You can use this SQL query to find offending entries and remediate the configuration errors before continuing. | ||
|
|
||
| ```sql | ||
| select parent.alias, | ||
| subflow.alias, | ||
| execution.alias | ||
| from authentication_execution execution | ||
| join realm r on execution.realm_id = r.id | ||
| join authentication_flow parent on execution.flow_id = parent.id | ||
| join authentication_flow subflow on execution.auth_flow_id = subflow.id | ||
| where execution.auth_flow_id is not null | ||
| and execution.authenticator is not null | ||
| and subflow.provider_id <> 'form-flow' | ||
| and r.name = 'REALMNAME'; | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
126 changes: 126 additions & 0 deletions
126
src/main/java/de/adorsys/keycloak/config/KeycloakConfigNormalizationRunner.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,126 @@ | ||
| /*- | ||
| * ---license-start | ||
| * keycloak-config-cli | ||
| * --- | ||
| * Copyright (C) 2017 - 2022 adorsys GmbH & Co. KG @ https://adorsys.com | ||
| * --- | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| * ---license-end | ||
| */ | ||
|
|
||
| package de.adorsys.keycloak.config; | ||
|
|
||
| import com.fasterxml.jackson.databind.ObjectMapper; | ||
| import com.fasterxml.jackson.dataformat.yaml.YAMLMapper; | ||
| import de.adorsys.keycloak.config.properties.NormalizationConfigProperties; | ||
| import de.adorsys.keycloak.config.properties.NormalizationKeycloakConfigProperties; | ||
| import de.adorsys.keycloak.config.provider.KeycloakExportProvider; | ||
| import de.adorsys.keycloak.config.service.normalize.RealmNormalizationService; | ||
| import org.slf4j.Logger; | ||
| import org.slf4j.LoggerFactory; | ||
| import org.springframework.beans.factory.annotation.Autowired; | ||
| import org.springframework.boot.CommandLineRunner; | ||
| import org.springframework.boot.ExitCodeGenerator; | ||
| import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; | ||
| import org.springframework.boot.context.properties.EnableConfigurationProperties; | ||
| import org.springframework.stereotype.Component; | ||
|
|
||
| import java.io.FileOutputStream; | ||
| import java.nio.file.Files; | ||
| import java.nio.file.Paths; | ||
| import java.text.SimpleDateFormat; | ||
| import java.util.Date; | ||
|
|
||
| import static de.adorsys.keycloak.config.properties.NormalizationConfigProperties.OutputFormat.YAML; | ||
|
|
||
| @Component | ||
| @ConditionalOnProperty(prefix = "run", name = "operation", havingValue = "NORMALIZE") | ||
| @EnableConfigurationProperties({NormalizationConfigProperties.class, NormalizationKeycloakConfigProperties.class}) | ||
| public class KeycloakConfigNormalizationRunner implements CommandLineRunner, ExitCodeGenerator { | ||
|
|
||
| private static final Logger logger = LoggerFactory.getLogger(KeycloakConfigNormalizationRunner.class); | ||
| private static final long START_TIME = System.currentTimeMillis(); | ||
|
|
||
| private final RealmNormalizationService normalizationService; | ||
| private final KeycloakExportProvider exportProvider; | ||
| private final NormalizationConfigProperties normalizationConfigProperties; | ||
| private final YAMLMapper yamlMapper; | ||
| private final ObjectMapper objectMapper; | ||
| private int exitCode; | ||
|
|
||
| @Autowired | ||
| public KeycloakConfigNormalizationRunner(RealmNormalizationService normalizationService, | ||
| KeycloakExportProvider exportProvider, | ||
| NormalizationConfigProperties normalizationConfigProperties, | ||
| YAMLMapper yamlMapper, | ||
| ObjectMapper objectMapper) { | ||
| this.normalizationService = normalizationService; | ||
| this.exportProvider = exportProvider; | ||
| this.normalizationConfigProperties = normalizationConfigProperties; | ||
| this.yamlMapper = yamlMapper; | ||
| this.objectMapper = objectMapper; | ||
| } | ||
|
|
||
| @Override | ||
| public void run(String... args) throws Exception { | ||
| try { | ||
| var outputLocation = Paths.get(normalizationConfigProperties.getFiles().getOutputDirectory()); | ||
| if (!Files.exists(outputLocation)) { | ||
| logger.info("Creating output directory '{}'", outputLocation); | ||
| Files.createDirectories(outputLocation); | ||
| } | ||
| if (!Files.isDirectory(outputLocation)) { | ||
| logger.error("Output location '{}' is not a directory. Aborting", outputLocation); | ||
| exitCode = 1; | ||
| return; | ||
| } | ||
|
|
||
| for (var exportLocations : exportProvider.readFromLocations().values()) { | ||
| for (var export : exportLocations.entrySet()) { | ||
| logger.info("Normalizing file '{}'", export.getKey()); | ||
| for (var realm : export.getValue()) { | ||
| var normalizedRealm = normalizationService.normalizeRealm(realm); | ||
| var suffix = normalizationConfigProperties.getOutputFormat() == YAML ? "yaml" : "json"; | ||
| var outputFile = outputLocation.resolve(String.format("%s.%s", normalizedRealm.getRealm(), suffix)); | ||
| try (var os = new FileOutputStream(outputFile.toFile())) { | ||
| if (normalizationConfigProperties.getOutputFormat() == YAML) { | ||
| yamlMapper.writeValue(os, normalizedRealm); | ||
| } else { | ||
| objectMapper.writeValue(os, normalizedRealm); | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } catch (NullPointerException e) { | ||
| throw e; | ||
| } catch (Exception e) { | ||
| logger.error(e.getMessage()); | ||
|
|
||
| exitCode = 1; | ||
|
|
||
| if (logger.isDebugEnabled()) { | ||
| throw e; | ||
| } | ||
| } finally { | ||
| long totalTime = System.currentTimeMillis() - START_TIME; | ||
| String formattedTime = new SimpleDateFormat("mm:ss.SSS").format(new Date(totalTime)); | ||
| logger.info("keycloak-config-cli running in {}.", formattedTime); | ||
| } | ||
| } | ||
|
|
||
| @Override | ||
| public int getExitCode() { | ||
| return exitCode; | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.