Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[kie-issues#1606] Modify columns in the flyway migration scripts which uses sql reserved words #2139

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

josedee
Copy link

@josedee josedee commented Nov 5, 2024

Closes apache/incubator-kie-issues#1606

The flyway migration sql scripts creates few tables with column names like 'key' and 'value' which are reserved words in H2. So in order to avoid that

Create new flyway scripts to alter the existing column names
Verify/modify any jpa mappings if required.

Many thanks for submitting your Pull Request ❤️!

Please make sure that your PR meets the following requirements:

  • You have read the contributors guide
  • Pull Request title is properly formatted: KOGITO-XYZ Subject
  • Pull Request title contains the target branch if not targeting main: [0.9.x] KOGITO-XYZ Subject
  • Pull Request contains link to the JIRA issue
  • Pull Request contains link to any dependent or related Pull Request
  • Pull Request contains description of the issue
  • Pull Request does not include fixes for issues other than the main ticket
How to replicate CI configuration locally?

Build Chain tool does "simple" maven build(s), the builds are just Maven commands, but because the repositories relates and depends on each other and any change in API or class method could affect several of those repositories there is a need to use build-chain tool to handle cross repository builds and be sure that we always use latest version of the code for each repository.

build-chain tool is a build tool which can be used on command line locally or in Github Actions workflow(s), in case you need to change multiple repositories and send multiple dependent pull requests related with a change you can easily reproduce the same build by executing it on Github hosted environment or locally in your development environment. See local execution details to get more information about it.

How to retest this PR or trigger a specific build:
  • for pull request checks
    Please add comment: Jenkins retest this

  • for a specific pull request check
    Please add comment: Jenkins (re)run [kogito-apps|kogito-examples] tests

  • for quarkus branch checks
    Run checks against Quarkus current used branch
    Please add comment: Jenkins run quarkus-branch

  • for a quarkus branch specific check
    Run checks against Quarkus current used branch
    Please add comment: Jenkins (re)run [kogito-apps|kogito-examples] quarkus-branch

  • for quarkus main checks
    Run checks against Quarkus main branch
    Please add comment: Jenkins run quarkus-main

  • for a specific quarkus main check
    Run checks against Quarkus main branch
    Please add comment: Jenkins (re)run [kogito-apps|kogito-examples] quarkus-main

  • for quarkus lts checks
    Run checks against Quarkus lts branch
    Please add comment: Jenkins run quarkus-lts

  • for a specific quarkus lts check
    Run checks against Quarkus lts branch
    Please add comment: Jenkins (re)run [kogito-apps|kogito-examples] quarkus-lts

  • for native checks
    Run native checks
    Please add comment: Jenkins run native

  • for a specific native check
    Run native checks
    Please add comment: Jenkins (re)run [kogito-apps|kogito-examples] native

  • for native lts checks
    Run native checks against quarkus lts branch
    Please add comment: Jenkins run native-lts

  • for a specific native lts check
    Run native checks against quarkus lts branch
    Please add comment: Jenkins (re)run [kogito-apps|kogito-examples] native-lts

How to backport a pull request to a different branch?

In order to automatically create a backporting pull request please add one or more labels having the following format backport-<branch-name>, where <branch-name> is the name of the branch where the pull request must be backported to (e.g., backport-7.67.x to backport the original PR to the 7.67.x branch).

NOTE: backporting is an action aiming to move a change (usually a commit) from a branch (usually the main one) to another one, which is generally referring to a still maintained release branch. Keeping it simple: it is about to move a specific change or a set of them from one branch to another.

Once the original pull request is successfully merged, the automated action will create one backporting pull request per each label (with the previous format) that has been added.

If something goes wrong, the author will be notified and at this point a manual backporting is needed.

NOTE: this automated backporting is triggered whenever a pull request on main branch is labeled or closed, but both conditions must be satisfied to get the new PR created.

Quarkus-3 PR check is failing ... what to do ? The Quarkus 3 check is applying patches from the `.ci/environments/quarkus-3/patches`.

The first patch, called 0001_before_sh.patch, is generated from Openrewrite .ci/environments/quarkus-3/quarkus3.yml recipe. The patch is created to speed up the check. But it may be that some changes in the PR broke this patch.
No panic, there is an easy way to regenerate it. You just need to comment on the PR:

jenkins rewrite quarkus-3

and it should, after some minutes (~20/30min) apply a commit on the PR with the patch regenerated.

Other patches were generated manually. If any of it fails, you will need to manually update it... and push your changes.

@josedee josedee marked this pull request as ready for review November 5, 2024 11:51
@pefernan pefernan self-requested a review November 5, 2024 12:43
@kie-ci3
Copy link
Contributor

kie-ci3 commented Nov 5, 2024

PR job #1 was: UNSTABLE
Possible explanation: This should be test failures

Reproducer

build-chain build full_downstream -f 'https://raw.githubusercontent.com/${AUTHOR:apache}/incubator-kie-kogito-pipelines/${BRANCH:main}/.ci/buildchain-config-pr-cdb.yaml' -o 'bc' -p apache/incubator-kie-kogito-apps -u #2139 --skipParallelCheckout

NOTE: To install the build-chain tool, please refer to https://github.com/kiegroup/github-action-build-chain#local-execution

Please look here: https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/pullrequest_jobs/job/kogito-apps-pr/job/PR-2139/1/display/redirect

Test results:

  • PASSED: 1631
  • FAILED: 7

Those are the test failures:

org.kie.kogito.index.service.InmemoryPostgreSqlIndexingServiceIT.(?) java.lang.RuntimeException: Failed to start quarkus
org.kie.kogito.index.service.KeycloakInmemoryPostgreSqlIndexingServiceIT.testUnauthorizedUserAccess java.lang.RuntimeException: Failed to start quarkus
org.kie.kogito.index.postgresql.reporting.ProcessInstanceVariableMappingIT.testProcessInstanceVariableMapping java.lang.RuntimeException: Failed to start quarkus
org.kie.kogito.index.postgresql.mapper.JobEntityMapperIT.testMapToModel java.lang.RuntimeException: Failed to start quarkus
org.kie.kogito.persistence.postgresql.PostgresStorageServiceIT.testQuery_Equal java.lang.RuntimeException: Failed to start quarkus
org.kie.kogito.persistence.postgresql.reporting.BasicTypeMappingIT.testBasicTypeMappingDelete java.lang.RuntimeException: Failed to start quarkus
org.kie.kogito.trusty.service.postgresql.PostgreSQLTrustyServiceIT.testStoreExplainabilityResult_Counterfactual java.lang.RuntimeException: Failed to start quarkus

@yesamer yesamer self-requested a review November 5, 2024 16:13
Signed-off-by: Deepak Joseph <[email protected]>
Signed-off-by: Deepak Joseph <[email protected]>
@kie-ci3
Copy link
Contributor

kie-ci3 commented Nov 5, 2024

PR job #2 was: UNSTABLE
Possible explanation: This should be test failures

Reproducer

build-chain build full_downstream -f 'https://raw.githubusercontent.com/${AUTHOR:apache}/incubator-kie-kogito-pipelines/${BRANCH:main}/.ci/buildchain-config-pr-cdb.yaml' -o 'bc' -p apache/incubator-kie-kogito-apps -u #2139 --skipParallelCheckout

NOTE: To install the build-chain tool, please refer to https://github.com/kiegroup/github-action-build-chain#local-execution

Please look here: https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/pullrequest_jobs/job/kogito-apps-pr/job/PR-2139/2/display/redirect

Test results:

  • PASSED: 575
  • FAILED: 11

Those are the test failures:

org.kie.kogito.persistence.postgresql.reporting.BasicTypeMappingIT.testBasicTypeMappingDelete JDBC exception executing SQL [SELECT ROW_NUMBER() OVER (ORDER BY name, key) as id, name, key, field1MappedField, field2MappedField FROM BasicTypeExtract] [ERROR: relation "basictypeextract" does not exist
Position: 107] [n/a]
org.kie.kogito.persistence.postgresql.reporting.BasicTypeMappingIT.testBasicTypeMapping JDBC exception executing SQL [SELECT ROW_NUMBER() OVER (ORDER BY name, key) as id, name, key, field1MappedField, field2MappedField FROM BasicTypeExtract] [ERROR: relation "basictypeextract" does not exist
Position: 107] [n/a]
org.kie.kogito.persistence.postgresql.reporting.BasicTypeMappingIT.testBasicTypeMappingUpdate JDBC exception executing SQL [SELECT ROW_NUMBER() OVER (ORDER BY name, key) as id, name, key, field1MappedField, field2MappedField FROM BasicTypeExtract] [ERROR: relation "basictypeextract" does not exist
Position: 107] [n/a]
org.kie.kogito.persistence.postgresql.reporting.ComplexHierarchicalTypeMappingIT.testComplexHierarchicalTypeMapping JDBC exception executing SQL [SELECT ROW_NUMBER() OVER (ORDER BY name, key) as id, name, key, root, nestedBasicMappedField, nestedComplexCollectionMappedField1, nestedComplexCollectionMappedSubField1 FROM ComplexHierarchicalTypeExtract] [ERROR: relation "complexhierarchicaltypeextract" does not exist
Position: 176] [n/a]
org.kie.kogito.persistence.postgresql.reporting.ComplexHierarchicalTypeMappingIT.testComplexHierarchicalTypeMappingDelete JDBC exception executing SQL [SELECT ROW_NUMBER() OVER (ORDER BY name, key) as id, name, key, root, nestedBasicMappedField, nestedComplexCollectionMappedField1, nestedComplexCollectionMappedSubField1 FROM ComplexHierarchicalTypeExtract] [ERROR: relation "complexhierarchicaltypeextract" does not exist
Position: 176] [n/a]
org.kie.kogito.persistence.postgresql.reporting.ComplexHierarchicalTypeMappingIT.testComplexHierarchicalTypeMappingUpdate JDBC exception executing SQL [SELECT ROW_NUMBER() OVER (ORDER BY name, key) as id, name, key, root, nestedBasicMappedField, nestedComplexCollectionMappedField1, nestedComplexCollectionMappedSubField1 FROM ComplexHierarchicalTypeExtract] [ERROR: relation "complexhierarchicaltypeextract" does not exist
Position: 176] [n/a]
org.kie.kogito.persistence.postgresql.reporting.HierarchicalTypeMappingIT.testHierarchicalTypeMappingDelete JDBC exception executing SQL [SELECT ROW_NUMBER() OVER (ORDER BY name, key) as id, name, key, root, nestedBasicMappedField, nestedBasicCollectionMappedField FROM HierarchicalTypeExtract] [ERROR: relation "hierarchicaltypeextract" does not exist
Position: 133] [n/a]
org.kie.kogito.persistence.postgresql.reporting.HierarchicalTypeMappingIT.testHierarchicalTypeMappingUpdate JDBC exception executing SQL [SELECT ROW_NUMBER() OVER (ORDER BY name, key) as id, name, key, root, nestedBasicMappedField, nestedBasicCollectionMappedField FROM HierarchicalTypeExtract] [ERROR: relation "hierarchicaltypeextract" does not exist
Position: 133] [n/a]
org.kie.kogito.persistence.postgresql.reporting.HierarchicalTypeMappingIT.testHierarchicalTypeMapping JDBC exception executing SQL [SELECT ROW_NUMBER() OVER (ORDER BY name, key) as id, name, key, root, nestedBasicMappedField, nestedBasicCollectionMappedField FROM HierarchicalTypeExtract] [ERROR: relation "hierarchicaltypeextract" does not exist
Position: 133] [n/a]
org.kie.kogito.persistence.postgresql.reporting.api.PostgresMappingsApiV1IT.dynamicMappingDefinitionCreation expected: <200> but was: <500>
org.kie.kogito.persistence.postgresql.reporting.database.sqlbuilders.PostgresApplyMappingSqlBuilderIT.testApplyMappingToExistingData JDBC exception executing SQL [CREATE TABLE DynamicTypeExtract (
key null,
name varchar,
field1MappedField text,
field2MappedField numeric
);
] [ERROR: syntax error at or near "null"
Position: 42] [n/a]

Signed-off-by: Deepak Joseph <[email protected]>
Copy link
Contributor

@LightGuard LightGuard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm honestly surprised we haven't done this before. Good idea to change the names from reserved words. It also helps make the names more descriptive.

@yesamer
Copy link
Contributor

yesamer commented Nov 12, 2024

@pefernan I guess we can merge it, WDYT?

@gmunozfe
Copy link
Contributor

gmunozfe commented Nov 12, 2024

hi, before merging, have you explore the possibility of using the property NON_KEYWORDS in the h2 jdbc URL (that was done in the past for jbpm having the same issues). This would avoid any conflict without migrating table columns which affects previous deployments.
https://www.h2database.com/html/commands.html#set_non_keywords

@domhanak
Copy link

I would like to highlight, that we will soon have users running 10.0.0 applications ( see the ongoing ML vote ). This change might be problematic for users that will try to upgrade their existing DataIndes deployments to next version after 10.0.0.
Did you consider such scenario during testing please? I want to avoid breaking changes, or changes that would require user to use flyway configuration properties to avoid issues during migration.

@fjtirado fjtirado self-requested a review November 12, 2024 10:01
Copy link
Contributor

@fjtirado fjtirado left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fully support @gmunozfe comments, this change is not needed and should not be merged as it is
Key (probably the best option here would have been "name", since "key" is also reserved in mysql) and value (here, even value is a reserved word in latest sql standard, but significantly not reserved by all major vendors, I would keep the current name even if designing the system from scratch) are column names that makes sense for this table.
But even if they were objectively wrong, we should not change them if we do not have other alternative, and the fact is that we have.
The right (in my opinion) solution for the flyway scripts (since the problem does not affect runtime, because we are using JPA, which proprely handle this out of box) is to use the property Gonzalo mentions for H2. For mysql, qoutes will have to be used.
Please note that, in my opinion, it is perfectly fine to have different flyway scripts for different dbs (in fact we already have) and the expectation if that we will have one directory for each db we support. (if we can reuse the same dir for several dbs, the better, but it should not be a constraint and definitely it should not trigger changes in existing scripts or db tables)

@yesamer
Copy link
Contributor

yesamer commented Nov 12, 2024

Thank you @gmunozfe for sharing that, I agree with you is a valuable option we should consider as an alternative.
@josedee @pefernan Please consider the above suggestion and share your thoughts, since you're leading this initiative.

@elguardian
Copy link
Contributor

@fjtirado @gmunozfe @yesamer @pefernan
In this case the problem with H2 has surfaced the problem of column names. Sometimes we use names that are sql keywords for certain database. I would argue that even if we fix the H2 problem without altering the database, we should strive to a more proper naming revisiting columns and check if some future support database have it.
I would revisit this to a more broader scope to check databases supported and decide from there (after analising the impact) what to do. but this will come back for sure at some point in the future. It is not urgent as there is workaround for this case, but it is not a proper solutions as we are going to need to revisit this at some point.
I can only see two options when the time comes:

  1. offer a proper orm file remapping the names of those conflicting columns (the thing here is that we need add that to the classpath
  2. redefining column names.

* under the License.
*/

alter table definitions_nodes_metadata alter column key rename to metadata_key;
Copy link
Contributor

@fjtirado fjtirado Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we are going to rename, I think we should rename key to name or id


alter table definitions_nodes_metadata alter column key rename to metadata_key;
alter table definitions_nodes_metadata alter column value rename to metadata_value;
alter table definitions_metadata alter column key rename to metadata_key;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here, key to name

*/

alter table definitions_nodes_metadata alter column key rename to metadata_key;
alter table definitions_nodes_metadata alter column value rename to metadata_value;
Copy link
Contributor

@fjtirado fjtirado Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think value should be ok, but since it was reserved by standard, maybe stringValue ;)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Modify columns in the flyway migration scripts which uses sql reserved words
8 participants