Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DAT-18792: added cluster by support for snapshot and diff table related change types #199

Merged
merged 21 commits into from
Oct 21, 2024

Conversation

SvampX
Copy link
Contributor

@SvampX SvampX commented Oct 15, 2024

Tested following liquibase commands:

  • diff
  • diff-changelog
  • snapshot
  • generate-changelog

Changelog generates with present CLUSTER BY part with appropriate columns. Tested case with 1 and 2 clustering columns.
Generated changeset example:

    <changeSet author="msav1 (generated)" id="1728984464968-4">
        <databricks:createTable clusterColumns="test_id,test_present_new" tableName="test_table_clustered_new">
            <column name="test_id" type="INT"/>
            <column name="test_present_new" type="INT"/>
            <databricks:extendedTableProperties tableLocation="s3://databricks-test-harness-metastore/c93bad90-fc3e-4083-a910-1cad48d283f2/tables/3b4b3312-cd20-41d1-9d74-451be370869f" tblProperties="delta.checkpoint.writeStatsAsJson=false,delta.checkpoint.writeStatsAsStruct=true,delta.checkpointPolicy=v2,delta.columnMapping.maxColumnId=3,delta.columnMapping.mode=name,delta.enableDeletionVectors=true,delta.enableRowTracking=true,delta.feature.allowColumnDefaults=supported,delta.feature.clustering=supported,delta.feature.columnMapping=supported,delta.feature.deletionVectors=supported,delta.feature.domainMetadata=supported,delta.feature.rowTracking=supported,delta.feature.v2Checkpoint=supported,delta.minReaderVersion=3,delta.minWriterVersion=7,delta.rowTracking.materializedRowCommitVersionColumnName=_row-commit-version-col-f039a2c1-7756-4597-9259-825d27ddce58,delta.rowTracking.materializedRowIdColumnName=_row-id-col-4af75f6c-2d74-4fcf-bc2b-8a2ddce45ef4"/>
        </databricks:createTable>
    </changeSet>

@SvampX SvampX requested a review from filipelautert as a code owner October 15, 2024 09:28
@SvampX
Copy link
Contributor Author

SvampX commented Oct 15, 2024

As per described in extracting and cleaning methods possible issues we might try different way to read and store TBLPROPERTIES. Partially it was done on the view side here https://github.com/liquibase/liquibase-databricks/blob/main/src/main/java/liquibase/ext/databricks/snapshot/jvm/ViewSnapshotGeneratorDatabricks.java#L81-L89
I suggest storing in the https://github.com/liquibase/liquibase-databricks/blob/main/src/main/java/liquibase/ext/databricks/change/createTable/ExtendedTableProperties.java tblProperties parameter as a Map, not a raw String. This would make easier and fail proof cleaning up of system table properties and parsing needed stored there data.

Base automatically changed from DAT-18787 to main October 16, 2024 11:39
@suryaaki2
Copy link

Are we writing tests for this PR?

@filipelautert
Copy link
Contributor

Are we writing tests for this PR?
@suryaaki2 so far for Databricks we are only adding new test cases for test harness, as in createClusteredTableNew.* files added in this PR.

…blProperty 'clusteringColumns', extended snapshot check, removed unnecessary user defined property cleanup. Moved sanitizing logic to snapshot generator to avoid spreading of responsibilities.
Mykhailo Savchenko added 3 commits October 18, 2024 13:33
…erties in quotes, createTable changetype namespace as ext.
…on how to enable serializableFieldType as an element against defaulting attribute
Copy link

@KushnirykOleh KushnirykOleh merged commit 2cd2f40 into main Oct 21, 2024
17 checks passed
@KushnirykOleh KushnirykOleh deleted the DAT-18792 branch October 21, 2024 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants