Data Product versioning

[PriyaBasker23]

Versioning strategy to use for the Data as a Product

We have made some exploration for data product and schema versioning , which is recorded in the confluence page below. https://dsdmoj.atlassian.net/wiki/spaces/DataPlatform/pages/4479156541/Data+product+Schema+Versioning.

We need to compile a list of scenarios that would necessitate the release of either a major or minor version of the data product.

This is the initial list, which can be refined and expanded upon as we make progress. Having this list in place will serve as a valuable foundation for system design and will enable us to initiate the development process confidently

[murdo-moj]

Scenarios are listed as part of that document:

Minor version exceptions list

• Add whole new schema
• Add new metadata field at the data product level
• Descriptions added or updated for an existing column in a table

Any change not on that list will trigger an automatic major version bump

[LavMatt]

Options for versioning implementation slides are here:
https://docs.google.com/presentation/d/1Hla-lFlqpEFMElri6PfwtInbI3NAQMdapuWXPjyRPyY/edit?usp=sharing

[MatMoore]

Middle ground we discussed today:

• We version at the data product level
• Each version of the data product has its own metadata and schema files. If either the metadata or the schema change, the version will change
• So version 1.1 of data product foo would have its own copy of the schemas for all tables, e.g. foo/1.1/schemas, regardless of how many schemas (if any) changed

The advantages of this approach are:

1. Since we retain the schemas for all versions, we can compare files to create a version history
2. Consumers don't have to deal with the extra complexity of schema versions vs data product versions

We also discussed that every major version update is effectively a new data product, with its own data & tables.

[PriyaBasker23]

Thanks Mat for the update

[PriyaBasker23]

Agreed Approach

[tom-webber]

How do we ensure that we capture events where the underlying calculation for a column changes? E.g. to fix a bug resulting from an incorrect calculation.

There are definitely scenarios where the following would capture it:

• Descriptions added or updated for an existing column in a table

But there may be other occasions where the calculation/derivation method isn't described in the column description, so it could be that there is no signifying a change that might require reprocessing of past data.

[MatMoore]

This makes me wonder if the minor versions exception list is really something we can infer automatically vs something that should always be confirmed by the data producer.

Even if a calculation doesn't change, there might be changes to the way data is collected or reported that consumers would need to be aware of, and if that is masked by a minor version change, consumers could be unaware they need to treat the data differently.

[PriyaBasker23]

The data producer should create as seperate column rather than renaming the existing column