[FEATURE REQUEST]: Graceful Handling of Breaking Changes in Hyperspace

[rapoth]

Feature requested

As Hyperspace is still at its inception, some core aspects have not been finalized. As users request new features, it is possible that Hyperspace needs to change the meta-data to accommodate these new features. However, this would break existing users in that they would have to recreate their indexes. While this is acceptable to most users, there are cases where index creation might be expensive so users would want ways to circumvent paying this cost (unless of course, the physical index structures have changed altogether).

As a general user of Hyperspace, I want an easy-to-use migration utility so I can upgrade Hyperspace faster in production scenarios without the fear of breaking my existing pipelines.

Acceptance criteria

Describe what you consider (e.g., functionality requirement) has to be true to mark this feature as complete

• Clear guidelines for
  • developers on how to introduce a breaking change
  • users on how to upgrade Hyperspace in their environments from a given version
• In-built migration utility that allows for:
  • Upgrade from any version to the latest version
  • Check if an upgrade is possible. If not, explain clearly to the end user on what needs to be done
  • If the underlying physical index structures have changed, and it is not possible to use the existing indexes anymore, explain clearly what needs to be done (even if it is recreating the indexes)
  • For cases where the physical index structure has not changed, provide a way to do an in-place migration i.e., only the meta-data gets upgraded but the user does not need to recreate their indexes
• Documentation updates to reflect all the work done to describe graceful handling of breaking changes in Hyperspace

Success criteria

Define any measurements that need to be performed

• I am able to upgrade from Hyperspace v to latest - while the migration utility may not work for every single version, we should define the minimum version it will start working with
• Time to migrate indexes for TPC-DS 1 TB dataset