Hyperspace v0.4.0

We are excited to announce the release of Hyperspace 0.4.0!

Notable new features / improvements:

• Delta Lake support: Hyperspace v0.4.0 supports creating index on Delta Lake tables. Please refer to the user guide for more info.
• Support for Databricks: #303 fixes the known issue when Hyperspace was run on Databricks. Hyperspace v0.4.0 can now run on Databricks Runtime 5.5 LTS & 6.4!
• Globbing patterns for indexes: Globbing patterns can be used to specify a subset of source data to create/maintain index on. Please refer to the user guide on the usage.
• Enhanced index statistics: A new API (hyperspace.index(indexName)) is introduced to get detailed index statistics such as the number of index files, index size in bytes, source update info, etc. Please check #286 for the sample output.
• Hybrid Scan improvements: Hyperspace 0.4.0 brings in several improvements on Hybrid Scan such as a better mechanism to enable/disable the feature, rank algorithm improvements (#164), quick index refresh (#238), etc.
• Pluggable source provider: This release introduces a (evolving) pluggable source provider API set so that different source formats can be plugged in. This enabled Delta Lake source to be plugged in, and there is on-going PR to support Iceberg tables (#320).
• This release also includes various bug fixes / performance improvements. Please check here for the complete list of commits that went into the v0.4.0 release.

Breaking changes:

#268 fixes the issue where the signature calculation may produce a different result depending on the order of the input files. Thus, the indexes generated with v0.3.0 are not compatible with v0.4.0 and need to be reconstructed.

Thank you for trying it out and we look forward to your feedback!

Credits

Andrei Ionescu, Andrew Fogarty, Apoorve Dave, Eunjin Song, Gurleen Singh, Justin Breese, kaustubhkhare, Pouria Pirzadeh, Rahul Potharaju, Tarun Rajput, Terry Kim, Veysi Ertekin, Yash Datta