Add a CI job to generate backwards compatibility test indexes and a python unit test to query them

[jparismorgan]

What

Here we add:

• A new siftmicro_base.fvecs file which just contains the first 100 vectors from siftsmall_base.fvecs.
  • The input data is 100 vectors, each with 5 dimensions.
  • We will index float32 and uint8 versions of the input data with all available index types (Flat and IVFFlat today).
• A script which will generate indices using the current version of the engine and place them in backwards-compatibility-data/data/x.x.x. This is the proposed naming structure:

/TileDB-Vector-Search/backwards-compability-data/0.0.19/flat_f32
/TileDB-Vector-Search/backwards-compability-data/0.0.19/flat_u8
/TileDB-Vector-Search/backwards-compability-data/0.0.19/ivf_flat_f32
/TileDB-Vector-Search/backwards-compability-data/0.0.19/ivf_flat_u8

• A new CI job in the release process which will run this script and commit + push these indices to main.
• A Python unit test which will read all indices in backwards-compatibility-data/data and verify we can query them.

Testing

In this job https://github.com/TileDB-Inc/TileDB-Vector-Search/actions/runs/7558031052/job/20578613117 I tested running this and pushing to jparismorgan/temp-testing-2. You can see we make one commit to jparismorgan/temp-testing-2 with the new indices: https://github.com/TileDB-Inc/TileDB-Vector-Search/commits/jparismorgan/temp-testing-2/. I then downloaded these indices and ran test_backwards_compatibility.py locally and it passed:

(TileDB-Vector-Search-2) ~/repo/TileDB-Vector-Search-2/apis/python pytest test/test_backwards_compatibility.py -s
================================= test session starts =================================
platform darwin -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /Users/parismorgan/repo/TileDB-Vector-Search-2/apis/python
collected 1 item                                                                      

test/test_backwards_compatibility.py datasets_path /Users/parismorgan/repo/TileDB-Vector-Search-2/apis/python/test/../../../backwards-compatibility-data/data

Checking version_path /Users/parismorgan/repo/TileDB-Vector-Search-2/apis/python/test/../../../backwards-compatibility-data/data/0.0.0
Skipping because this version had relative paths in it

Checking version_path /Users/parismorgan/repo/TileDB-Vector-Search-2/apis/python/test/../../../backwards-compatibility-data/data/0.0.1
Testing index_uri /Users/parismorgan/repo/TileDB-Vector-Search-2/apis/python/test/../../../backwards-compatibility-data/data/0.0.1/flat_uint8
Testing index_uri /Users/parismorgan/repo/TileDB-Vector-Search-2/apis/python/test/../../../backwards-compatibility-data/data/0.0.1/ivf_flat_float32
Testing index_uri /Users/parismorgan/repo/TileDB-Vector-Search-2/apis/python/test/../../../backwards-compatibility-data/data/0.0.1/flat_float32
Testing index_uri /Users/parismorgan/repo/TileDB-Vector-Search-2/apis/python/test/../../../backwards-compatibility-data/data/0.0.1/ivf_flat_uint8
.

================================== 1 passed in 0.31s ==================================

TODO

I have not yet tested in real release, so I didn't make the CI job a prerequisite for build_wheels, but once it's tested it seems like a good thing to do.