perf: transpose the PQ codes to improve search performance

[BubbleCal]

30% improved:

5242880,L2,PQ=96,DIM=1536
                        time:   [161.24 ms 162.71 ms 164.08 ms]
                        change: [-30.390% -29.789% -29.192%] (p = 0.00 < 0.10)
                        Performance has improved.

Benchmarking 5242880,Cosine,PQ=96,DIM=1536: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 9.2s or enable flat sampling.
5242880,Cosine,PQ=96,DIM=1536
                        time:   [165.53 ms 167.12 ms 168.49 ms]
                        change: [-30.725% -30.233% -29.772%] (p = 0.00 < 0.10)
                        Performance has improved.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 97.08738% with 6 lines in your changes missing coverage. Please review.

Project coverage is 77.13%. Comparing base (4ab270b) to head (b5302f7).

Files with missing lines	Patch %	Lines
rust/lance-index/src/vector/pq/storage.rs	94.82%	1 Missing and 2 partials ⚠️
rust/lance/src/index/vector/builder.rs	85.71%	0 Missing and 1 partial ⚠️
rust/lance/src/index/vector/ivf.rs	83.33%	0 Missing and 1 partial ⚠️
rust/lance/src/index/vector/pq.rs	98.03%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3120      +/-   ##
==========================================
+ Coverage   77.12%   77.13%   +0.01%     
==========================================
  Files         240      240              
  Lines       80652    80759     +107     
  Branches    80652    80759     +107     
==========================================
+ Hits        62203    62296      +93     
  Misses      15279    15279              
- Partials     3170     3184      +14     

Flag	Coverage Δ
unittests	77.13% <97.08%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[westonpace]

Very cool trick! It looks like transpose is a configurable option of sorts (maybe just at lower levels). Is it possible to setup a unit test that calculates distances with transpose and calculates distances without transpose and verifies they are the same?

[westonpace]

Why did these results change? Shouldn't transposing not affect the calculation?

[BubbleCal]

not related to transposing
I think IVF_PQ can always reach this recall, these tests were added when working on HNSW, which would result in low recall with the other distance types, looks like I copied HNSW's tests and forgot to modify the thresholds for IVF_PQ

[BubbleCal]

added a test to verify the results are the same as without transposing