lsdb server large string issue

[Schwarzam]

I implemented the changes we discussed in both hipscat and lsdb, and these changes are now in the lsdb-server branches.

Here's a code snippet demonstrating how to retrieve data from the SPLUS server using the lsdb-server:

!pip3 install --quiet git+https://github.com/astronomy-commons/lsdb@lsdb-server
!pip3 install --quiet git+https://github.com/astronomy-commons/hipscat@lsdb-server

from dask.distributed import Client
import lsdb

links = [
    'https://splus.cloud/HIPS/catalogs/dr4/dual', 
    'https://splus.cloud/HIPS/catalogs/dr4/dual_2arcsec'
]

client = Client(n_workers=10, memory_limit="8GB")

dual_splus_margin = lsdb.read_hipscat(links[1])
dual_splus_hips = lsdb.read_hipscat(
    links[0],
    margin_cache=dual_splus_margin,
    columns = ["ID", "RA", "DEC", "r_auto"], # Try messing with the columns
    filters=[("r_auto", "<", "13")], # Try messing with this query
    dtype_backend="numpy_nullable" # This here "solves" the  pyarrow large_string error.
)

res = dual_splus_hips.cone_search(
    0.1,
    0.1,
    1 * 3600
)
res.compute()

• If you remove the dtype_backend="numpy_nullable" it raises the large string error.

My understanding of the lsdb code and this metadata issue is still limited, making it challenging to find a "workaround" on the client side.

Meanwhile I'll be working on the server trying to figure out this.

[Schwarzam]

Solved the issue on server side. Now lsdb works fine with the server.

lsdb-server PR #3