Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support of large files for grpc_kvstore #137

Open
zlobober opened this issue Feb 16, 2024 · 1 comment
Open

Support of large files for grpc_kvstore #137

zlobober opened this issue Feb 16, 2024 · 1 comment

Comments

@zlobober
Copy link

This issue is related to grpc_kvstore driver, which is currently not available for using with vanilla tensorstore, but was discussed in the issue #134 and is going to become public.

ML teams in my company are using a privately patched version of tensorstore with grpc_kvstore driver, whose backend is implemented by YTsaurus. Current grpc protocol imposes the following problem: both Read and Write requests are one-shot, which limits the length of a value by 2 GiB, which is a fundamental upper limit of Protobuf message size. Also, it is not a good idea in general to write read or write large blobs within a single request, because it is not a fault-tolerant solution.

Our proposal is to make Read and Write requests respectively server-side and client-side streaming methods, limiting the size of a single message with something reasonable like 32 MiB. It seems that this change may be done in a backward-incompatible manner, as grpc_kvstore interface is not public and stable yet.

If you are ok with this proposal, we would be glad to bring a PR implemeting this idea.

@zlobober zlobober changed the title Suppiort of large files for grpc_kvstore Support of large files for grpc_kvstore Feb 16, 2024
@laramiel
Copy link
Collaborator

I concur that the read and write need to be streaming gRPC apis; the gcs_grpc driver is likely a reasonable model for the kvstore driver code.

Note that default gRPC server GRPC_ARG_MAX_RECEIVE_MESSAGE_LENGTH is 4MB; the chunk size should be configurable.

zlobober added a commit to zlobober/tensorstore that referenced this issue Apr 8, 2024
This commit changes tsgrpc protocol in backward-incompatible manner, addressing issue google#137.
After this change, it becomes possible to read and write values that exceed the theoretic maximum
of 2 GiB per proto message.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants