You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This issue is related to grpc_kvstore driver, which is currently not available for using with vanilla tensorstore, but was discussed in the issue #134 and is going to become public.
ML teams in my company are using a privately patched version of tensorstore with grpc_kvstore driver, whose backend is implemented by YTsaurus. Current grpc protocol imposes the following problem: both Read and Write requests are one-shot, which limits the length of a value by 2 GiB, which is a fundamental upper limit of Protobuf message size. Also, it is not a good idea in general to write read or write large blobs within a single request, because it is not a fault-tolerant solution.
Our proposal is to make Read and Write requests respectively server-side and client-side streaming methods, limiting the size of a single message with something reasonable like 32 MiB. It seems that this change may be done in a backward-incompatible manner, as grpc_kvstore interface is not public and stable yet.
If you are ok with this proposal, we would be glad to bring a PR implemeting this idea.
The text was updated successfully, but these errors were encountered:
zlobober
changed the title
Suppiort of large files for grpc_kvstore
Support of large files for grpc_kvstore
Feb 16, 2024
This commit changes tsgrpc protocol in backward-incompatible manner, addressing issue google#137.
After this change, it becomes possible to read and write values that exceed the theoretic maximum
of 2 GiB per proto message.
This issue is related to grpc_kvstore driver, which is currently not available for using with vanilla tensorstore, but was discussed in the issue #134 and is going to become public.
ML teams in my company are using a privately patched version of tensorstore with grpc_kvstore driver, whose backend is implemented by YTsaurus. Current grpc protocol imposes the following problem: both Read and Write requests are one-shot, which limits the length of a value by 2 GiB, which is a fundamental upper limit of Protobuf message size. Also, it is not a good idea in general to write read or write large blobs within a single request, because it is not a fault-tolerant solution.
Our proposal is to make Read and Write requests respectively server-side and client-side streaming methods, limiting the size of a single message with something reasonable like 32 MiB. It seems that this change may be done in a backward-incompatible manner, as grpc_kvstore interface is not public and stable yet.
If you are ok with this proposal, we would be glad to bring a PR implemeting this idea.
The text was updated successfully, but these errors were encountered: