S3 Gateway use case

[rkunnamp]

I just thought of sharing a use case that I have.

I am trying to store data lake files in S3 (in delta lake table format) and issue queries against it for a user facing analytics app . I think this is a very common use case these days. The flexibility associated with storing data in S3 is immense. But the downside of such a solution is the latency associated with accessing the data from S3 .

So I was looking for a solution that would help me implement a read through cache , that could possibly reduce the number of round trips to S3 , and reduce the latency associated with my queries.

To my surprise there are not many tools out there that help does that. The only one which I could find was aistore from nvidia (https://github.com/NVIDIA/aistore).

With Versity, since I have posix file system support and S3 gateway support, I am hoping that I could build such a read through cache on the application side on top of Versity. I also want to record this use case here, so that when Versity grows to an industry standard solution down the line, this is perhaps something that could be considered to be natively supported.

[benmcclelland]

so that when Versity grows to an industry standard solution down the line

I like the confidence!

Let us know if there are any features that would help to enable this use case. This sounds like a great use of the gateway!