Support S3 storage backends

[tobbbles]

It would be great if Polaris natively supported S3 object store as a backend for retrieving and streaming music. Koel Briefly details how they manage this (with a Lambda triggered to update the database.)

It feels there would be multiple ways to implement this, what are peoples thoughts on this matter?

[agersant]

I think this is a interesting subject. I did look into storing my own music collection on S3 a while ago, but I found it prohibitively expensive: $15 a month just for 1TB of slow storage, plus some more for actually using the files. I do have backups in Glacier though.

Some technical thoughts:

• The Koel solution linked above surprised me. Their Lambda worker parses song metadata and makes individual requests to the Koel server to add them to its index. This sounds like it would take hours to populate an initial collection, which doesn't sound great. It would also be a lot of changes in Polaris to add an external API to alter the song index, and to extract the song parsing code into a separate crate for a Lambda worker.
• I wonder how far one could get by using something like s3fs to make Polaris think the S3 bucket is just another local directory. Of course, this is only viable with a Polaris server hosted in AWS too (otherwise library scans would cost a fortune).
• First-class support within Polaris code would not be too hard to add but has the same cost issue as S3FS

Probably not something I am doing to work on in the near or medium future but happy to keep the discussion open.

[tobbbles]

Thanks for the input.

As you mentioned, I think s3fs is a very good starting point. I will make a note to play around with setting up Polaris + s3fs on a test set up, and hopefully produce some documentation to aid other users who want a similar approach.

Regarding 1st class support; perhaps it's not yet viable to consider this with just s3, but I wonder if there are any 'storage backends' that people would be keen to use? Some that come to mind are Google Drive, Google Cloud Storage/Azure Storage. Pluggable storage backends could be a very neat feature of Polaris.

[xorander00]

Just found this project and building it now to try out :)

S3 support would be really nice to have, and is something that I would definitely use. I have a self-hosted S3-compilant server (minio in my case, but there are other options like wasabi). s3fs is a neat project, but it's kind of a hacky bridge between object storage & file systems (not making fun of the project, just saying it's a stopgap measure).

The nice thing about S3-like object storage is that the APIs & clients are relatively simple. There are also added benefits like easily being able to create signed URLs for share links, streaming the contents directly, etc. For example, the S3 option in funkwhale is used to offer direct streaming to the client from storage instead of needing the funkwhale server in the middle.

Anyway, just my $0.02, food for thought.

[agersant]

This is good to know, I didn't realize there were self-hosted S3 lookalikes. Thanks for chiming in!

[tobbbles]

Would there be any other non-filesystem, S3-like backends that could potentially be supported by Polaris?

I know Google Drive is often used an archive tool, but I'm not sure about it's viability for audio/file streaming

[xorander00]

The main storage providers that I can think of are local file system, S3-compatible object store, & maybe sftp/ssh. The latter could work since ssh allows piping stdin/stdout transparently over the connection.

I just tried compiling the project again and it failed for me, but then again I'm building master and it's on FreeBSD. I'll either open another issue later once I look further into it or maybe send a PR if it's simple enough to fix. I'd like to try this out and possibly replace my existing service with it if it works out well.

[zamu-flowerpot]

Dropping by to link MinIO and Garage which are both self-hosted solutions to S3.