File sources for gdrive, gcs, onedata, basespace

[nuwang]

This PR partly addresses: #11784

It includes filesources for google drive, google cloud storage and onedata. While these work in varying degrees, they all have issues that hamper their use.

Google Drive

Google drive in particular requires an oauth2 flow to obtain user credentials. Without adding client side support to authorize access, these are extremely tedious for end-users to obtain.

Google Cloud Storage

Google cloud storage also requires oauth2 credentials, but presumably, these could be configured by admins as opposed to end-users. It should also work without credentials in Google cloud. However, there's a library clash with CloudVE/cloudbridge#275 which will be sorted out once a new version is released.

One Data

Onedata does not work because of: onedata/fs-onedatafs#5
Should it be resolved, the code should in principle work, but I've squashed it to one commit so we can drop it easily.

BaseSpace

See separate comment below.

Summary

There are concerns about the practical usability of all of these providers unfortunately. We should probably add some kind of oauth2 authorization mechanisms to smoothen the flow. That would roughly entail:

1. Registering an app with each provider. E.g. A Google app which requires full gdrive permissions - this also needs some kind of verification process from Google as we would need to request full permissions on gdrive.
2. Displaying a link for each filesource that would take the user to the relevant oauth2 authorization page.
3. Adding an endpoint to Galaxy to receive the oauth2 callback
4. Upon receiving the callback, saving the oauth2 auth token and refresh token in the user's profile and eventually, a vault.

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these contributions under Galaxy's current license.
• I agree to allow the Galaxy committers to license these and all my past contributions to the core galaxy codebase under the MIT license. If this condition is an issue, uncheck and just let us know why with an e-mail to [email protected].

[nuwang]

Have also added a filesource for basespace. However, the python basespacesdk itself was last released in 2018 and the pyfilesystem2 plugin needed some patches for things to work.
emedgene/fs_basespace#9

In addition, the basespacesdk appears to ignore the provided credentials and expect them in: '/home/.basespace/default.cfg'. That too requires a patch: basespace/basespace-python-sdk#32

Once configured though, it does work fine. One issue is that basespace returns a numeric id for the filename, and the actual filename as 'alias': https://github.com/emedgene/fs_basespace/blob/bb560444bdec3cbcbc5d66344f877bd823a4bfb6/fs_basespace/_basespacefs.py#L132

This makes it difficult to use in practice. @jmchilton Any thoughts on this?

[nuwang]

This should now be good to go. The cloudbridge update has been made and upstream libraries fixed. Onedata continues to have the installation and configuration issue, but this can in principle be done, even if the process is a bit tedious, so I'd propose we merge this anyway in the hope that those issues are resolved upstream at a later date.

Adding oauth support to streamline the process of obtaining tokens etc. would also be a future enhancement.

[luke-c-sargent]

@nuwang good stuff! I tested out GCS/GDrive access, and had some success and some issues; preliminary thoughts:

Google Drive:

• binary files work a-ok
• I tried a Google doc download, and it gave:
  "Only files with binary content can be downloaded. Use Export with Google Docs files."
  • a stackoverflow post about it
  • seems like, from the post, google-native formats (slides, sheets, docs, etc) need a different method

GCS:

• created an open access bucket, able to browse it in Galaxy, but get Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS when trying to download any files
  • to resolve this error when testing the anvil pyfilesystem2 plugin i prepend an export of the above env var to sh run.sh, but that does not seem to work in this case

I will keep testing; please let me know if there is something I am missing as its entirely possible i've just borked up my configuration

[nuwang]

@luke-c-sargent Thanks for testing. The google drive issues is more of an upstream issue I guess: https://github.com/rkhwaja/fs.googledrivefs/blob/9568eb49a9084d4d9751eb27fb558ee77558cbfc/fs/googledrivefs/googledrivefs.py#L427
As that Stack Overflow post you linked suggests, docs need the export_media method, but I don't know whether that breaks normal files. I guess we should file a bug there?

Regarding GCS, this will work if you plug in your OIDC credentials. However, there is an anonymous access mode for public buckets, let me see whether I can activate that, but we'll need an extra setting like: anonymous: true in file_sources_conf.

[luke-c-sargent]

I guess we should file a bug there?

👍 issue filed. TL;DR: they could detect mimetype and change the download mechanism for google native items easily enough; the only considerations i can see are a) what type to export stuff to (e.g. sheets as csv or excel?) and b) what to do about files >10MB that export_media() disallows.

Regarding GCS, this will work if you plug in your OIDC credentials

ahh i only used parameters from test/unit/files/gcsfs_file_sources_conf.yml, and as such did not have any tokens in the mix. works now!

one additional note - when using an old token the refresh fails with:

 Problem listing file source path FileSourcePath(file_source=<galaxy.files.sources.googledrive.GoogleDriveFilesSource object at 0x120b968e0>, path='/')
<...>
google.auth.exceptions.RefreshError: The credentials do not contain the necessary fields need to refresh the access token. You must specify refresh_token, token_uri, client_id, and client_secret.

you mentioned that this is preliminary work that requires a better OIDC solution ...is the refresh token a placeholder for future functionality or have i misconfig'd something again?

[nuwang]

@luke-c-sargent Thanks for filing the issue, looks like it's already been fixed since!

Have added back client_id, client_secret and token_uri, so that should now be specifiable along with the refresh token.

Also, I've added an anonymous: true option for GCS public buckets, so credentials will no longer be required.

[nuwang]

@mvdbeek Would you be able to give a quick once over on this? In particular, whether the regeneration of requirements looks ok?

[mvdbeek]

That should be a conditional requirement and not appear in pyproject.toml

[nuwang]

How do I specify it as a conditional requirement only for tests?

[mvdbeek]

I'd maybe do it the other way round and skip the test if the dependency can't be imported.

[nuwang]

So are you suggesting not running the test by default? Since it's accessing a public bucket, the test can be run successfully, provided fs-gcsfs is available.

[mvdbeek]

Yes, ideally unit tests shouldn't contact external services. We can run release tests with the conditional dependencies installed.

[nuwang]

BTW, fs-gcsfs appears to have been correctly included in dev-requirements, but not pinned requirements. Isn't that ok?

[mvdbeek]

Though we already do that for some other tests, I guess that is just a personal preference. If you want to run this one then this is ok and we do need to add it

[nuwang]

I've rolled back the always on gcsfs test in favour of skipping the test if the dependency is unavailable.

[mvdbeek]

The added unit tests are all using the same test structure, can you parameterize them ?

[nuwang]

Yes, I think they can be. It's something that's generally the case for many file source tests. Maybe a refactoring run that should be done outside of this PR?

[mvdbeek]

Can you at least push _configured_file_sources into the helper ? That is literally the same in all those tests.

[nuwang]

Have refactored the tests and helpers. Can you take a look?

[mvdbeek]

Thanks a lot, I really appreciate the test simplification