Is using GitHub to share research data a good idea?

[karres-illinois]

Hello

I am aware of an active research project that is being sponsored by a number of non-campus companies. The project has passed IRB review. The project takes in audio data from research subjects. An on-campus team deidentifies the audio data and measures whatever they measure in relation to the samples. The deidentified sound samples along with the accompanying deidentified
measurement "metadata" is to be shared with select representatives of the sponsoring companies.

The thought is to use a private GitHub repo to store a monthly "dump" of all collected data with the company reps. Questions are:

• Is this a wise choice in general? The researchers have other sharing options but want to explore using GitHub.

• Research participants can remove themselves from the project. If they do, there is an option for them to delete their research samples. This would be reflected in the corresponding files not being in the next data "release". The sponsoring companies are supposed to also remove "missing"/deleted data. This brought up the question of GitHub version history. If participants have their files removed and a new version of the data is uploaded that does not contain those files then are the removed files still in the "old" version history? Is there a way to purge history with each noew release?

[edthedev]

Files removed from a GitHub repository are not removed from the history.

It is possible to remove files from the history, with advanced git commands. The commands and tools available for fully removing files from history are designed for the rare accidental case that something gets added that should never have been added in the first place, such as large generated files or secrets.

To me, git doesn't sound like a good fit for this use case.

[jtgorman]

I'd also add in while it's not impossible, tracking large files and binary files in git and github has some extra headaches. I could see using it for the metadata, but I'd also wonder how frequently that changes. It might be better to use some sort of object store that has version history.

[karres-illinois]

Using GitHub came up mostly because the researchers "know it" and creating a private repo that they invite the reps into seemed like an easy answer. "Versioning" really isn't a primary concern in this project.

[jtgorman]

I also just remembered another caveat. Changing the history of the repository does NOT necessarily change the history of the various distributed repositories. There can be issues if people don't do a fresh checkout if I remember right. If they're only using the web interface for uploading files, it's probably ok. But if they have multiple people who are pushing stuff up, someone could "fix" the history and then it gets brought back in a later push.

[dlstrong]

If what you're talking about sharing is sound files in particular, and if they stay as sound files afterward -- you might want to look at Mediaspace and its Unlisted publishing status? (That's where people who have the link can find the file but it's not searchable in the broader Mediaspace directory.)

Mediaspace's auto transcription tools aren't perfect, but it;s also a TON faster for a first pass than having a human have to figure out both the words and the fraction-of-a-second timing to caption from scratch, and the web interface for the transcripts allow easy editing for free (much easier than getting and learning the Adobe suite or something similar).

And folks with system admin powers (I can introduce you to a few but I'm not going to name them out on GitHub :D) can do a lot of bulk and batch processing stuff.

I've worked with some researchers who found that bulk uploading their sound files to Mediaspace and then having the grad students clean it up was hundreds of hours and thousands of dollars faster / cheaper than doing everything from scratch in other tools.

Might be worth a look.

[1heidi]

Lots of useful comments here. I'll just add that for active projects in general, it’s certainly common and convenient for many to have research data in Github repos, though we don’t recommend using it as a permanent archive, especially for a private repo which are not part of Github’s Archive Program. And alternative back-ups are always important (I haven’t used Git-annex myself but great that Matt mentioned it).

Re: the human subjects aspect does make this question more complicated. Jon’s point about various distributed repositories is important. It could be okay, but it depends on what the participants consented to – for example, there is guidance that specifically calls out that information already shared with others cannot be taken back.

https://research.illinois.edu/sites/default/files/Representing%20Data%20Sharing%20in%20Informed%20Consent%20Guidance%20for%20Researchers_0.pdf

If they do want to move forward with using Github for the project, I recommend updating the IRB office and consulting with [email protected] just to make sure all’s copacetic.

[matthewturk]

git-annex has a bit of a learning curve, but is what I've found for this specific use case. Before Box disabled WebDAV it was usable with the box.illinois.edu service through git-annex.