Design of issue tracking/reporting via GitHub

[yarikoptic]

To address #863

TODOs:

• incorporate feedback from Roni

[CodyCBakerPhD]

Overall I think this looks good; one thing I would strongly recommend is a way to ensure the dandiset owners are alerted whenever an issue is reported

Ideas from the meeting included using the dandi user ID (which is GitHub ID too) to auto subscribe to the repo upon creating or otherwise use the '@' operator pre-populated in the issue template

[bendichter]

I agree, this looks good to me.

[yarikoptic]

having discussed this a little more during ODIN with @rly (also ping @magland) -- decided to switch approach a little to

• not rely on @handles but rather to invite authors to become Contributors in Triage role.
• aim to develop a "bot" which would perform prescribed actions (user management, changes acceptance, etc) - Design a "bot" to assist "managing" of the dandisets by authorized users dandisets#360

Please chime in with what do you think - may be there is some complication you can see which I have missed.

[bendichter]

@yarikoptic I agree with the proposed changes and with the justification of those changes.

[waxlamp]

Overall, I think this could be a good way to handle dandiset-specific issues. I have the following concerns:

• How do future deployments of dandi-archive deal with this? There won't always be a magical DataLad org filled with one repository per dandiset. Perhaps a better way to do this is simply to use a single issue tracker for all issues (tagged with dandiset ID, perhaps); then, the configuration for a dandi archive deploy can point to a single issue tracker to fulfill this functionality.
• What are the alternative approaches here? It seems a little bit odd for neuroscientists to use a code-specific issue tracker to report problems with datasets. Is there a different service out there that might integrate more cleanly with the archive?

[bendichter]

@waxlamp Thanks for the thoughts.

There won't always be a magical DataLad org filled with one repository per dandiset.

Why not? Why do you consider this magic?

Perhaps a better way to do this is simply to use a single issue tracker for all issues

We could do this but we'd need to sort out a few features.

1. How to associate issues with specific dandisets. (could be done with an issue form I guess)
2. How to notify the dandiset owners when an issue is filed in relation to that dandiset.
3. An easy way for users to filter issues by dandiset

Is there a different service out there that might integrate more cleanly with the archive?

There certainly are other ticketing systems out there, but GitHub is free and more familiar to neuroscientists than any other platform. It also has a number of useful features:

1. We could set up GitHub Actions to automate processing of issues.
2. We can assign issues to projects to prioritize them
3. GitHub has a REST API we can use to integrate issues with the main website if we want to

[yarikoptic]

Valid concerns, thanks @waxlamp and thanks @bendichter for the points too!

After all we also have staging dandi and it would need to have "its own" . I think it might be possible to plan for both ie template out the URLs for issues, given e.g. {gh_org} (e.g. dandisets), gh_repo (e.g. empty '' or 'dandisets-staging/' if a single repo), and {dandiset} placeholder

1. main instance - per dandiset repo like we have gh_org = dandisets, gh_repo = ''
   • list of issues: https://github.com/{gh_org}/{gh_repo}/{dandiset}/issues, e.g. https://github.com/dandisets/000026/issues
   • file a new issue: https://github.com/{gh_org}/{gh_repo}/{dandiset}/issues/new e.g. https://github.com/dandisets/000026/issues/new
2. staging - a single repo, e.g. https://github.com/dandi/dandisets-staging : gh_org = dandi, gh_repo = dandisets-staging

• list of issues: https://github.com/{gh_org}/{gh_repo}/issues?q=is%3Aissue+is%3Aopen+%5B{dandiset}%5D e.g. https://github.com/dandi/dandisets-staging/issues?q=is%3Aissue+is%3Aopen+%5B600789%5D
• file a new issue: https://github.com/{gh_org}/{gh_repo}/issues/new?title=[{dandiset}]: e.g. https://github.com/dandi/dandisets-staging/issues/new?title=[600789]:%20

But such flexibility might complicate future "tighter" integration, e.g. querying for a number of open issues etc to include that information on DLP.

Probably a more lightweight alternative to aiming for 2 setups would be to concentrate on the proposed one here, and

1. Setup dandisets-staging organization
2. Add minimal functionality of establishing a new dandiset repository under a specified organization to dandi-archive itself (now it is part of the dataladification script)
   • it would require specification/storing of a github auth token for each instance.
   • would not be called if no configuration to do so is setup (e.g. in the unittests)
3. Adjust dataladification script to "tollerate" a repository already existing but empty (I think we actually do not need to do anything since we have existing="reconfigure": https://github.com/dandi/dandisets/blob/HEAD/tools/backups2datalad/adataset.py#L374 ).

@waxlamp WDYT about adding functionality to dandi-archive to call out to GITHUB API to create a new repository upon request to create a new dandiset? Or would you prefer to go "2 possible setups" way? (that would require only 1 time manual setup but more parametrization for each instance)

[magland]

So if there are thousands of dandisets, there would be thousands of gh repos under an organization maintained by dandi? Seems like it could become difficult to manage. What about having the author of the dandiset optionally contribute and maintain their own gh repo for this?

[yarikoptic]

So if there are thousands of dandisets, there would be thousands of gh repos under an organization maintained by dandi?

yes. Glorious the time when we get 1000s of nice non-empty datasets! we will celebrate!

Seems like it could become difficult to manage.

with the right tools should be doable and not unprecedented:

• we already have them all under https://github.com/dandisets/ and https://github.com/dandizarrs
• OpenNeuro also uses https://github.com/OpenNeuroDatasets/ and https://github.com/OpenNeuroDerivatives/

What about having the author of the dandiset optionally contribute and maintain their own gh repo for this?

an interesting idea. In principle we can already allow people to link github repos within dandiset metadata.

quick grep already shows a good number of mentionings of github within dataset metadata

dandi@drogon:/mnt/backup/dandi/dandisets$ grep github */dandiset.yaml | grep -v dandi/schema
000008/dandiset.yaml:  url: https://github.com/berenslab/mini-atlas
000016/dandiset.yaml:  NWB files is provided at https://github.com/ttngu207/najafi-2018-nwb/blob/master/notebooks/Najafi-2018_example.ipynb.'
000026/dandiset.yaml:  url: https://biccn.github.io/Quarterly_Submission_Receipts/000026-dashboard.html
000027/dandiset.yaml:  ATM contains only a few files from http://github.com/dandi-datasets/nwb_test_data
000035/dandiset.yaml:  url: https://github.com/berenslab/mini-atlas
000037/dandiset.yaml:  url: https://colleenjg.github.io/
000037/dandiset.yaml:  url: https://github.com/jeromelecoq/allen_openscope_metadata/tree/master/projects/credit_assignement
000037/dandiset.yaml:  url: https://github.com/colleenjg/OpenScope_CA_Analysis
000037/dandiset.yaml:  url: https://github.com/colleenjg/cred_assign_stimuli
000060/dandiset.yaml:  here: \n https://github.com/arsenyf/FinkelsteinFontolan_2021NN"
000064/dandiset.yaml:description: This is data produced by the Soltesz Lab NeuroH5 software (https://github.com/iraikov/neuroh5).
000064/dandiset.yaml:  The data has been converted to NWB using the ndx-simulation-output extension (https://github.com/catalystneuro/ndx-simulation-output).
000064/dandiset.yaml:  url: https://github.com/iraikov/neuroh5
000108/dandiset.yaml:  url: https://biccn.github.io/Quarterly_Submission_Receipts/000108-dashboard.html
000122/dandiset.yaml:  can be found at https://github.com/rob-luke/experiment-fNIRS-tapping.
000122/dandiset.yaml:  url: https://github.com/rob-luke/BIDS-NIRS-Tapping
000122/dandiset.yaml:  url: https://github.com/rob-luke/experiment-fNIRS-tapping
000127/dandiset.yaml:  of the Neural Latents Benchmark: https://neurallatents.github.io.'
000128/dandiset.yaml:  Latents Benchmark: https://neurallatents.github.io.'
000129/dandiset.yaml:  Neural Latents Benchmark: https://neurallatents.github.io.'
000130/dandiset.yaml:  of the Neural Latents Benchmark: https://neurallatents.github.io.'
000138/dandiset.yaml:  Benchmark: https://neurallatents.github.io.'
000139/dandiset.yaml:  Benchmark: https://neurallatents.github.io.'
000140/dandiset.yaml:  Benchmark: https://neurallatents.github.io.'
000165/dandiset.yaml:  url: https://github.com/emilyasterjones/interneurons_modulate_drive
000168/dandiset.yaml:  repository: github
000168/dandiset.yaml:  url: https://github.com/rozmar/jGCaMP8_ground_truth_dataset
000207/dandiset.yaml:  Example code on how to plot this data can be found at https://github.com/rutishauserlab/cogboundary-zheng
000207/dandiset.yaml:  repository: github
000207/dandiset.yaml:  url: https://github.com/rutishauserlab/cogboundary-zheng
000221/dandiset.yaml:  Example codes to plot data is at https://github.com/hidehikoinagaki/InagakiAndChenEtAl2022'
000222/dandiset.yaml:  Code and README can be found at https://github.com/JustinOHare/ICR_2022.git'
000231/dandiset.yaml:  repository: github
000231/dandiset.yaml:  url: https://github.com/cxrodgers/NwbDandiData2022
000402/dandiset.yaml:  url: https://github.com/datajoint/microns_phase3_nda
000404/dandiset.yaml:  https://github.com/pkhanna104/bmi_dynamics_code and archived at https://zenodo.org/record/8006653"
000405/dandiset.yaml:  table.\n\nPre-print DOI: \nhttps://doi.org/10.1101/2022.12.15.520660\n\nGithub:\nhttps://github.com/alexgonzl/TMA\n"
000462/dandiset.yaml:  Scripts used for analysis can be found on https://github.com/seethakris/HPCrewardpaper'
000465/dandiset.yaml:  [Electrode mapping information & Basic analysis codes] Github: https://ytchoe.github.io/'
000469/dandiset.yaml:  provided: \nhttps://github.com/rutishauserlab/workingmem-release-NWB\n"
000469/dandiset.yaml:  url: https://github.com/rutishauserlab/workingmem-release-NWB
000473/dandiset.yaml:  url: https://github.com/PierreLeMerre/Esr1_NPX_code
000483/dandiset.yaml:  url: https://github.com/ucsb-goard-lab/Neurotar-HD-Experiments
000540/dandiset.yaml:  on https://rhythm-n-rodents.github.io/software/.
000554/dandiset.yaml:  [Electrode mapping information & Basic analysis codes] Github: https://ytchoe.github.io/'
000557/dandiset.yaml:  https://ytchoe.github.io/"
000574/dandiset.yaml:  url: https://github.com/janhohenheim/usz-neuro-conversion
000574/dandiset.yaml:  url: https://github.com/janhohenheim/nwb-example
000575/dandiset.yaml:  url: https://github.com/janhohenheim/usz-neuro-conversion
000575/dandiset.yaml:  url: https://github.com/janhohenheim/nwb-example
000576/dandiset.yaml:  url: https://github.com/janhohenheim/usz-neuro-conversion
000576/dandiset.yaml:  url: https://github.com/janhohenheim/nwb-example
000579/dandiset.yaml:  notebook for a tutorial to read and extract information from these NWB files: https://github.com/sytseng/Notebook_for_Dandiset_000579\n\n-
000579/dandiset.yaml:  NWB extension code for custom lab meta data (required for reading NWB files): https://github.com/sytseng/ndx-harvey-swac
000579/dandiset.yaml:  \n\n- Code and tutorials for fitting GLM to neural activity in Tensorflow 2: https://github.com/sytseng/GLM_Tensorflow_2"
000579/dandiset.yaml:  url: https://github.com/sytseng/Notebook_for_Dandiset_000579
000579/dandiset.yaml:  url: https://github.com/sytseng/ndx-harvey-swac
000579/dandiset.yaml:  url: https://github.com/sytseng/GLM_Tensorflow_2
000582/dandiset.yaml:    url: https://github.com/dandi/dandi-cli
000618/dandiset.yaml:  This dataset was prepared using the following script: https://github.com/flatironinstitute/spikeforest/blob/main/devel/dandiset/prepare_dandiset.py
000623/dandiset.yaml:  Git Link: https://github.com/rutishauserlab/bmovie-release-NWB-BIDS'
000625/dandiset.yaml:  email: [email protected]
000630/dandiset.yaml:  Analysis code and extracted features available at https://github.com/AllenInstitute/patchseq_human_L1.
000630/dandiset.yaml:  Feature extraction package available at https://github.com/AllenInstitute/ipfx.'
000678/dandiset.yaml:  url: https://github.com/sjara/uobrainflex/tree/master/hulsey2023

Might be worthwhile working out a complete use case example. FWIW

Cons I see

• might end up being "more difficult" due to possible various ways they decide to label etc issues. In particular if we were to do some "overall overhaul" (e.g. that consistent labeling etc)
• it would be for authors to do that, and they likely would not, and users end up without a dandiset specific issues board linked from the dandiset landing page

[bendichter]

Does GitHub limit the number of repos in an organization?

Edit: Answer: no. "All organizations can own an unlimited number of public and private repositories." (source)

[yarikoptic]

Ok, seems no further questions/concerns. I have added section on minimal developments to be done on dandi-archive backend, and made it all conditional on having github organization provisioned.

@jwodder please also have a look/provide feedback.

[waxlamp]

I am tentatively ok with this design (we will learn more about its suitability/viability as we go), but I'd like to also consult with others in Kitware who may know of alternatives we haven't covered.

[yarikoptic]

@bendichter mentioned https://github.com/giscus/giscus which interfaces discussions (not issues), and also that opens the ecosystem of other attempts at similar platforms (https://github.com/gitalk/gitalk, https://github.com/utterance/utterances etc) but they all seems to be "dead" as no devel/support.

[bendichter]

Discussion may actually be a slightly better option. It psychologically opens up the discussion to messages like questions or explanations rather than just problems. An issue can easily be created from a discussion as well in the GitHub web interface.

[waxlamp]

#1313 mentioned utterances, which builds a comment stream over github issues. That is something to consider as well.

[yarikoptic]

attn @bendichter on his request during DANDI/NWB meetup. Can resolve later