Design document for the Zenodo like DOI per dandiset #2012

yarikoptic · 2024-08-22T01:45:53Z

A design doc composed with @djarecka to avoid dummy DOIs for dandisets

refs:

TODOs

complete initial pass
seek review
make explicit that DOI generation is optional overall, since LINC does not even need it (ref)
potentially add a sequence diagram of interactions between user, archive, datacite across different stages of embargoed -> public dandisets

but could already be checked out by @dandi/archive-maintainers folks since overall idea is formulated already and some early concerns/questions could already be asked/answered

doc/design/doi-generation-2.md

djarecka · 2025-01-19T00:43:08Z

I created some test to simulate the workflow in dandi/dandi-schema#275

yarikoptic · 2025-03-05T20:34:37Z

doc/design/doi-generation-2.md

+  - We might want a dedicated 404 page for deleted dandisets, or at least a message that the dandiset was deleted, and ideally describe the reason why it was deleted ("Upon request of maintainer", "Due to violation of terms of service", etc.)
+  - Then we adjust DOI record to point to that page.
+
+- Should we do anything at dandischema level?


@djarecka we are to answer those I think before we could call this "done" ;-)

doc/design/doi-generation-2.md

yarikoptic · 2025-04-14T20:09:14Z

doc/design/doi-generation-2.md

+- Upon changes to a non-embargoed, draft dandiset metadata record:
+  - If `Draft DOI`, attempt to "promote" it to `Findable`.
+    - If validation fails - keep `Draft DOI` (very limited validation), attempt to update datacite metadata record while keeping the same target URL.
+    - **Question to clear up**: what happens to `Draft DOI` if metadata record is invalid? It seems to create one with no metadata, but does it update only the fields it knows about?


@djarecka I feel like you clarified on this but we did not put it "in writing" here. What do you remember on this aspect?

ping @djarecka

yes, sorry, i just gave "thumbs up", since you were right, Draft DOI doesn't have to have metadata, it can only have url, and if has more, it will be added

doc/design/doi-generation-2.md

asmacdo · 2025-04-23T13:08:17Z

doc/design/doi-generation-2.md

+
+A django-admin script should be created and executed to create a `Dandiset DOI` for all existing dandisets.
+
+**Question to address**: Will adding a `Dandiset DOI` in addition to `Version DOI` require a db migration?


In the POC I've added a doi field to the Dandiset model which does add a migration.

isn't there also a "draft" Version with DOI (and that's where I guess we inject a fake one), i.e. could we avoid changing DB model?

I have to admit that I "just dont like that" since the Dandiset DOI semantically belongs to the Dandiset. I cant think of a reason this wouldnt work, but just feels messy.

To retrieve the Dandiset DOI via Django, someversion.dandiset.draft_version.doi IMO violates the "principle of least surprise".

Prior to publication the Dandiset DOI will point to the draft version (via the DLP) but after publication the dandiset DOI will point to the latest publication, so that would also be surprising

doc/design/doi-generation-2.md

asmacdo · 2025-04-23T13:12:10Z

doc/design/doi-generation-2.md

+  - **Follow up concern**: after dandiset and DOI publish, metadata of the Draft version of the dandiset could still be changed.
+    This potentially making changed record again "invalid".
+    Should be Ok'ish
+- Test site of datacite had different result of validation that the primary one


Do we have any more information about this?

It might have been a bad memory, @djarecka do you have any information on this or should we just remove this?

Suggested change

- Test site of datacite had different result of validation that the primary one

let's remove it. It was based on my memory of some old issue, but wasn't able to reproduce, or find old records.

doc/design/doi-generation-2.md

asmacdo · 2025-04-24T16:25:49Z

For dandi-schema, we may also want to pull the validation out of to_datacite. This validation does not use the datacite API, rather validation occurs against the schema which has been pre-fetched and committed into the dandi-schema repo.

Currently to_datacite accepts an optional arg validate, which defaults to False-- the only use of to_datacite in dandi-archive DOES NOT enable validation!. I'm curious what problems this could cause, and if those problems have actually occurred-- maybe there are more Dandisets with DOIs stored in the data model without a DOI minted? Or maybe they were created but set to Draft due to validation errors? We would have logged exceptions if the datacite API did not accept a new DOI.

From this design doc only, I gather that we believe Draft DOIs have less stringent validation than Findable, but I have not found any upstream documentation that confirms this. If it is the case, I suggest we pull the validation out of the to_datacite function, which would only be responsible for constructing the API payload. Then, in dandi-archive, we perform a validation. If valid we publish a Findable DOI, and if invalid we fallback to Draft DOI if it fails validation.

Either way, we need to consider using some kind of validation. @djarecka @yarikoptic I suggest this as topic for our discussion tomorrow.

djarecka · 2025-04-25T15:13:58Z

For dandi-schema, we may also want to pull the validation out of to_datacite. This validation does not use the datacite API, rather validation occurs against the schema which has been pre-fetched and committed into the dandi-schema repo.

correct!

Currently to_datacite accepts an optional arg validate, which defaults to False-- the only use of to_datacite in dandi-archive DOES NOT enable validation!. I'm curious what problems this could cause, and if those problems have actually occurred-- maybe there are more Dandisets with DOIs stored in the data model without a DOI minted? Or maybe they were created but set to Draft due to validation errors? We would have logged exceptions if the datacite API did not accept a new DOI.

That's probably not good, if the validation is not used before publishing. Was not aware.

From this design doc only, I gather that we believe Draft DOIs have less stringent validation than Findable, but I have not found any upstream documentation that confirms this. If it is the case, I suggest we pull the validation out of the to_datacite function, which would only be responsible for constructing the API payload. Then, in dandi-archive, we perform a validation. If valid we publish a Findable DOI, and if invalid we fallback to Draft DOI if it fails validation.

Yes, no documentation, but that's the case...

doc/design/doi-generation-2.md

yarikoptic · 2025-05-09T21:45:30Z

doc/design/doi-generation-2.md

+## Concerns to keep in mind/address
+
+- **Question to clear up**: what happens to `Draft DOI` if metadata record is invalid?
+    - It seems to create one with no metadata, but does it update only the fields it knows about?


@djarecka did you check in your experiments?

doc/design/doi-generation-2.md

- Dandiset DOI will redirect to the DLP - Example: 10.80507/dandi.000004 - Dandiset DOI is stored in the doi field of the draft version - Dandiset DOI metadata (on Datacite) will match the draft version until first publication - Once a Dandiset is published, the Dandiset DOI metadata will match the latest publication See the design document for more details: dandi#2012

…ndisets

…ublished" draft version of dandiset

…koptic discussion

…previous failures Clarify no DOIs for dandisets while embargoed

Co-authored-by: Austin Macdonald <[email protected]>

could not reproduce

Co-authored-by: Yaroslav Halchenko <[email protected]>

fallback to Dandiset pydantic model would require changes to that model.

yarikoptic mentioned this pull request Aug 23, 2024

Create and maintain a "Findable" DOI for the Dandiset as a whole #1319

Open

djarecka reviewed Sep 30, 2024

View reviewed changes