OpenAIRE guidelines for Data Archives

[abollini]

This PR introduce compliancy with the OpenAIRE guidelines for Data Archives, see
https://guidelines.openaire.eu/en/latest/data/index.html
as requested in the issue #4257

It is target to the new version of the guidelines the 4.0 not yet published but it was anticipated that the main change will be the switch to the datacite schema 4.1

This development has been provided by 4Science thanks to the funding from the OpenAIRE Open Call for Innovation Jan 2018 see https://www.openaire.eu/openaire-tender-calls-winners

As we doesn't currently host installation with real data it will be useful to have some feedback and test from Insititution with data already suitable for exposure to openaire, better if with reference to EC funded projects.

---

Update by @pdurbin on 2018-05-14

connects to #4257 As an installation admin, I want my repository to export OpenAIRE-compliant metadata to improve discoverability, reusability of research data

I added the "connects to" line above to associate this pull request with the issue it addresses in Waffle ( https://waffle.io/IQSS/dataverse ).

---

Update by @pdurbin on 2018-05-17

connects to #3697 Users should be able to download DataCite metadata from Export Metadata pulldown

[pdurbin]

@abollini thanks for the pull request! The first think I noticed is that the Travis build failed with a compilation error. If you click "Details" ...

[FIXME]

you will be taken to https://travis-ci.org/IQSS/dataverse/builds/378185244 and if you scroll down you can see the error:

[FIXME]

I'm sorry but I can't seem to upload images from the airport WiFi in Qatar so I put FIXME above for now.

Can you please take a look at the tests that are failing. We run the test suite on every build. I'm not sure if this is a Netbeans default. You seem to be using Eclipse. Update: whoops, the failure is not a test, it's in ImportDDIServiceBean.java.

[pdurbin]

Can you please look at the compilation failure in ImportDDIServiceBean.java?

[coveralls]

Coverage increased (+1.2%) to 19.596% when pulling 5d876aa on 4Science:openaire into 5991f9c on IQSS:develop.

[pameyer]

06ce109 is ok for integration tests: Tests run: 72, Failures: 0, Errors: 0, Skipped: 0

[juancorr]

getRecords.zip

Hi @pdurbin , there are some errors in the GetRecord and ListRecords oai verbs yet. I have checked the oai results with the jhove tool from ubuntu. There are linked in this comment the http://oaimadrono.uned.es:8080/oai?verb=GetRecord&metadataPrefix=oai_datacite&identifier=doi:10.5072/FK2/040BAT result original and one other updated that is xml valid. To obtain a result 100% valid is recommendable to use the OpenAIRE validator: https://www.openaire.eu/validator/populateRules.action .

There are some validation errors that are only capital letters typos and some mandatory fields missing Here are some examples:

ErrorMessage: cvc-complex-type.2.4.a: Se ha encontrado contenido no válido a partir del elemento 'nameIdentifier'. Se esperaba uno de '{"http://datacite.org/schema/kernel-4":affiliation}'.: Line = 21, Column = 46

ErrorMessage: cvc-enumeration-valid: El valor 'isCitedBy' no es de faceta válida con respecto a la enumeración '[IsCitedBy, Cites, IsSupplementTo, IsSupplementedBy, IsContinuedBy, Continues, IsNewVersionOf, IsPreviousVersionOf, IsPartOf, HasPart, IsReferencedBy, References, IsDocumentedBy, Documents, IsCompiledBy, Compiles, IsVariantFormOf, IsOriginalFormOf, IsIdenticalTo, HasMetadata, IsMetadataFor, Reviews, IsReviewedBy, IsDerivedFrom, IsSourceOf, Describes, IsDescribedBy, HasVersion, IsVersionOf, Requires, IsRequiredBy]'. Debe ser un valor de la enumeración.: Line = 63, Column = 76

ErrorMessage: cvc-attribute.3: El valor 'isCitedBy' del atributo 'relationType' del elemento 'relatedIdentifier' no es válido con respecto a su tipo, 'relationType'.: Line = 63, Column = 76

ErrorMessage: cvc-enumeration-valid: El valor 'handle' no es de faceta válida con respecto a la enumeración '[ARK, arXiv, bibcode, DOI, EAN13, EISSN, Handle, IGSN, ISBN, ISSN, ISTC, LISSN, LSID, PMID, PURL, UPC, URL, URN]'. Debe ser un valor de la enumeración.: Line = 63, Column = 76

ErrorMessage: cvc-attribute.3: El valor 'handle' del atributo 'relatedIdentifierType' del elemento 'relatedIdentifier' no es válido con respecto a su tipo, 'relatedIdentifierType'.: Line = 63, Column = 76

ErrorMessage: cvc-datatype-valid.1.2.1: '

This work is licensed under a Creative Commons Attribution 4.0 International License.

' no es un valor válido para 'anyURI'.: Line = 74, Column = 288

ErrorMessage: cvc-attribute.3: El valor '

This work is licensed under a Creative Commons Attribution 4.0 International License.

' del atributo 'rightsURI' del elemento 'rights' no es válido con respecto a su tipo, 'anyURI'.: Line = 74, Column = 288

ErrorMessage: cvc-complex-type.2.4.b: El contenido del elemento 'resource' no está completo. Se esperaba uno de '{"http://datacite.org/schema/kernel-4":publisher, "http://datacite.org/schema/kernel-4":publicationYear, "http://datacite.org/schema/kernel-4":resourceType, "http://datacite.org/schema/kernel-4":alternateIdentifiers, "http://datacite.org/schema/kernel-4":geoLocations}'.: Line = 100, Column = 12

I am available for any further details or tests

[pdurbin]

@juancorr thanks for testing. If @abollini gives you push access to his branch, are you able to work on these issues?

[djbrooke]

ping @abollini - I think the feedback above is for you. Thanks @juancorr for the feedback!

[juancorr]

Thanks @pdurbin, yes I can work until 24 Mai, I will out of the office two of three weeks after this date.

[pdurbin]

@juancorr ok, and you could always push a branch to your own fork and make a pull request from your fork to @abollini 's fork. That way he wouldn't need to give you push access.

[abollini]

Hi @pdurbin , @juancorr thanks for the feedback, sorry for the delay in the response but I was travelling in the past days. You should be able to add commit yourself as we keep flagged the "Allow edits from maintainers." flag, anyway we can make the chages ourself on Monday if you don't have done them already!

Thanks again for the fast feedback

[pdurbin]

@abollini no problem and thanks again for the pull request. If you could make the changes, it would be great. @juancorr is not a maintainer so he won't be able to push to your branch unless you give him special access. I'll let you two coordinate how you'd like to make further improvements to the code before we advance it to code review at https://waffle.io/IQSS/dataverse . Please just let us know when you're ready. One other bit of feedback I'd like to give is that it would be nice if there were JUnit tests for OpenAireExporter.java. If this is too much trouble, don't worry about it, but you could look at SchemaDotOrgExporterTest.java as an example.

(Thank you to @francescopioscognamiglio for fixing the compilation error in 06ce109 !)

[juancorr]

Thanks @abollini and @pdurbin, I can have the code ready Monday or Tuesday

[pdurbin]

No rush but please note that there are now merge conflicts because pull request #4691 was merged.

[lap82]

As said at 4Science#2 we working on cleaning code merged from @juancorr and we will try to introduce the publisher (as @juancorr pointed out that "Publisher is mandatory in xsd file (not in guidelines)"). We will share asap the code in this PR. Thank you!

[pdurbin]

@lap82 thanks for the update and for pointing out 4Science#2 to me. Sounds like great progress! Thanks!

[francescopioscognamiglio]

@pdurbin I done modification and the PR is ready to be verified. We have trouble to write unit test we don't able to provide it before 2 weeks. I hope that it's ok for you.

[pdurbin]

@francescopioscognamiglio thanks! It sounds like you're ready for code review so I just dragged this pull request (and #4257 and #3697) to that column at https://waffle.io/IQSS/dataverse

[djbrooke]

Hi @abollini @lap82 and @francescopioscognamiglio (and anyone else interested) - we'll soon start taking a look at the code, but can you add some guides documentation as well? If you're unable to update the .rst files directly, a document that we can use to update the documentation would be fine as well! Thanks for the work on this!

[pdurbin]

Specifically, adding the new export format to this file would be a good start: doc/sphinx-guides/source/admin/metadataexport.rst

[pdurbin]

I see your examples and I will correct the contributor's name type to handle the proposed case as a non-personal nameType

@fcadili hi! I'm not too worried about the Guido van Rossum example I added in 03c959b if that's what you mean.

Much more important than anything I wrote in my review of this pull request is the feedback from QA at #4257 (comment) . I just want to make sure you've seen that comment. When you're ready, please reply on the issue. Thanks!!

[fcadili]

An update version of code was committed.

The following elements was improved:

• Organizational value of contributor's nameType is used when contributorName is not a person.

[fcadili]

I posted an updated version of the code that resolve @kcondon questions.

nameType attribute is still used, but it can be removed when needed.
However I note when we replace:

<creators>
    <creator>
      <creatorName nameType="Personal">Finch, Fiona</creatorName>
      <givenName>Fiona</givenName>
      <familyName>Finch</familyName>
      <nameIdentifier nameIdentifierScheme="ORCID">0000-0002-9528-9470</nameIdentifier>
      <affiliation>Birds Inc.</affiliation>
    </creator>
  </creators>

with

<creators>
    <creator>
      <creatorName>Finch, Fiona</creatorName>
      <givenName>Fiona</givenName>
      <familyName>Finch</familyName>
      <nameIdentifier nameIdentifierScheme="ORCID">0000-0002-9528-9470</nameIdentifier>
      <affiliation>Birds Inc.</affiliation>
    </creator>
  </creators>

we sill need to establish that creatorName is a person, since elements givenName and familyName have to be generated.

[pdurbin]

@fcadili hmm. Good point about those separate fields for first and last name. Is there a database setting to control the loadHint boolean? If not, how does a sysadmin change it?

[fcadili]

At the moment there is no database field. However it can be done with some help to identify database table and column and method used to change the value (using api or direct database modification).

[fcadili]

I released a new version: the problem was related to resource files that was not deployed with dataverse.war. I checked all @kcondon examples and all seems work fine.

[fcadili]

I released a new version which fixed Organizational nameType. I've checked all @kcondon examples and they work fine with the new version.

[fcadili]

I released a new version which fixes Operational nameType when the name contains commas or dashes. I've checked the latest @kcondon examples and they work fine with the new version.