Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Importing and Exporting DCAT Metadata #1592

Open
posixeleni opened this issue Mar 5, 2015 · 19 comments
Open

Support for Importing and Exporting DCAT Metadata #1592

posixeleni opened this issue Mar 5, 2015 · 19 comments
Assignees
Labels
Feature: Harvesting Feature: Metadata Type: Feature a feature request User Role: Superuser Has access to the superuser dashboard and cares about how the system is configured

Comments

@posixeleni
Copy link
Contributor

To support importing data from data.gov and exporting in the same format for data.gov to ingest our metadata.

Need to research this:

Here is the metadata schema used in the US Government, which is based on DCAT: https://project-open-data.cio.gov/v1.1/schema/

As you can see, there are a lot of optional fields, but only a few required ones (title, description, keywords, contact, URL). We recently moved to the 1.1. of the schema.

Also note that Data.gov ingest local, state, and university data

http://catalog.data.gov/dataset?organization_type=Federal+Government

http://catalog.data.gov/dataset?organization_type=University

Regardless of what we do, we should make sure Data.gov harvests/ingests us.

Some additional links:

https://project-open-data.cio.gov/

https://github.com/project-open-data/project-open-data.github.io

And here is how Data.gov does the harvesting:

http://www.data.gov/developers/harvesting

@posixeleni posixeleni self-assigned this Mar 5, 2015
@posixeleni posixeleni added this to the 4.0.4 milestone Mar 5, 2015
@scolapasta scolapasta modified the milestones: 4.0.4, In Review - Short Term Apr 18, 2015
@scolapasta scolapasta removed this from the Not Assigned to a Release milestone Jan 28, 2016
@posixeleni
Copy link
Contributor Author

Here is a tool we can look at when working on support for DCAT/RDF http://rdforms.com/editors/dcat/

@pdurbin
Copy link
Member

pdurbin commented Oct 22, 2016

I just added the "harvesting" label to this issue but http://www.data.gov/developers/harvesting doesn't mention OAI-PHM (which is now supported as of Dataverse 4.5) so maybe this is a different type of harvesting.

@pdurbin pdurbin added User Role: Superuser Has access to the superuser dashboard and cares about how the system is configured and removed zTriaged labels Jun 30, 2017
@tlchristian
Copy link

tlchristian commented Jul 20, 2021

ICSPR just announced that it's retiring OAI-PMH harvesting for their repository, and is "exploring an API-focused solution that will involve delivering metadata using the DCAT-US schema." Has Dataverse considered DCAT-US for metadata harvesting? Please say "yes."

https://resources.data.gov/resources/dcat-us/

@pdurbin
Copy link
Member

pdurbin commented Oct 4, 2022

@tlchristian not that I'm aware of. Would you be able to create a fresh issue so we can close this one?

@philippconzett
Copy link
Contributor

ICSPR just announced that it's retiring OAI-PMH harvesting for their repository, and is "exploring an API-focused solution that will involve delivering metadata using the DCAT-US schema." Has Dataverse considered DCAT-US for metadata harvesting? Please say "yes."

https://resources.data.gov/resources/dcat-us/

I just had a look at ICPSR, and it seems they still support OAI-PMH: https://www.icpsr.umich.edu/web/ICPSR/cms/3965.

@DS-INRAE
Copy link
Member

Should this issue title be updated with a more recent version of DCAT than 1.1 as version 3 is already in review https://www.w3.org/TR/vocab-dcat-3/ ?

@pdurbin pdurbin changed the title Support for Importing and Exporting DCAT v.1.1 Metadata Support for Importing and Exporting DCAT Metadata Jul 17, 2024
@pdurbin
Copy link
Member

pdurbin commented Jul 17, 2024

@DS-INRA I removed "v1.1" from the title. I hope that helps!

@cmbz
Copy link

cmbz commented Aug 19, 2024

2024/08/19: @sbarbosadataverse will post announcement to Community to ask if this support is still needed.


Update. Done: https://groups.google.com/g/dataverse-community/c/KP9DPlOk3Po/m/wvAB1hIHBAAJ

@philippconzett
Copy link
Contributor

Hi all,

DataverseNO is interested in Dataverse support for DCAT.

According to Wikipedia, "DCAT is the foundation for open dataset descriptions in the European Union public sector and was adapted by the ISA programme of the European Commission". It seems many data portals, especially in Europe, are based on DCAT. For example, Norwegian public data are collected in a data portal based on a Norwegian DCAT profile, DCAT-AP-NO, which is based on the European Commission DCAT profile.

In Norway (and I guess in other countries as well), research data produced by (public) universities should be made findable in public data portals. That's why DCAT support in Dataverse is important to us.

@CeesH
Copy link

CeesH commented Aug 28, 2024

Hi All, Thanks to Philipp for pointing me to this issue.

DCAT is a big thing in The Netherlands. Just as in Norway, a derived version from DCAT-AP-EU is becoming the national standaard for governmental (meta)data. A public consultation on this new version of DCAT (to be precise the Dutch profile of the European DCAT-AP-3.0 standaard) was finished in May. For the Dataverse based DANS Data Stations, especially DCAT related to Health data and DCAT related to Geospatial data, are top-priority, in order to create the connection to services like the European Health Data Space (EHDS), and our national Health data and Geo data catalogues.

So count us in on these Dataverse DCAT developments.... we have to do this anyway, and are even gathering resources at the moment to start the developments.

Cees

@philippconzett
Copy link
Contributor

Good to hear, @CeesH! I just earlier today was informed that there is a public consultation on the new version of DCAT running also in Norway. Also, earlier this year, an Official Norwegian Report (NOU) was issued on sharing and reuse of public data. They suggest the introduction of a new national law on data sharing. Among other things, the proposal suggests a) using DCAT as the metadata standard for public data (cf. section 11.5.2); and b) that publicly funded research data published in (institutional) repositories need to comply with the proposed data sharing law (cf. section 5.2). For DataverseNO, this implies that we at some point should be able to support the description of dataset based on DCAT.

@CeesH
Copy link

CeesH commented Aug 28, 2024

@philippconzett yes.... there is no escape. In NL, most DCAT developments seem to come from/start at the geospatial community. That is why we start our investigations with the GeoDCAT developments. In the Health sciences, this is also a development not to ignore: https://doi.org/10.1093/eurpub/ckad160.037

@CeesH
Copy link

CeesH commented Nov 5, 2024

Two important initiatives in Europe, that is the European Health Data Space (EHDS) and the so called Health Data Access Bodies (HDABs), are based on importing DCAT metadata. In the Netherlands, our national Health Data Catalogue will also be established around DCAT. In other words, quite urgent to be able to export metadata from a Dataverse following DCAT. Would this be something for EU Dataverse users to collaborate on? Something to discuss at the next Dataverse Community Meeting 2025 maybe?
@philippconzett , @DieuwertjeBloemen, @DS-INRAE

@DieuwertjeBloemen
Copy link
Contributor

I didn't know EHDS would use DCAT as its standard for imports. Good to know! And we'd definitely like to explore how we can work on this as well with the growing body of health data in our Dataverse.

@sjaakd
Copy link

sjaakd commented Jan 13, 2025

DCAT is a big thing in The Netherlands. Just as in Norway, a derived version from DCAT-AP-EU is becoming the national standaard for governmental (meta)data. A public consultation on this new version of DCAT (to be precise the Dutch profile of the European DCAT-AP-3.0 standaard) was finished in May

@CeesH : we (Geological Survey of the Netherlands) are currently investigating whether Dataverse is a viable solution for publishing datasets (internally and externally). Soon are going into a proof of concept phase to get a better feeling on what Dataverse offers. One of the major requirements is indeed the Dutch profile that you mentioned above. Depending on this proof-of-concept I would be interested in what it would take to get support for DCAT in Dataverse.

I'm not an expert on metadata. It seems there is relation between different classes in the DCAT profile. The Dataset does not even seem to be the entrance point (Catalog has that role). Is this all relevant to a Dataverse deployment and, if so, does Dataverse support such structures?

Alternatively: if it is "only" the Dataset that is relevant, my assumption is that defining TSV / property files as described in the Dataverse documentation could be sufficient. Or, should Database have an integration point with a different product like geonetwork where the Dataset is somehow imported?

@pdurbin
Copy link
Member

pdurbin commented Jan 13, 2025

It seems there is relation between different classes in the DCAT profile. The Dataset does not even seem to be the entrance point (Catalog has that role).

@sjaakd hi! As you investigate Dataverse, please let us know if you have any questions!

Here's a screenshot from that DCAT profile link you shared:

overzicht klassen

I know almost nothing about DCAT but at https://www.w3.org/TR/vocab-dcat-3/#dcat-scope it looks like dcat:Catalog is defined like this:

"dcat:Catalog represents a catalog, which is a dataset in which each individual item is a metadata record describing some resource; the scope of dcat:Catalog is collections of metadata about datasets, data services, or other resource types."

In Dataverse, a collection of metadata about datasets sounds like a collection to me. That is, in Dataverse, a dataset lives inside a collection along side other datasets.

Dataverse's OAI_ORE dataset metadata exporter does include information about the parent collection. For example, the dataset at https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/UXIBNO lives inside the Dutch Parliamentary Behaviour Dataset Dataverse collection which can be seen under "isPartOf":

Screenshot 2025-01-13 at 10 09 15 AM

Screenshot 2025-01-13 at 10 09 33 AM

A good place to start might be to create an external exporter for DCAT. This can be done outside the Dataverse code base as a small, standalone plugin as described at https://guides.dataverse.org/en/6.5/installation/advanced.html#external-metadata-exporters

Here are two examples of external exporters:

(The OAI_ORE exporter is internal and the code can be found here.)

Sure, additional fields could be added to metadata blocks but it might be interesting to create an exporter with the existing fields to see which fields are not yet available.

I'm not sure what to say about geonetwork. It looks interesting but I've never heard of it.

@sjaakd
Copy link

sjaakd commented Jan 13, 2025

@pdurbin : thanks for your comments.

This quarter (Q1) my colleagues have planned to install Dataverse so we can play a bit more with the product and see how it integrates in our environment. I'll get back to this issue as soon as we've booked some progress. It would be nice to see if the other classes can be mapped as well. I can also imagine that perhaps they have fixed / constant values in a Dataverse scenario ( DCAT seems geared towards OGC or REST data services).

If we build a DCAT exporter, I guess we could contribute this to this project. Would that be an interesting proposition?

@pdurbin
Copy link
Member

pdurbin commented Jan 13, 2025

@sjaakd yes, absolutely! Say the word and we'll create an empty repo under https://github.com/gdcc for you to push to. We can even automate the publishing of the artifact (a Java jar file) to Maven Central for you. And we'd add it to https://github.com/gdcc/dataverse-exporters#list-of-known-exporters so others can find it.

@CeesH
Copy link

CeesH commented Jan 13, 2025

Hi Phil, hi Sjaak,

Since my message(s) on this topic, we actually went on at DANS to probe the different ways to have Dataverse interact with DCAT based platforms and harvesters, and I'll ask our technical team through (andrecastro0o) to provide an update.

Sorry to bother Dataversers with inland Dutch matters.....
@sjaakd great to share your thoughts and efforts with the larger Dataverse community. It might be an option to (also) get in touch with us at DANS directly. We are collaborating with the National Institute for Public Health and the Environment (RIVM), who just started to use Dataverse, basically as a metadata catalogue. So... currently there are two major Dutch Dataverse users (DANS & RIVM) who more or less have to delve into the DCAT export matter because of our involvement in EU and national (health) data developments.
On our wish list is also to figure out what is currenly the status of the GeoDCAT-AP, and your organisation might know more about this?

(And @sjaakd for your BRO/Subsurface services... our Dataverse based archeological datasets might be of interest, but that would be a project on its own.)

Cees

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature: Harvesting Feature: Metadata Type: Feature a feature request User Role: Superuser Has access to the superuser dashboard and cares about how the system is configured
Projects
Status: ⚠️ Needed/Important
Development

No branches or pull requests