Add harvesting of schema.org

[steingod]

Add support for harvesting of schema.org records. Need to decide on whether this is a separate function or to be integrated in the ordinary harvester. There are quite some differences in how the harvesting is done, so could be that a separate harvester should be used.

[steingod]

Harvesting from NSF ADC works, working on adaptations for GEM. Those are the two working examples for the time being.

[ferrighi]

Some parsing issues to fix:

• parse only records of type "Dataset"
• handing of identifier field (e.g. type url or PropertyValue (prefixed or not))
• handle missing title
• description can be text or dict
• temporalCoverage can be of different formats and with different separators (slash or dash)
• datePublished can be of different formats, sometimes only a year
• keywords: can be list of strings, string (comma or colon separated), list of dict (of DefinedTerm), check for empty list
• variableMeasured similar to keywords
• spatialCoverage: should use GeoCoordinate or GeoShape (S W N E) space separated, but it should support also (W,S E,N)
• add module vocab to validate license
• add conditionsOfAccess where "unrestricted" can be mapped to "Open" (access_constraint)
• distribution, encodingFormat is not mandatory
• add some default MMD values and follow MMD sequence

Some technical issues:

• avoid for now using requests-html (it is slow and it installs pyppeteer by default), use requests/json instead. Lazy loading will be handle in a future version
• add the possibility to harvest only new records
• add possibility to harvest sitemapindex and sitemap (PANGAEA)
• add possibility to harvest html documents directly (GEM)

[ferrighi]

I've made a branch https://github.com/steingod/mdharvest/tree/issue16 with the current modifications. It is working for:

• ./harvestschemaorg.py https://doi.pangaea.de/sitemap.xml path-to-dir (all levels)
• ./harvestschemaorg.py https://doi.pangaea.de/sitemap-0.xml.gz path-to-dir (single level)
• ./harvestschemaorg.py https://api.g-e-m.dk/api/dataset/harvest path-to-dir

for harvesting only the records of the last X days the "-l X" can be used. This is if the lastmod is provided.

I will wait for a PR as I want to add a unique function to check datetime a bit more consistently for datePublished, dateModified, temporalCoverage. But the branch can be used for testing.