Add a validation step to the workflow ensuring proper XML records and HTML views

[SvenLieber]

Currently Walder is used to provide XML-based exports per collection and to provide HTML views on collections (pending commit, internal mockup). These access-means are based on queries on our Knowledge Graph and thus rely on certain data qualities.

Constraints can be expressed in SHACL or ShEx and a validation step can be used to verify that the Knowledge Graph fulfills needed quality. Such a validation step could be included in our workflow such that the workflow looks (more or less) like the following when receiving a warc_created message:

• map item-level data of the newly created WARC file
• execute SPARQL CONSTRUCT/INSERT queries to create collection-level-related aggregations based on item-level data
• execute a validation process with use case specific constraints (e.g. based on HTML views or XML record exports)
• In case of invalid data, query the validation report to obtain information about what needs to be fixed