Port GrandTheatreQuebec Huginn to Ruby

[saumier]

The GrandTheatreQuebec already has a Planet. This is to remove the crawling still happening on Huginn. The workflow in Huginn has an extra step when crawling each page, that is to scrape the html for the keywords of each event page. The keywords is missing from the JSON-LD and is added to JSON-LD by the workflow and then mapped to the GrandTheatreQuebec event type SKOS.

If needed, I can give you access to Huginn

So I propose working in steps (each step can be loaded into Artsdata for review)

• normal crawl using Orion to get JSON-LD of each webpage into Artsdata
• custom scrape to extract keywords from event webpages
• mapping keywords to GrandTheatreQuebec event type SKOS
• add specific SPARQL transforms (review Huginn's list of SPARQLs with Gregory, some may not be needed)

[saumier]

@dev-aravind Only a couple of Huginn scenarios left to migrate ;-)

[dev-aravind]

@saumier will add the huginn crawling details here.

[saumier]

@dev-aravind Here is the agent from Huginn. Instead of a CSS class it uses "xpath": "//article[@class=\"show\"]//a" to get the list of @href for the events.

{
  "expected_update_period_in_days": "100",
  "url": [
    "https://grandtheatre.qc.ca/programmation/"
  ],
  "type": "html",
  "mode": "all",
  "extract": {
    "url": {
      "xpath": "//article[@class=\"show\"]//a",
      "value": "concat(\"https://grandtheatre.qc.ca\",@href)"
    }
  },
  "template": {
    "graph_name": "{{graph_name}}"
  }
}

[dev-aravind]

@saumier The reason why the workflow was stalled is because the replace blank nodes SPARQL is taking more time than expected to execute. The initial output of the crawler before the transformation is a 400,000 line JSON-LD file with 16,000+ xhv:role entities. These roles are blank and top-level nodes, hence the SPARQL will try to assign a temporary URI for them. What do you suggest?

[dev-aravind]

Task for @dev-aravind - remove the vocab role types and reorder the SPARQL run to make the replace blank nodes to run last.

[saumier]

@dev-aravind Here is the list of the SPARQLs created for GTQ. Please look at each one and decide if they can be added to our pipeline for all crawls. Some will already be covered in our Github Artsdata Pipeline Action (like specific/lavitrine/fix-schemaorg-date-datatype), some will need to be added (like specific/lavitrine/fix-isni).

The question to ask: Can this SPARQL apply to all data feeds and improve the data quality for everyone? If yes then it should be added to the Action for everyone.

This is an important step and I will want to review each individual SPARQL in the list to check that we are using the best approach for maintainability. We need to make it as easy as we can for someone else to understand what each SPARQL does.

The paths are relative to this folder https://github.com/culturecreates/sparql-library/tree/master/artsdata/ETL/huginn

• 1. specific/lavitrine/fix-schemaorg-https-objects
• 2. specific/lavitrine/fix-wikidata-uri
• 3. specific/lavitrine/add-artsdata-uri-using-wikidata-bridge
• 4. specific/lavitrine/fix-schemaorg-date-datatype
• 5. specific/lavitrine/create-eventseries
• 6. specific/lavitrine/copy-subevent-data-to-eventseries
• 7. specific/lavitrine/fix-isni
• 8. specific/lavitrine/add-artsdata-uri-using-isni-bridge
• 9. specific/lavitrine/collapse_duplicate_contact_point_blanknodes
• 10. specific/lavitrine/add-keywords-additional-type-mapping
• 11. specific/lavitrine/fix-offer-availability

Things to consider in SPARQL

• several SPARQLs from the Huginn pipeline use graph <graph_name_placeholder> {... } which should be removed from the SPARQLs added to the artsdata pipeline action. Unless the graph is inside a federated part using an external end point (like service <http://db.artsdata.ca/repositories/artsdata>)
• the Huginn pipeline runs the SPARQLs in graphdb with inferencing turned on. The Artsdata pipeline does not (yet) using inferencing. Special attention should be given to check that inferencing is not needed. For example: select * {?s a schema:Event} will include all subtypes such as schema:MusicEvent, schema:DanceEvent, etc.
• several SPARQLs use federated queries to get artsdata uris. This SPARQL specific/lavitrine/add-keywords-additional-type-mapping is used to add the mapping of keyword to event type (using additionalType) without having to download the mapping file. However, for maintainability, I think this can be replaced by loading the event type mapping file gtq-event-type-mapping.ttl into the graph data uploaded to Artsdata so everything is in a single data feed. This way the workflow will not depend on the mapping file being already loaded into Artsdata.

[dev-aravind]

@saumier I added a set of SPARQLs to our pipeline and the PR can be found here. Please approve this if you find everything to be okay.

I was not able to run the add-artsdata-uri-using-wikidata-bridge and add-artsdata-uri-using-isni-bridge because of this error:
SERVICE operator not implemented (NotImplementedError). Have you encountered this previously?

[saumier]

@dev-aravind The error you received NotImplementedError is because the feature called "federated SPARQL" is not implemented.

I will move these 2 SPARQLs to the Artsdata pipeline. https://github.com/culturecreates/artsdata-api/issues/20

I reviewed the PR.

[dev-aravind]

@saumier I updated the tests to run as separate files for each SPARQLS in the artsdata pipeline repository, and updated the GTQ repository to add the workflow to fetch the data, add the concepts and push the data to artsdata. You can find the PR here.