Pipeline Action support for multiple index pages

[saumier]

@dev-aravind Please design an enhancement to the Artsdata Pipeline Action that would allow multiple page-url and entity-identifier to be crawled for the same artifact.

The test case can be with theplayhouse-ca which has 2 sitemaps: https://theplayhouse.ca/fr/sitemap.xml and https://theplayhouse.ca/en/sitemap.xml. The idea is to load all pages from both sitemaps into the artifact orion/theplayhouse-ca

[dev-aravind]

@saumier I propose this design. Let me know what you think.

with:
  mode: "fetch-push"
  artifact: "artifact123"
  publisher: "${{ secrets.PUBLISHER_URI_GREGORY }}"
  downloadFile: "file.jsonld"
  page-url: '["www.example.com/events", "www.example.com/evenements]'
  entity-identifier: '["identifier1", "identifier2"]'
  token: "${{ secrets.GITHUB_TOKEN }}"

[saumier]

@dev-aravind I have a question. Do you see one entity-identifier applied per page-url?

In my use case:
page-url: ["https://theplayhouse.ca/fr/sitemap.xml", "https://theplayhouse.ca/en/sitemap.xml"]
entity-identifier: "loc"

I am wondering if we need to have a list of entity-identifier or not.

Perhaps the entity-identifier could apply to all page-urls and when we come across a need for multiple entity-identifiers then we add a check if entity-identifier is a list and if true we apply one entity-identifier per page-url?

But I like the consistency of your solution. So let's go ahead with one entity-identifier applied per page-url as you propose.

Please go ahead to implementation of a minor version as long as it remains backwards compatible.

[dev-aravind]

@saumier You can test this now using our custom crawl test workflow(use enhancement/issue-12 branch). Please let me know if you find any issues, if not we can move on to release this.

If you want to test the multiple page crawl, the input should be like:

page-url: "https://theplayhouse.ca/fr/sitemap.xml,https://theplayhouse.ca/en/sitemap.xml"
entity-identifier: 'loc' 

both without the quotes.

[saumier]

@dev-aravind I ran a test with theplayhouse.ca and it collected all the pages. Excellent. Please go ahead and release.

[dev-aravind]

@saumier Done, let me know if you want to add any workflows in orion to use this feature.

Release notes