noarchive detection?

[matteocargnelutti]

The first versions of Scoop shipped with partial support for the detection of the noarchive directive, inspecting HTTP responses to find <meta> tags containing noarchive, and listing "positive" URLs both in the logs and in the provenance summary.

This implementation was only providing partial support, as the noarchive directive can be expressed in different ways (meta tags, HTTP headers, robots.txt file ...), most of which likely fall outside of the scope of Scoop.

This feature was removed in Scoop 0.5.0.

Dear reader, if you believe Scoop should have a dedicated noarchive detection feature:

• Let us know 😄
• Please feel free to open a pull request with a suggested implementation for this feature