New command: logreport

[jpmckinney]

This command would implement the steps in the logs documentation, to report the most relevant lines from the log file. https://kingfisher-collect.readthedocs.io/en/latest/logs.html

We might want to make this a separate package, and extract the ScrapyLogFile class from Kingfisher Archive: https://github.com/open-contracting/kingfisher-archive/blob/master/ocdskingfisherarchive/scrapy_log_file.py

[jpmckinney]

Started at https://github.com/open-contracting/scrapy-log-analyzer

[jpmckinney]

Note: scrapy-log-analyzer's logparser dependency is GPL. Make the command optional as documented at https://ocp-software-handbook.readthedocs.io/en/latest/python/preferences.html#license-compliance

[jpmckinney]

Here's the stub I had started (was in a git stash). It would mostly call the scrapy-log-analyzer package.

from scrapy.commands import ScrapyCommand
from scrapy.exceptions import UsageError
from scrapyloganalyzer import ScrapyLogFile


class LogReport(ScrapyCommand):
    def short_desc(self):
        return "Analyze a crawl's log file to assess the quality of the crawl"

    def syntax(self):
        return '[options] <logfile>'

    def run(self, args, opts):
        if len(args) < 1:
            raise UsageError()
        elif len(args) > 1:
            raise UsageError("Exactly one log file must be provided.")

[jpmckinney]

Another idea from #1048. The advantage is that it can interrupt a crawl, instead of waiting for it to end. Can maybe use the same approach as #1055

A more intensive option is to add a new feature, that checks the rate of 500 errors and cancels the crawl if too high. This should also send a new type of message to Kingfisher Process, to cancel processing.