DEV-1203: Additional ingest performance monitoring (collate/storage)

[mwarin]

This PR does the following changes to JobMetrics:

• Changed metric naming conventions
• Added labels
• Expanded coverage
• TIDYing of some prominent classes
• Split new class into JobMetrics and JobMetricsCLI, to simplify main class and not pollute it with CLI stuff.

In most cases where monitoring is added, I've added some variation on:

my $start_time = $self->{job_metrics}->time;
# ...
# ... the existing piece of code doing the work that we want to measure
# ...
my $end_time   = $self->{job_metrics}->time;
my $delta_time = $end_time - $start_time;
my $labels = {foo => 'bar'}; # something about the operation being measured
$self->{job_metrics}->add("ingest_xxx_bytes_r_total", -s $infile, $labels); # if we did any r/w op
$self->{job_metrics}->add("ingest_xxx_bytes_w_total", -s $outfile, $labels); # if we did any r/w op
$self->{job_metrics}->add("ingest_xxx_seconds_total", $delta_time, $labels); # the time it took
$self->{job_metrics}->inc("ingest_xxx_items_total", $labels); # if we did anything with an item
$self->{job_metrics}->inc("ingest_xxx_images_total", $labels); # if we did anything with an image

... plus a matching metric declaration in HTFeed::JobMetrics.

I've also added 2 convenience subs to HTFeed::JobMetrics:

• time for providing high resolution time
• dir_size for calculating the size of a directory

... which are both data points that are often needed for measuring.

Reviewer info

I originally tried to separate and label commits thematically, but this started falling apart as the PR grew too large. There are some TIDY commits, a big FUNC commit with mostly additions (that nevertheless has some tidyings in it, discipline slipped), and a TEST commit. I will squash them all before merging.

Please check:

• the new tests in t/job_metrics.t
• the changes to JobMetrics::_setup_metrics
• that the TIDY commits aren't going too far
• that I'm measuring what I say I'm measuring (i.e. not using the wrong metric in the wrong place)

Please be on the lookout for any log warnings about WARN: invalid metric name "ingest_xxx_yyy", this is a sign that we are trying to record data for a metric that has not been declared in HTFeed::JobMetrics.

[moseshll]

See two comments, one on naming conventions and the other on tabs. Don't feel like making either an impediment to merging, though I think the naming convention is worth further thought/brainstorming.

[aelkiss]

@mwarin Have we looked at perltidy https://metacpan.org/pod/Perl::Tidy recently? It might be able to do some of these tidyings/reformattings automatically, although it looks like it has quite a few options...

[aelkiss]

Overall I think this is looking like a good approach. There are a couple things I'd like to see:

• labels for the storage name
• labels for the specific image remediate operation
• consider extracting the metrics recording stuff to a method to reduce duplication

[aelkiss]

This is looking in pretty good shape; I don't see anything I think needs to block merge, although there are a couple small things to potentially consider now and a few things I think we should note as issues for future work.

I did want to verify that although this PR adds support for labels on metrics, there aren't yet places we're using those labels -- correct?