MRG: improve restart by optionally writing batched zipfiles

[bluegenes]

This PR introduces a new optional param --batch-size, which allows users to build smaller zipfiles with gbsketch or urlsketch. These zipfiles are populated sequentially, with all signatures associated with batch_size accessions (not batch_size signatures). If gbsketch/urlsketch fail, they can read any zipfiles that were finished in order to restart. Zip names will be generated from the --output, so if output is output.zip, batches will be output.1.zip, output.2.zip, etc. I'm not really sure what batch_size to recommend, but I think the overhead is fairly low for creating new small zips -- the main issue will be if users later want to concatenate them into a single zip.

Uses the changes from #101 to enable writing batched zipfiles as a way to improve restart.

• make batch_size a user modifiable parameter
• For cases where the total number of signatures is less than the batch_size, we could write the regular *zip file, with no .1, etc.
• functions to enable reading from existing batched zips to allow restart
  • build filename: paramset Hashmap, use that to filter the template sigs for each filename using filter
• add tests for batched zipfile writing, recovery from existing batches
• move zip_writer creation inside writing loop to avoid empty final zip
• check what happens if we have an unclosed zip (i.e. from unexpected failure)
  • sourmash panics on invalid zips. Here I've caught the panic and ignored it, but it may ultimately be better to handle + return error at the sourmash level (ZipStorage::from_file panics)
  • Note that we will likely have an invalid zip upon any restart from failure, because the zip file would not have properly been closed/finished.

Issue for later:

• copy batched zips to a single zipfile? #107

Fixes:

• improve recovery after failures #69

[bluegenes]

@ctb ready for review

@ccbaumler - better later than never, right? Does this fit your use cases? Do you have a recommendation for the # of files we should recommend in each batch?

[ctb]

maybe create an issue in branchwater repo? I didn't really design around re-use 😅 - you could also link to sourmash-bio/sourmash#3321.

so, ok, suggest:

• create issue in this repo
• link to sourmash issue consider how to support more flexible Collection in RevIndex for external storage sourmash#3321

[bluegenes]

sure, will do. Note, it doesn't really need to be replaced with MultiCollection - just, if we eventually get MultiCollection in sourmash core, it'd be good to just have one supported implementation.

[ctb]

only skimmed, but looks great to me ;)