consider adding sourmash gather execution directly to fastgather as postprocessing

[ctb]

could run w picklist via Python API

[ctb]

I looked into this and it's a bit annoying because fastgather really does take different (and more limited) database types than gather... I think it will be easier after #134 is merged.

[bluegenes]

after #134, I was hoping to provide all gather columns in the branchwater fastmultigather results csv by using the rust GatherResult and adding in the query-specific columns within branchwater.

I think that's feasible, but might require a little more work in rust core:

• disk_revindex.rs: need to populate remaining GatherResult cols https://github.com/sourmash-bio/sourmash/blob/latest/src/core/src/index/revindex/disk_revindex.rs#L327-L334

• linear.rs: need to populate remaining GatherResult cols https://github.com/sourmash-bio/sourmash/blob/ccb1d491807767ab9d3c9b005ff21ac953c34883/src/core/src/index/linear.rs#L186-L193

• mem_revindex.rs: is this actually using GatherResult? https://github.com/sourmash-bio/sourmash/blob/latest/src/core/src/index/revindex/mem_revindex.rs#L201-L233

Ref:

• rust GatherResult: https://github.com/sourmash-bio/sourmash/blob/ccb1d491807767ab9d3c9b005ff21ac953c34883/src/core/src/index/mod.rs#L27-L59

• python GatherResult (all columns needed for output file parity). Tthe query-specific ones could/should remain outside of the rust GatherResult, since they're just extra info that help us later, once we no longer have access to the query sketch. https://github.com/sourmash-bio/sourmash/blob/ccb1d491807767ab9d3c9b005ff21ac953c34883/src/sourmash/search.py#L428

[bluegenes]

.. hmm, this actually doesn't directly help with fastgather since it's not using the underlying database gather. Sorry! But you might be interested in using GatherResult directly if/when I get that working for fastmultigather.

[ctb]

after #134, I was hoping to provide all gather columns in the branchwater fastmultigather results csv by using the rust GatherResult and adding in the query-specific columns within branchwater.

Two thoughts - I love the details in here about what needs to be done! I think maybe it should be its own issue or set of issues!

But, also, my experience with a performance issue in branchwater, see #71, makes me wary here. I don't want to slow fastgather down by accident. I guess I am of two minds: is the only purpose of fastgather to do a full gather, faster? Or are there situations where we might want to take the output of fastgather and NOT do a gather afterwards?

OK, I feel dumb for even saying it. But anyway, be wary of performance issues, is all.

[bluegenes]

great point!

I think I want the full results often enough that it would be useful to enable. If there's a significant performance hit, it might be worth passing a flag to toggle between lightweight and full versions of gather.

I'll move the steps above over to a new fastmultigather issue :).

[ctb]

soooo looking at #188, I have a hot take:

both fastgather and full gather are annoying and big and slow for some samples - for a collection of rumen, it's been taking ~36 hours per sample with 64 threads and fastgather against GTDB!, and 500 GB of RAM to do the full gather on the resulting ~6000 matches with a picklist.

this feels a bit hacky, but I think there's opportunity for a version of gather that just calculates the statistics without doing the full search and so on. That is, it should be possible to just take the fastgather output and flesh out the full stats by "believing" the fastgather output - not using it as a picklist, and instead using it as an ordered scaffold. It would presumably be much faster and lower memory too...

[ctb]

This is becoming quite the tangle of issues and PRs 😅 but I wanted to connect them a bit more by pointing out we've gone this route with calc-full-gather.

ref PR dib-lab/sourmash-slainte#18 for sourmash-slainte workflow, and sourmash-bio/sourmash#2950 for problems revealed in sourmash.

[ctb]

You know what? I'm going to close this issue. #187 has all the important stuff remaining.