vine: remove items after iterating hash table

[JinZhou5042]

Proposed Changes

For #4081

Merge Checklist

The following items must be completed before PRs can be merged.
Check these off to verify you have completed all steps.

• make test Run local tests prior to pushing.
• make format Format source code to comply with lint policies. Note that some lint errors can only be resolved manually (e.g., Python)
• make lint Run lint on source code prior to pushing.
• Manual Update: Update the manual to reflect user-visible changes.
• Type Labels: Select a github label for the type: bugfix, enhancement, etc.
• Product Labels: Select a github label for the product: TaskVine, Makeflow, etc.
• PR RTM: Mark your PR as ready to merge.

[dthain]

Removing things from a hash table inside of an iteration seems like a common operation that should be supported more directly. Is there a good abstraction that we can provide so that it's easy for the caller to get it right?

[Ian2327]

After running blast.py using this fix, the number of valgrind errors are reduced from 411572 to 244766. The 2 major sources of errors are still the same, those involving either vine_manager_choose_resources_for_task or vine_file_replica_table_remove

[btovar]

Removing things from a hash table inside of an iteration seems like a common operation that should be supported more directly. Is there a good abstraction that we can provide so that it's easy for the caller to get it right?

We could add a delayed delete and then a commit deletes. I'm not sure we can do much better than what @JinZhou5042 did here, though.