desi_submit_prod rechecks running jobs hundreds of times

[sbailey]

desi_submit_prod starts with log messages like:

INFO:queue.py:533:get_jobs_in_queue: Querying Slurm with the following: squeue -u desi -o "%i,%P,%j,%u,%t,%M,%D,%R"
INFO:submit_prod.py:257:submit_production: night=20201214 already has a proctable, skipping.
INFO:queue.py:533:get_jobs_in_queue: Querying Slurm with the following: squeue -u desi -o "%i,%P,%j,%u,%t,%M,%D,%R"
INFO:submit_prod.py:257:submit_production: night=20201215 already has a proctable, skipping.
INFO:queue.py:533:get_jobs_in_queue: Querying Slurm with the following: squeue -u desi -o "%i,%P,%j,%u,%t,%M,%D,%R"
INFO:submit_prod.py:257:submit_production: night=20201216 already has a proctable, skipping.
INFO:queue.py:533:get_jobs_in_queue: Querying Slurm with the following: squeue -u desi -o "%i,%P,%j,%u,%t,%M,%D,%R"
INFO:submit_prod.py:257:submit_production: night=20201217 already has a proctable, skipping.
...

I.e. it appears to be invoking squeue to check the currently running jobs for every night that it is skipping while trying to find the next time to submit. This adds up to a lot of slurm load and delayed start of actual submission while trying to find a night that needs to be submitted.

Check if this squeue check could be run once, skip over N nights already submitted, and then proceed.

[sbailey]

fixed by PR #2348; closing.