OOM issues when calculating memory for bwa_mem2 jobs

[cat-bro]

GA has been using a rule borrowed from Galaxy Main for calculating memory for bwa_mem2 jobs but it has been removed because some jobs run out of memory.

    - id: bwa_mem2_history_reference_rule
      # This rule is copied from usegalaxy.org with bwa_mem2_max_mem increased from 120 to 250
      # See https://github.com/galaxyproject/usegalaxy-playbook/blob/5be90699d922c85228836a5a974b05cfd08a6c4b/env/common/templates/galaxy/config/tpv/tools_vgp.yaml.j2#L217C1-L224C93
      if: |
        helpers.job_args_match(job, app, {"reference_source": {"reference_source_selector": "history"}})
      # per https://github.com/bwa-mem2/bwa-mem2/issues/41 it's 28 * reference
      mem: |
        bwa_mem2_max_mem = 250
        options = job.get_param_values(app)
        size = options["reference_source"]["ref_file"].get_size()
        min(max(float(size/1024**3) * 28, (input_size - float(size/1024**3)) * 2, 7.6), bwa_mem2_max_mem)

oom job 1:

$ glq job-input-datasets <job-id>
  hda_id  |        hda_name        |   d_id   | d_state | d_file_size | d_total_size
----------+------------------------+----------+---------+-------------+--------------
 12345678 | abc.fa uncompressed    | 23456789 | ok      | 3042 MB     | 3042 MB
 12345679 | SRR_1.fastq.gz         | 23456790 | ok      | 24 GB       | 24 GB
 12345680 | SRR_2.fastq.gz         | 23456791 | ok      | 25 GB       | 25 GB

mem value assigned: 96.60GB

oom job 2:

$ glq job-input-datasets <job-id>
  hda_id  |       hda_name        |   d_id   | d_state | d_file_size | d_total_size
----------+-----------------------+----------+---------+-------------+--------------
 12345681 | bionano assembly      | 23456792 | ok      | 12 MB       | 12 MB
 12345682 | SRR_1.fastq           | 23456793 | ok      | 9363 MB     | 9363 MB

mem value assigned: 18.29GB