feat(ena-submission): Create ena projects

[anna-parker]

resolves #2399

preview URL: https://create-ena-projects.loculus.org/

Summary

This adds the following rule to the ena-submission snakemake file :

• project_creation rule . This function will continuously (in a loop) scan for new sequences where a project needs to be created and trigger their creation. It will also update both the submission_table and the project_table.
• It also adds some basic unit tests for sub-functions used by create_projects.

High level overview of project_creation:

In a loop:

1. Get sequences in submission_table in state READY_TO_SUBMIT

• if (there exists an entry in the project_table for the corresponding (group_id, organism)):
  -- if (entry is in status SUBMITTED): update submission_table to SUBMITTED_PROJECT.
  -- else: update submission_table to SUBMITTING_PROJECT.
• else: create project entry in project_table for (group_id, organism).

2. Get sequences in submission_table in state SUBMITTING_PROJECT

• if (corresponding project_table entry is in state SUBMITTED): update entries to state SUBMITTED_PROJECT.

3. Get sequences in project_table in state READY, prepare submission object, set status to SUBMITTING

• if (submission succeeds): set status to SUBMITTED and fill in results: the result of a successful submission is a "bioproject_accession" and an ena-internal "ena_submission_accession".
• else: set status to HAS_ERRORS and fill in errors

4. Get sequences in project_table in state HAS_ERRORS for over 15min and sequences in status SUBMITTING for over 15min: #TODO (handle failure ena-submission: Recover from failed project/sample/assembly submission #2311), currently just throw an error

ENA Project

This PR will create projects in ENA with the following attributes:

<PROJECT_SET>
    <PROJECT center_name="{{ Institution }}" alias="{{ group_id }}:{{ organism }}:{{ unique_id }}">
        <NAME>{{ scientific_name }}</NAME>
        <TITLE>{{ scientific_name }} Genome sequencing</TITLE>
        <DESCRIPTION>Automated upload of {{ scientific_name }} sequences submitted by {{ Institution }} from {{ db }}.</DESCRIPTION>
        <SUBMISSION_PROJECT>
            <SEQUENCING_PROJECT/>
            <ORGANISM>
            <TAXON_ID>{{ taxon_id }}</TAXON_ID>
            <SCIENTIFIC_NAME>{{ scientific_name }}</SCIENTIFIC_NAME>
            </ORGANISM>
        </SUBMISSION_PROJECT>
        <PROJECT_LINKS>
            <PROJECT_LINK>
                <XREF_LINK>
                <DB>{{ db }}</DB>
                <ID>{{ group_accession }}</ID>
                </XREF_LINK>
            </PROJECT_LINK>
        </PROJECT_LINKS>
    </PROJECT>
</PROJECT_SET>

Screenshot

PR Checklist

• Send a message to our slack channel when submission fails in unhandled cases: https://github.com/GenSpectrum/servers/blob/1560a0fc1dfc0f6dcbf1be7742e2a1913babbaef/ansibleSetup/roles/gisaid_ingest/templates/gisaid-change-detector/check_source.py#L24
• Check project creation on ENA dev instance works locally
  Confirmed locally this will resend messages to our slack channel every 15minutes
• Add username and password for our ENA test account into kubernetes secrets: use https://github.com/bitnami-labs/sealed-secrets
• confirm center name is set on broker account: this works!
  
• Test upload to ENA dev instance works from preview instance (use port forwarding to check the database): upload triggers message:

it gets uploaded to the submission_table and then the project_table: when it succeeds: (also the submission_table is updated) when not successful:

[theosanderson]

For final PR it would be nice to do this in the values_preview_server.yaml rather than in the default Helm chart - until ENA submission is fully mature

[theosanderson]

(and actually even after that, because many Loculus instances wouldn't use this)

[anna-parker]

added in #2831

[corneliusroemer]

Drive-by edits in this PR - might be good to merge these frequently to prevent merge conflicts from getting out of hand with the many branches open: #2663

[anna-parker]

Closing as this was part of #2417