use-cases.md

Batch Queue Use Cases

• Design

Use Case: Successful Batch

User submits manifest with 3 items

graph TD
  Manifest[/Batch Manifest/]
  Ingest(Ingest Service)
  Batch[Batch: Pending]
  Manifest --> Ingest
  Ingest --> Batch

Loading

Batch Queue starts Batch

graph TD
  Batch[Batch: Processing]

Loading

Batch downloads manifest and creates 3 jobs

graph TD
  Batch[Batch: Processing]
  Job1[Job 1: Pending]
  Job2[Job 2: Pending]
  Job3[Job 3: Pending]
  Batch --> |job1_payload_url| Job1
  Batch --> |job2_payload_url| Job2
  Batch --> |job3_payload_url| Job3

Loading

Jobs Begin

graph TD
  Batch[Batch: Processing]
  Job1[Job 1: Processing]
  Job2[Job 2: Processing]
  Job3[Job 3: Processing]
  Batch --- Job1
  Batch --- Job2
  Batch --- Job3

Loading

Job 2 Completes

graph TD
  Batch[Batch: Processing]
  Job1[Job 1: Processing]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Processing]
  Batch --- Job1
  Batch --- |notify| Job2
  Batch --- Job3

Loading

Job 3 completes

graph TD
  Batch[Batch: Processing]
  Job1[Job 1: Processing]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: COMPLETE]
  Batch --- Job1
  Batch --- Job2
  Batch --- |notify| Job3

Loading

Job 1 completes

graph TD
  Batch[Batch: Reporting]
  Job1[Job 1: COMPLETE]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: COMPLETE]
  Batch --- |notify| Job1
  Batch --- Job2
  Batch --- Job3

Loading

Batch Reports Job Status to Depositor

graph TD
  Batch[Batch: Reporting]
  Job1[Job 1: COMPLETE]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: COMPLETE]
  Batch --- Job1
  Batch --- Job2
  Batch --- Job3
  StatusReport[/StatusReport/]
  Email(Email Status to Depositor)
  Batch --> StatusReport
  StatusReport --> Email

Loading

Batch Completes

graph TD
  Batch[Batch: COMPLETED]

Loading

Use Case: Failed Batch

User submits manifest with 3 items

graph TD
  Manifest[/Batch Manifest/]
  Ingest(Ingest Service)
  Batch[Batch: Pending]
  Manifest --> Ingest
  Ingest --> Batch

Loading

Batch Queue starts Batch

graph TD
  Batch[Batch: Processing]

Loading

Batch downloads manifest and creates 3 jobs

graph TD
  Batch[Batch: Processing]
  Job1[Job 1: Pending]
  Job2[Job 2: Pending]
  Job3[Job 3: Pending]
  Batch --> |job1_payload_url| Job1
  Batch --> |job2_payload_url| Job2
  Batch --> |job3_payload_url| Job3

Loading

Jobs Begin

graph TD
  Batch[Batch: Processing]
  Job1[Job 1: Processing]
  Job2[Job 2: Processing]
  Job3[Job 3: Processing]
  Batch --- Job1
  Batch --- Job2
  Batch --- Job3

Loading

Job 2 Completes

graph TD
  Batch[Batch: Processing]
  Job1[Job 1: Processing]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Processing]
  Batch --- Job1
  Batch --- |notify| Job2
  Batch --- Job3

Loading

Job 3 fails

graph TD
  Batch[Batch: Processing]
  Job1[Job 1: Processing]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Failed]
  Batch --- Job1
  Batch --- Job2
  Batch --- |notify| Job3

Loading

Job 1 completes

graph TD
  Batch[Batch: Reporting]
  Job1[Job 1: COMPLETE]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Failed]
  Batch --- |notify| Job1
  Batch --- Job2
  Batch --- Job3

Loading

Batch Reports Job Status to Depositor

graph TD
  Batch[Batch: Reporting]
  Job1[Job 1: COMPLETE]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Failed]
  Batch --- Job1
  Batch --- Job2
  Batch --- Job3
  StatusReport[/StatusReport/]
  Email(Email Status to Depositor)
  Batch --> StatusReport
  StatusReport --> Email

Loading

Batch Goes to Failed state

graph TD
  Batch[Batch: Failed]
  Job1[Job 1: COMPLETE]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Failed]
  Batch --- Job1
  Batch --- Job2
  Batch --- Job3
  StatusReport[/StatusReport/]
  Batch --- StatusReport

Loading

Admin Deletes Batch after determining that resubmission of failed job is not possible

graph TD
  Batch[Batch: DELETED]

Loading

Use Case: Failed Batch with Successful Retry

User submits manifest with 3 items

graph TD
  Manifest[/Batch Manifest/]
  Ingest(Ingest Service)
  Batch[Batch: Pending]
  Manifest --> Ingest
  Ingest --> Batch

Loading

Batch Queue starts Batch

graph TD
  Batch[Batch: Processing]

Loading

Batch downloads manifest and creates 3 jobs

graph TD
  Batch[Batch: Processing]
  Job1[Job 1: Pending]
  Job2[Job 2: Pending]
  Job3[Job 3: Pending]
  Batch --> |job1_payload_url| Job1
  Batch --> |job2_payload_url| Job2
  Batch --> |job3_payload_url| Job3

Loading

Jobs Begin

graph TD
  Batch[Batch: Processing]
  Job1[Job 1: Processing]
  Job2[Job 2: Processing]
  Job3[Job 3: Processing]
  Batch --- Job1
  Batch --- Job2
  Batch --- Job3

Loading

Job 2 Completes

graph TD
  Batch[Batch: Processing]
  Job1[Job 1: Processing]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Processing]
  Batch --- Job1
  Batch --- |notify| Job2
  Batch --- Job3

Loading

Job 3 fails

graph TD
  Batch[Batch: Processing]
  Job1[Job 1: Processing]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Failed]
  Batch --- Job1
  Batch --- Job2
  Batch --- |notify| Job3

Loading

Job 1 completes

graph TD
  Batch[Batch: Reporting]
  Job1[Job 1: COMPLETE]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Failed]
  Batch --- |notify| Job1
  Batch --- Job2
  Batch --- Job3

Loading

Batch Reports Job Status to Depositor

graph TD
  Batch[Batch: Reporting]
  Job1[Job 1: COMPLETE]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Failed]
  Batch --- Job1
  Batch --- Job2
  Batch --- Job3
  StatusReport[/StatusReport/]
  Email(Email Status to Depositor)
  Batch --> StatusReport
  StatusReport --> Email

Loading

Batch Goes to Failed state

graph TD
  Batch[Batch: Failed]
  Job1[Job 1: COMPLETE]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Failed]
  Batch --- Job1
  Batch --- Job2
  Batch --- Job3
  StatusReport[/StatusReport/]
  Batch --- StatusReport

Loading

Job 3 is restarted

graph TD
  Batch[Batch: Failed]
  Job1[Job 1: COMPLETE]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Processing]
  Batch --- Job1
  Batch --- Job2
  Batch --- Job3
  StatusReport[/StatusReport/]
  Batch --- StatusReport

Loading

Job 3 completes

graph TD
  Batch[Batch: Failed]
  Job1[Job 1: COMPLETE]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: COMPLETE]
  Batch --- Job1
  Batch --- Job2
  Batch --- |notify| Job3
  StatusReport[/StatusReport/]
  Batch --- StatusReport

Loading

Admin changes batch state to UpdateReporting

graph TD
  Batch[Batch: UpdateReporting]
  Job1[Job 1: COMPLETE]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: COMPLETE]
  Batch --- Job1
  Batch --- Job2
  Batch --- Job3
  StatusReport[/StatusReport/]
  Batch --- StatusReport

Loading

Email is sent to depositor showing the status change for Job 3

graph TD
  Batch[Batch: UpdateReporting]
  Job1[Job 1: COMPLETE]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: COMPLETE]
  Batch --- Job1
  Batch --- Job2
  Batch --- Job3
  StatusReport[/StatusReport/]
  Batch --> StatusReport
  StatusReport -.-> Batch
  Email(Email Status to Depositor)
  StatusReport --> Email

Loading

Batch Status is COMPLETE

graph TD
  Batch[Batch: COMPLETE]

Loading

Use Case: Failed Batch with Unsuccessful Retry

User submits manifest with 3 items

graph TD
  Manifest[/Batch Manifest/]
  Ingest(Ingest Service)
  Batch[Batch: Pending]
  Manifest --> Ingest
  Ingest --> Batch

Loading

Batch Queue starts Batch

graph TD
  Batch[Batch: Processing]

Loading

Batch downloads manifest and creates 3 jobs

graph TD
  Batch[Batch: Processing]
  Job1[Job 1: Pending]
  Job2[Job 2: Pending]
  Job3[Job 3: Pending]
  Batch --> |job1_payload_url| Job1
  Batch --> |job2_payload_url| Job2
  Batch --> |job3_payload_url| Job3

Loading

Jobs Begin

graph TD
  Batch[Batch: Processing]
  Job1[Job 1: Processing]
  Job2[Job 2: Processing]
  Job3[Job 3: Processing]
  Batch --- Job1
  Batch --- Job2
  Batch --- Job3

Loading

Job 2 Completes

graph TD
  Batch[Batch: Processing]
  Job1[Job 1: Processing]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Processing]
  Batch --- Job1
  Batch --- |notify| Job2
  Batch --- Job3

Loading

Job 3 fails

graph TD
  Batch[Batch: Processing]
  Job1[Job 1: Processing]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Failed]
  Batch --- Job1
  Batch --- Job2
  Batch --- |notify| Job3

Loading

Job 1 completes

graph TD
  Batch[Batch: Reporting]
  Job1[Job 1: COMPLETE]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Failed]
  Batch --- |notify| Job1
  Batch --- Job2
  Batch --- Job3

Loading

Batch Reports Job Status to Depositor

graph TD
  Batch[Batch: Reporting]
  Job1[Job 1: COMPLETE]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Failed]
  Batch --- Job1
  Batch --- Job2
  Batch --- Job3
  StatusReport[/StatusReport/]
  Email(Email Status to Depositor)
  Batch --> StatusReport
  StatusReport --> Email

Loading

Batch Goes to Failed state

graph TD
  Batch[Batch: Failed]
  Job1[Job 1: COMPLETE]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Failed]
  Batch --- Job1
  Batch --- Job2
  Batch --- Job3
  StatusReport[/StatusReport/]
  Batch --- StatusReport

Loading

Job 3 is restarted

graph TD
  Batch[Batch: Failed]
  Job1[Job 1: COMPLETE]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Processing]
  Batch --- Job1
  Batch --- Job2
  Batch --- Job3
  StatusReport[/StatusReport/]
  Batch --- StatusReport

Loading

Job 3 fails again

graph TD
  Batch[Batch: Failed]
  Job1[Job 1: COMPLETE]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Failed]
  Batch --- Job1
  Batch --- Job2
  Batch --- |notify| Job3
  StatusReport[/StatusReport/]
  Batch --- StatusReport

Loading

Admin changes batch state to UpdateReporting

graph TD
  Batch[Batch: UpdateReporting]
  Job1[Job 1: COMPLETE]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Failed]
  Batch --- Job1
  Batch --- Job2
  Batch --- Job3
  StatusReport[/StatusReport/]
  Batch --- StatusReport

Loading

Since there is no job state change since last report, no email is sent

graph TD
  Batch[Batch: UpdateReporting]
  Job1[Job 1: COMPLETE]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Failed]
  Batch --- Job1
  Batch --- Job2
  Batch --- Job3
  StatusReport[/StatusReport/]
  Batch --> StatusReport
  StatusReport -.-> Batch

Loading

Batch Status is Failed

graph TD
  Batch[Batch: Failed]
  Job1[Job 1: COMPLETE]
  Job2[Job 2: COMPLETE]
  Job3[Job 3: Failed]
  Batch --- Job1
  Batch --- Job2
  Batch --- Job3
  StatusReport[/StatusReport/]
  Batch --> StatusReport

Loading

Admin Deletes Batch after determining that resubmission of failed job is not possible

graph TD
  Batch[Batch: DELETED]

Loading