Data lifecycle management

[aneojgurhem]

Current status

As of now, in ArmoniK, we

• delete input payloads when they are consummed
• since API 3.10.0 and core 0.14.0, we added an RPC method to delete results data from the object storage
• use redis TTL to delete data

Issue

1. We need a simpler way for the users to manage their data.
2. We also need a finer grain for data management in case of large amount of data to process. Deleting data when they are not needed anymore may allow to ingest more data into ArmoniK during the same amount of time.

Proposal

1. Delete results at the end of the session

• This brings questions about the purpose of the session
• Close session
  • Delete results
  • Cannot submit new results and tasks
  • Running tasks are in an undefined state since results were deleted
• Cancel session
  • Delete results
  • Cannot submit new results and tasks
  • Lazy cancellation of running tasks
• What is the difference between cancelling or closing session when results are deleted ? Without results, running tasks does not mean much. Should we rename Cancel to Close ?
• We should add a flag for active task cancellation

2. Delete results when they are not referenced anymore

• Results expected client side can have a reference to the client (session ?) and this reference is removed when client deletes the result (they may want to download it beforehand)

[wkirschenmann]

IMHO, we should provide a way to make a distinction between different kind of data:

• data internal to the graph => can be deleted by ArmoniK when there are no more dependencies
• data external to the graph => have a longer life cycle to be retrieved when customer wants ; can be copied to another session ; can be deleted manually

Also, I would distinguish the different situation:

• end of submission from the client side => client cannot submit anymore, running tasks can still submit new tasks with new internal data (and existing external data)
• cancellation of execution => all is stopped, no data is deleted in case customer want to get some data for debugging purpose ; a flag should allow data to be removed
• end of usage => all remaining data (except logs & accounting) can be deleted.

[aneojgurhem]

1-2/ To me there are no distinctions between "internal" or "external" data, especially with the separation between task and data submission. They are data. Moreover, with a ref for the client in some of the data, client should remove this ref for data to be deleted. Data should be migrated to another session before the end of the session. I'd say that data are only associated to one session.

3/ So you mean another status for the session ? When do we delete ? The simpler way is during a call of some sort (cancel or close or whatever)
4/ I'd say we need flag for active task cancellation. If you want to debug data, don't close the session

[aneojgurhem]

We will focus first on session lifecycle and the data within the session. This will already help to improve data management in ArmoniK.
Then, we will open an new discussion on how to manage data at a finer grain.

We envision sessions as follows:

1/ The session is created with the status Open. Results can be added to the session and tasks can be submitted to the session.
2/ The session can be paused and resumed. When the session is paused, running tasks continue to run until completion but submitted tasks are removed from the queue and are not executed. Running tasks can still submit subtasks but they will not be inserted into the queue thus not being executed. When the session is resumed, tasks that can be submitted (dependencies resolved) are inserted into the queue. Tasks execution will resume.
3/ Terminating a session means that no new tasks will be submitted from the client side of the application. Tasks are still able to submit new tasks that will be executed. This status is optionnal and may help to clarify some use cases.
4/ Closing session will prevent task submission from client and tasks. Running tasks can finish and produce results but not submit new tasks. An option is available to cancel running tasks; these tasks will not run up to completion and will not be able to produce results.
5/ Delete a session will trigger lazy removing of data related to a session such as tasks payloads, data dependencies and ouputs in the case where these data are still here. Tasks, Data and Sessions metadata will not be removed at this step for accounting purposes.
6/ Metadata are removed when they exceed their configured Time To Live.

The following state diagram summarizes the worflow of a session.

stateDiagram-v2
    EOS: End of Submission
    [*] --> Open : create
    Open --> Pause : pause
    Pause --> Open : resume
    Open --> EOS : terminate
    Open --> Closed : close
    Pause --> Closed : close
    EOS --> Closed : close
    Closed --> Deleted : delete data
    Deleted --> [*] : delete metadata

Loading

[lemaitre-aneo]

State machine is good. I just think there is a missing state: I think you should be able to pause/unpause even when you have ended the submission.

[lemaitre-aneo]

stateDiagram-v2
    EOS: End of Submission
    PEOS: Pause End of Submission
    [*] --> Open : create
    Open --> Pause : pause
    Pause --> Open : resume
    Open --> EOS : terminate
    Open --> Closed : close
    Pause --> Closed : close
    Pause --> PEOS : terminate
    EOS --> PEOS : pause
    PEOS --> EOS : unpause
    PEOS --> Closed : close
    EOS --> Closed : close
    Closed --> Deleted : delete data
    Deleted --> [*] : delete metadata

Loading