Enhance overall VFolder removal experience

[kyujin-cho]

Motivation

File operations take a long time and are very fragile.
Backend.AI should assume that vfolder deletion process can fail at any time.

Currently, there are two design issues:

• Purging a user account depends on the deletion of their owned vfolders.
  • If vfolder deletion takes a long time or fails, it may prevent the admins from purging user accounts.
• VFolders in the trash bin should always be restorable.
  • We should hide the vfolders "being deleted" in the filesystem from the users.
  • However, we should show them to the admin so that they can inspect and delete interrupted ones manually.

This is a follow-up to:

• Introduce vfolder status #575 (feat: Introduce vfolder status #713)
• refactor: re-define vfolder delete status #1892
• Clean up vfolder statuses #1915

and the completion of:

• Advanced vfolder deletion features (aka trash bin) #767

Proposed implementation details

Decoupling user account purging and vfolder deletion

• We need to make the user account purging done instantly.
  • fix: Immediately move vfolders to trash bin when purging user #2405
    • replacing/rewriting fix: Purge user after deleting records that reference the user #2374
• Either:
  • Introduce a placeholder user (like @ghost) who owns any vfolder when its owner user is purged.
  • Or, refactor: Remove the foreign key constraint of vfolders.{user,group} #2404.

Moving to trash bin

• When a user moves a vfolder to trash bin by calling DELETE /vfolders or DELETE /vfolders/{name} API
  • The vfolder state transits to DELETE_PENDING.
  • The filesystem remains unaffected.

Deleting forever

• Upon one of the following events:
  • When the user deletes the vfolder from the trash bin via DELETE /vfolders/delete-from-trash-bin API
    • Including when the admin triggers manual deletion of a vfolder in the trash bin via the control panel
      • Update the control-panel UI to show the vfolders in the DELETE_{ONGOING|ERROR} state.
      • Update the control-panel UI to show the orphan vfolders in the DELETE_PENDING state with their owner users/projects gone.
  • When the user "flushes" the entire trash bin via POST /vfolders/flush,
  • When the user or an external service triggers immediate deletion of a vfolder via ... API,
  • When the system triggers periodic cleanup of old vfolders in the trash bin,
  • Or, when the system triggers periodic cleanup of interrupted vfolder deletions,
• Do:
  • The vfolder state transits to DELETE_ONGOING.
  • Hide it from the user-observable trash bin as its content is no longer fully recoverable.
  • The storage-proxy initiates a background filesystem-level deletion task.
    • When this is successfully done, the state now moves to DELETE_COMPLETED.
    • Whne this fails, move the state to DELETE_ERROR (and store the error log somewhere?)
    • Since a vfolder may enter DELETE_{ONGOING|ERROR} states multiple times, we need refactor: Replace vfolder's status_history's type map with list #2116 as well.
    • Reliable VFolder deletion(purge) #1797 (feat: improve reliability & usability of VFolder removal API #1884)

History cleanup

• Never delete the vfolders records during the deletion process. Only delete vfolders in the DELETE_COMPLETE state after the data rentention period.
• User removes database record of VFolder in DELETE_COMPLETED status by calling DELETE /vfolders/purge API
  • Replace this with backend.ai mgr clear-history.

[fregataa]

We made a glossary related to VFolder deletion.

• move to trash: transit VFolder status to DELETE_PENDING and actual contents remains
• delete forever: delete the actual contents of the VFolder
• restore: transit VFolder status from DELETE_PENDING to READY

Does this issue suggest to replace delete forever with flush or add support for flush?

[achimnol]

Since the filesystem-level deletion process can take a long time, or gets failed and retried afterwards, we should show a small notice to the users like: "The deletion process and restoring the available storage space may take up to 12 hours."