Clean up files in processingDir (temporary staging dir) #629
Conversation
|
This connects to #627 |
There was a problem hiding this comment.
Looks good overall @DanielCosme. I made one suggestion to improve the efficiency of the cleanup activities, but it's not a blocker.
| w.logger.Error(err, "failed to clean up", "path", tinfo.Bundle.FullPathBeforeStrip) | ||
| } | ||
| } | ||
| if tempBlob != "" { |
There was a problem hiding this comment.
Calling CleanUpActivity multiple times works, but there is some unnecessary overhead for temporal to schedule the activity and for the worker to pull it off the queue each time the activity runs. A more efficient option would be to have the clean up activity clean up all the temp files/directories at the same time. You could use https://github.com/artefactual-sdps/temporal-activities/blob/main/filesys/remove_activity.go or modify the local CleanUpActivity.
There was a problem hiding this comment.
Thanks for the feedback. Changes are here: 1c3f844
This PR will fix an issue where the files created in the
processingDir(when configured in the pipeline) were never cleaned up by Enduro.It also changes the (Temporal) context that the
clean-upactivity uses to a disconnected one, this avoids the (unintended) cancellation of the context when the session gets released.