fc.agent.s3users: add timeouts, silence errors after user was deleted, monitoring

[leona-ya]

PL-133447

@flyingcircusio/release-managers

Release process

• Created changelog entry using ./changelog.sh

PR release workflow (internal)

• PR has internal ticket
• internal issue ID (PL-…) part of branch name
• internal issue ID mentioned in PR description text
• ticket is on Platform agile board
• ticket state set to Pull request ready
• if ticket is more urgent than within the next few days, directly contact a member of the Platform team

Design notes

• Provide a feature toggle if the change might need to be adjusted/reverted quickly depending on context. Consider whether the default should be on or off. Example: rate limiting.
• All customer-facing features and (NixOS) options need to be discoverable from documentation. Add or update relevant documentation such that hosted and guided customers can understand it as well.

Security implications

• Security requirements defined? (WHERE)
  • Timeouts could forcefully end a good execution of the program. This shouldn't happen in any of our DCs
  • Timeouts + silencing should be effective
  • Monitoring should work
• Security requirements tested? (EVIDENCE)
  • In our largest RGW cluster the script currently runs <1min. With our normal monitoring procedures, we already ensure that we would detect a failed systemd unit (timed-out units are in the failed status)
  • Tested timeouts (added a sleep to the script) + silencing
  • Tested monitoring in dev