[PL-133327] nixos/platform: use nix.settings for connect-timeout

[Ma27]

@osnyx this can be backported to 24.05 & 23.11. For 21.05 I'll need to check.

PL-133327

As discussed, it might be useful to have some kind of feature-flag for the connect-timeout since modifying declarations of nix.extraOptions is hard.

Instead, decided to just use nix.settings here, that way the timeout can be adjusted with primitives of the module system and without an additional option.

@flyingcircusio/release-managers

Release process

• Created changelog entry using ./changelog.sh: Followup for [PL-133327] nixos/platform: set Nix's connect-timeout to 1s #1256, no behavioral change.

PR release workflow (internal)

• PR has internal ticket
• internal issue ID (PL-…) part of branch name
• internal issue ID mentioned in PR description text
• ticket is on Platform agile board
• ticket state set to Pull request ready
• if ticket is more urgent than within the next few days, directly contact a member of the Platform team

Design notes

• Provide a feature toggle if the change might need to be adjusted/reverted quickly depending on context. Consider whether the default should be on or off. Example: rate limiting.
  • The option nix.settings.connect-timeout is now used. This can overriden to any value with mkForce.
• All customer-facing features and (NixOS) options need to be discoverable from documentation. Add or update relevant documentation such that hosted and guided customers can understand it as well.
  • nix.settings has documentation already.

Security implications

• Security requirements defined? (WHERE)
  • Maintaining Availability: Being able to quickly change the timeout in case of any issues.
• Security requirements tested? (EVIDENCE)
  • Activated this change on a test VM. Then, confirmed that I can change the timeout with

{ lib, ... }: {
  nix.settings.connect-timeout = lib.mkForce 23;
}

[osnyx]

We can also move the other extraOptions over to settings.
It's okay not to do this in this PR, because this keeps the change down to its intended minimal core, and it remains backportable.