Remember likely bad device names for an hour

[ConnorJC3]

Is this a bug fix or adding new feature?

Bug fix

What is this PR about? / Why do we need it?

Despite our best efforts, it is difficult to conclusively determine a safe block device name to use, and sometimes we get it wrong.

After thorough testing with the AMI from #1674, it was discovered that AMIs can include block device mappings (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html#ami-block-device-mapping) that behave unintuitively: block device names used by AMIs do NOT appear in the block device mappings returned by DescribeInstances

This leaves us with two options:

1. Call DescribeImages for every attachment operation to check for block device mappings on the AMI.
   This was discarded because it requires spending a massive amount of additional API calls across all users for a relatively niche case. It also doesn't solve the next iteration of this problem if there's another block device name constraint we don't know about or is added.

2. Remember block device names that are likely to be duds, and don't reuse them
   This is the solution this PR implements. When an AttachDisk returns a 'block device name is in use' error, a map is updated that remembers the bad device name for that instance for a period of time (currently, an instance is remembered as long as it has had an attach in the last hour).

What testing is done?

CI/Manual/Added new unit test cases

[github-actions]

Code Coverage Diff

This PR does not change the code coverage

[AndrewSirenko]

/lgtm cancel

[torredil]

Given this PR is not a one way door and generically applies a band-aid over all "bad device name" issues, I'm inclined to proceed with the changes as written.

Iterative improvements and re-factoring discussions can be continued in follow-up PRs. Will hold off on applying approve label so that Connor can resolve any outstanding concerns and merge when ready.

/lgtm

[ConnorJC3]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ConnorJC3

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ConnorJC3]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment