Add middleware to log server errors

[ConnorJC3]

Is this a bug fix or adding new feature?

Both? Kinda?

What is this PR about? / Why do we need it?

Fixes #2154

When the SDK hits a retryable error (for example, most server errors or network issues), it does not log the error in the log, preventing users and maintainers from discovering the nature of the issue. There is a built in logger for retries, but it would be sub-optimal to use that because it would flood the logs any time RequestLimitExceeded errors occurred. Thus, this PR instead introduces a simple logging middleware that allows us to customize the verbosity of the error message based on whether the error is a RLE/throttle error.

What testing is done?

Manually tested (We don't have a way to test log messages atm, and maybe shouldn't even be doing so)

[github-actions]

Code Coverage Diff

File	Old Coverage	New Coverage	Delta
github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/cloud/handlers.go	4.8%	6.5%	1.7

[ConnorJC3]

/hold putting this on hold to test edge cases, but it is ready for review

[ConnorJC3]

/remove-hold

[torredil]

Broke DNS and can confirm the relevant errors aren't swallowed:

kubectl get pods -n kube-system
NAME                                   READY   STATUS              RESTARTS       AGE
coredns-54d6f577c6-zz27k               0/1     ContainerCreating   0              3m31s
coredns-78f4c5f7c6-js6w6               0/1     ContainerCreating   0              4m3s
coredns-78f4c5f7c6-qpqkt               0/1     ContainerCreating   0              4m3s

kubectl logs ebs-csi-controller-6dcfc9d6bf-p6t6m -n kube-system

Defaulted container "ebs-plugin" out of: ebs-plugin, csi-provisioner, csi-attacher, csi-snapshotter, csi-resizer, liveness-probe
I1031 19:45:35.639359       1 main.go:157] "Initializing metadata"
I1031 19:45:35.643350       1 metadata.go:48] "Retrieved metadata from IMDS"
I1031 19:45:35.645069       1 driver.go:69] "Driver Information" Driver="ebs.csi.aws.com" Version="v1.36.0"
E1031 19:53:15.909066       1 handlers.go:79] "Unknown error attempting to contact AWS API" err="https response error StatusCode: 0, RequestID: , request send failed, Post \"https://ec2.us-east-1.amazonaws.com/\": dial tcp: lookup ec2.us-east-1.amazonaws.com on 10.100.0.10:53: read udp 192.168.23.128:44050->10.100.0.10:53: i/o timeout"

Thank you!

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: torredil

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [torredil]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment