s3api error improvements when FS is unmounted or degraded

[anandhu-karattu]

Environment info

• NooBaa Version: noobaa-core-5.17.1-20241109.el9.x86_64
• Platform: Standalone noobaa

Actual behavior

If the FS is unmounted on the protocol node and CES IP is still active on that node, s3api will try to access the buckets using the same CES IP.
So we need to provide proper error messages to the user if the FS is down or degraded.

For eg: In below cases buckets exists , however the FS is unmounted on the node where CES IP is currently active.

1. List buckets
   [root@fa-akarattu-mgmt-host ~]# account7000 s3api list-buckets
   urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.119'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
   {
       "Buckets": [],
       "Owner": {
           "DisplayName": "NooBaa",
           "ID": "123"
       }
   }

2. List contents of the bucket
   [root@fa-akarattu-mgmt-host ~]# account7000 s3 ls s3://anandhu1000
   urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.119'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: The specified bucket does not exist.

Expected behavior

We should check the status of FS and throw an error accordingly.

Steps to reproduce

1. Create a bucket
2. Make the FS down or degraded in the node where CES IP is residing (no HA, just the FS is inaccessible)
3. Access the bucket via s3api or s3 using the CES IP.

More information - Screenshots / Logs / Other output

[romayalon]

@anandhu-karattu I agree that the user experience might be better with your suggestion but AFAIK noobaa should not be cluster aware, this is not something we should do IMHO during S3 requests or at all. Probably mmhealth or other health tools should report it.
@nimrod-becker @madhuthorat @guymguym WDYT?

[nimrod-becker]

I agree that we'd want to conform to the definitions of the errors returned by S3 and not invent our own, health checks (which are outside of the S3 flow) should find this and report it.

[romayalon]

@nimrod-becker I don't think that he meant to a non S3 error, but a S3 InternalError with detailed error message about the unmounted FS. This will require NooBaa to check if the FS is degraded during the S3 request, I don't think we want to do that.

[nimrod-becker]

Right, ok so makes more sense. But I agree, we probably don't want to go into health checks during I/O, it can be a trigger for a health check but not have it inline

[romayalon]

putting here a similar bug for reference - #8039