[FEATURE REQUEST] Add basic HDFS storage option for catalogs

[rdsarvar]

Is your feature request related to a problem? Please describe.
Currently it appears that the storage options are geared towards cloud providers. To support companies running on premise I would like to request HDFS support.

Describe the solution you'd like
Catalog R/W support HDFS as a storage option.

For the first implementation we would like something basic:

• The service is able to R/W against a HDFS cluster without authentication/authorization required.

[flyrain]

We can add HDFS support, but handling credentials might be difficult. @sfc-gh-schen, can you provide more insight? I think it may not be possible for credential vending.

[rdsarvar]

Would permissive HDFS be a simpler first ask with potential evolutions in the future to support credentials? Aka start off assuming that the HDFS cluster is open (no Kerberos, etc), running on-premise protected by networking, and then consider a proper strategy in the future?

Sorry if I'm off the mark on what you meant by your last comment but I assumed it's relating to authn/authz against HDFS (and not mapping to an internal strategy in Polaris).

[flyrain]

Yup, it's reasonable to start with non-authentication and non-authorization for HDFS.

[rdsarvar]

I don't mind trying my hand at this for the simple case and provide a baseline for people to extend in the future.

I think in order for this to work, due to the DFS is created inside of Iceberg core with the Hadoop configuration object we initialize, we'll need to rely on the HADOOP_USER_NAME environment variable (which will mean all DFS interaction would be under the same username). Otherwise it'll default access to be the user running the service.

I can see if I find time one of these days to throw something together in a PR (as an non-tested PoC) just to get feedback on if it's something we want to move forward

[rdsarvar]

@flyrain QQ about the repo which I noticed starting to write the changes required by this MR:

1. Do the regtests have Open API Python templates that aren't commit into the repo? When I generate the files with the provided commands in the README it results in generated Python files without the license header and in some cases it breaks the functionality. An example diff of regtests/client/python/polaris/management/models/aws_storage_config_info.py:

         _obj = cls.model_validate({
             "storageType": obj.get("storageType"),
-            "allowedLocations": obj.get("allowedLocations"),
-            "roleArn": obj.get("roleArn")
+            "allowedLocations": obj.get("allowedLocations")
         })

2. I was thinking about how to add integration / e2e tests and noticed that there isn't really any integration tests (outside of polaris-service/src/test/java/io/polaris/service/catalog/PolarisSparkIntegrationTest.java), do we rely on the e2e tests for testing against the storage providers? -- Curious to what your preference would be for this repo

[flyrain]

cc @dennishuo @collado-mike @eric-maynard for the first question.

for 2, we cannot really do that without a sponsor of cloud environments. We have discussed using minIO to simulate it. But for HDFS, it should be OK to add integration tests.