Hive: Add Hive 4 support and remove Hive runtime

[manuzhang]

As per consensus on https://lists.apache.org/thread/jfcqfw9vhq4j7h0kwnlf338jgyzcq8s4, this PR

1. Upgrade hive connectors to Hive 4.0.1
2. Remove hive 3 runtime
3. Remove hive 2 from iceberg-mr

[pvary]

Will check next week if it is not merged till then

[okumin]

LGTM!

[manuzhang]

I've created a separate PR #11801 to remove Hive runtime since we can't upgrade hive-metastore dependency until Spark 4.

[pvary]

What is the root cause behind the strict version requirements of older Spark versions? The 4.0.0 HMS supposed to be compatible with older clients

[manuzhang]

It's due to API changes like this and this.

[pvary]

It's due to API changes like this and this.

Could we just exclude the Hive dependency from the iceberg-hive-metastore in the old Spark module and rely on Spark to include the old Hive dependency? This would not work if we were using new Hive 4 features in the code, but it is not the case, so it seems OK for the Hive perspective.

[manuzhang]

Spark (and other modules) also depend on TestHiveMetastore from test modules of iceberg-hive-metastore. We cannot use old Hive dependency from Spark to run this class due to API changes.

[pvary]

Spark (and other modules) also depend on TestHiveMetastore from test modules of iceberg-hive-metastore. We cannot use old Hive dependency from Spark to run this class due to API changes.

TestHiveMetastore is a test only code. If we use a shaded version for testing and using our own Hive version for it, then it should be fine.

For the production code:
I would argue that if there are compatibility issues with the HMS Client (I don't know about any of them ATM), then as we use our own shaded avro version to write the manifest files, we probably HiveCatalog should use its own shaded HMSClient too.

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

[github-actions]

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

[deniskuzZ]

hi @manuzhang, do you plan to continue with the PR?

[manuzhang]

@deniskuzZ yes, I'm planning to work on Hive 4 support.