Skip to content

Commit

Permalink
[fix](iceberg) fix iceberg catalog bug and p2 test cases (apache#32898)
Browse files Browse the repository at this point in the history
1. Fix iceberg catalog bug

    This PR apache#30198 change the logic of `IcebergHMSExternalCatalog.java`,
    to get locationUrl by calling hive metastore's `getCatalog()` method.
    But this method only exists in hive 3+. So it will fail if we using hive 2.x.

    I temporary remove this logic, because this logic is only used from iceberg table writing.
    Which is still under development. We will rethink this logic later.

2. Fix test cases

    Some of P2 test cases missed `order_qt`. And because the output format of the floating point
    type is changed, some result in `out` files need to be regenerated.
  • Loading branch information
morningman authored Mar 27, 2024
1 parent 02d15d8 commit b29d395
Show file tree
Hide file tree
Showing 6 changed files with 17 additions and 35 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -17,16 +17,10 @@

package org.apache.doris.datasource.iceberg;

import org.apache.doris.common.Config;
import org.apache.doris.common.security.authentication.AuthenticationConfig;
import org.apache.doris.common.security.authentication.HadoopUGI;
import org.apache.doris.datasource.CatalogProperty;
import org.apache.doris.datasource.hive.HMSCachedClient;
import org.apache.doris.datasource.hive.HiveMetadataOps;
import org.apache.doris.datasource.property.PropertyConverter;
import org.apache.doris.datasource.property.constants.HMSProperties;

import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.iceberg.CatalogProperties;
import org.apache.iceberg.hive.HiveCatalog;

Expand All @@ -51,20 +45,8 @@ protected void initCatalog() {
Map<String, String> catalogProperties = new HashMap<>();
String metastoreUris = catalogProperty.getOrDefault(HMSProperties.HIVE_METASTORE_URIS, "");
catalogProperties.put(CatalogProperties.URI, metastoreUris);
HiveConf hiveConf = new HiveConf();
for (Map.Entry<String, String> kv : catalogProperty.getHadoopProperties().entrySet()) {
hiveConf.set(kv.getKey(), kv.getValue());
}
hiveConf.set(HiveConf.ConfVars.METASTORE_CLIENT_SOCKET_TIMEOUT.name(),
String.valueOf(Config.hive_metastore_client_timeout_second));
HadoopUGI.tryKrbLogin(this.getName(), AuthenticationConfig.getKerberosConfig(hiveConf,
AuthenticationConfig.HADOOP_KERBEROS_PRINCIPAL,
AuthenticationConfig.HADOOP_KERBEROS_KEYTAB));
initS3Param(hiveConf);
HMSCachedClient cachedClient = HiveMetadataOps.createCachedClient(hiveConf, 1, null);
String location = cachedClient.getCatalogLocation("hive");
catalogProperties.put(CatalogProperties.WAREHOUSE_LOCATION, location);
hiveCatalog.initialize(icebergCatalogType, catalogProperties);
catalog = hiveCatalog;
}
}

Loading

0 comments on commit b29d395

Please sign in to comment.