[BugFix] Fix a bug that when catalog is unified catalog the background refresh for hive connector do not work #55215
Conversation
duanyyyyyyy
force-pushed
the
background_refresh_bug_fix
branch
from
3c683fa to
40ae02d
Compare
duanyyyyyyy
force-pushed
the
background_refresh_bug_fix
branch
2 times, most recently
from
b5f5f94 to
55b0ee0
Compare
|
Hi @stephen-shelby |
|
@duanyyyyyyy Do you mean there is only one type catalog like hive catalog could refresh when you create a unified catalog? but hudi/delta lake catalog could not refresh? |
Yes, for example when I create the unified catalog, only delta lake can be refresh background. |
|
|
|
||
| public class ConnectorProcessorName { | ||
|
|
||
| private final String catalogName; |
There was a problem hiding this comment.
I think catalogType is better
| private final HiveConnectorInternalMgr internalMgr; | ||
| private final HiveMetadataFactory metadataFactory; | ||
|
|
||
| public HiveConnector(ConnectorContext context) { | ||
| this.properties = context.getProperties(); | ||
| this.catalogName = context.getCatalogName(); | ||
| this.processorName = new ConnectorProcessorName(catalogName, "hive_connector"); |
There was a problem hiding this comment.
| this.processorName = new ConnectorProcessorName(catalogName, "hive_connector"); | |
| this.processorName = new ConnectorProcessorName(catalogName, "hive"); |
There was a problem hiding this comment.
Maintain consistency with Delta Lake.
| @@ -41,6 +43,7 @@ public HudiConnector(ConnectorContext context) { | |||
| CloudConfiguration cloudConfiguration = CloudConfigurationFactory.buildCloudConfigurationForStorage(properties); | |||
| HdfsEnvironment hdfsEnvironment = new HdfsEnvironment(cloudConfiguration); | |||
| this.catalogName = context.getCatalogName(); | |||
| this.processorName = new ConnectorProcessorName(catalogName, "hudi_connector"); | |||
There was a problem hiding this comment.
| this.processorName = new ConnectorProcessorName(catalogName, "hudi_connector"); | |
| this.processorName = new ConnectorProcessorName(catalogName, "hudi"); |
| cacheUpdateProcessors.put(catalogName, cache); | ||
| public void registerCacheUpdateProcessor(ConnectorProcessorName processorName, CacheUpdateProcessor cache) { | ||
| LOG.info("register to update {} metadata cache from {} in the ConnectorTableMetadataProcessor", | ||
| processorName.getConnectorName(), processorName.getConnectorName()); |
There was a problem hiding this comment.
why log ConnectorName twice, plz print more explicit information.
There was a problem hiding this comment.
I think you could print like this
register to update {}:{} metadata cache in the ConnectorTableMetadataProcessor", catalogName, catalogType
44f5cbd to
af843a9
Compare
…for hive connector do not work Signed-off-by: duanyyyyyyy <[email protected]>
Signed-off-by: duanyyyyyyy <[email protected]>
duanyyyyyyy
force-pushed
the
background_refresh_bug_fix
branch
from
af843a9 to
accdf9e
Compare
|
[Java-Extensions Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[FE Incremental Coverage Report]✅ pass : 23 / 26 (88.46%) file detail
|
[BE Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
|
Hi @Youngwb |
Fix a bug that when catalog is unified catalog the background refresh for hive connector do not work
Why I'm doing:
We found that when we use unified catalog the background refresh will not work.
Because that when we register into the map, we will use the same key.
What I'm doing:
Introduce a new object named
CatalogNameTypeand use this object as the key.Fixes #issue
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check: