Added a fix for generating the target partition path if base and part… #213
Conversation
…ition location are not matching in the source
| ReplicaLocationManager locationManager) { | ||
| try { | ||
| return locationManager.getPartitionLocation(sourcePartition); | ||
| } catch (CircusTrainException e) { |
There was a problem hiding this comment.
This is thrown when base location doesn't match partition location, strangely enough the actual data copy doesn't have a problem with this so this logic fixes it by generating the partition name (instead of taking the name from the source partition folder).
There was a problem hiding this comment.
Hmm I'm actually a bit concerned this might not work correctly with how this is implemented: https://github.com/HotelsDotCom/circus-train/blob/main/circus-train-core/src/main/java/com/hotels/bdp/circustrain/core/source/HdfsSnapshotLocationManager.java#L124
I need to find out how the different copiers generate the target paths.
|
Gonna close this this is not going to work correctly for some use cases (e.g. partition x is in bucket x and partition y is in bucket /y/z) and might lead to us not getting a notification and data being not in a different location than advertised in the Metastore. I'll update the issue with findings and rewrite it to better capture the requirements unfortunately that's a lot more work there is no quick fix imo |
fixes #212