-
Notifications
You must be signed in to change notification settings - Fork 4.8k
HIVE-22167: TIMESTAMP - Backwards incompatible change: Hive 4 reads back binary RCFILE timestamps written by Hive 2.x incorrectly #5952
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
@check-spelling-bot Report🔴 Please reviewSee the files view or the action log for details. Unrecognized words (2)RCFILE Previously acknowledged words that are now absentaarry bytecode HIVEFETCHOUTPUTSERDE timestamplocal yyyyTo accept these unrecognized words as correct (and remove the previously acknowledged and now absent words), run the following commands... in a clone of the [email protected]:sercanCyberVision/hive.git repository
If the flagged items do not appear to be textIf items relate to a ...
|
@sercanCyberVision, Hive 3 or Hive 4? Note that Hive 3 is officially EOL. |
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryTimestamp.java
Outdated
Show resolved
Hide resolved
@deniskuzZ both Hive 3 and Hive 4 have the same behavior. |
@sercanCyberVision, please add the tests to validate the legacy functionality |
please remove the |
063b96d
to
8f31e61
Compare
@deniskuzZ, I have corrected the description and Jira title. |
@check-spelling-bot Report🔴 Please reviewSee the files view or the action log for details. Unrecognized words (2)RCFILE Previously acknowledged words that are now absentaarry bytecode HIVEFETCHOUTPUTSERDE timestamplocal yyyyTo accept these unrecognized words as correct (and remove the previously acknowledged and now absent words), run the following commands... in a clone of the [email protected]:sercanCyberVision/hive.git repository
If the flagged items do not appear to be textIf items relate to a ...
|
…ack binary RCFILE timestamps written by Hive 2.x incorrectly
8f31e61
to
ad9b416
Compare
@check-spelling-bot Report🔴 Please reviewSee the files view or the action log for details. Unrecognized words (2)RCFILE Previously acknowledged words that are now absentaarry bytecode HIVEFETCHOUTPUTSERDE timestamplocal yyyyTo accept these unrecognized words as correct (and remove the previously acknowledged and now absent words), run the following commands... in a clone of the [email protected]:sercanCyberVision/hive.git repository
If the flagged items do not appear to be textIf items relate to a ...
|
data.set(bytes.getData(), start); | ||
} else { | ||
// Legacy Hive 2.x behavior: interpret as local time | ||
data.set(bytes.getData(), start); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can be dropped
data.set(bytes.getData(), start); | ||
|
||
// Convert to java.sql.Timestamp | ||
Timestamp ts = data.getTimestamp(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if that is already handled by other file formats, can't we reuse/extract the conversion logic into util class?
|
@@ -52,6 +55,28 @@ public class LazyBinaryTimestamp extends | |||
*/ | |||
@Override | |||
public void init(ByteArrayRef bytes, int start, int length) { | |||
data.set(bytes.getData(), start); | |||
if (!legacyConversionEnabled) { | |||
// Hive 3.x default: interpret as UTC |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remove mention of Hive 3.x
MOTIVATION:
Hive 4 reads RCFile timestamps written by Hive 2 incorrectly. Please see https://issues.apache.org/jira/browse/HIVE-22167 for more details.
SOLUTION:
A new property has been added to switch between current and legacy interpretation.
CHECKS: