Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Core: Fix numeric overflow of timestamp nano literal #11775

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
+60 −8
Open

Core: Fix numeric overflow of timestamp nano literal #11775

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
efe1d14
Core: Fix numeric overflow of timestamp nano literal
ebyhr Dec 13, 2024
2a9c8b7
fixup! Core: Fix numeric overflow of timestamp nano literal
ebyhr Dec 16, 2024
6828e0c
fixup! Core: Fix numeric overflow of timestamp nano literal
ebyhr Dec 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions api/src/main/java/org/apache/iceberg/expressions/Literals.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -300,8 +300,7 @@ public <T> Literal<T> to(Type type) {
case TIMESTAMP:
return (Literal<T>) new TimestampLiteral(value());
case TIMESTAMP_NANO:
// assume micros and convert to nanos to match the behavior in the timestamp case above
return new TimestampLiteral(value()).to(type);
return (Literal<T>) new TimestampNanoLiteral(value());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change seems correct to me, the previous behavior for this was to assume the value was in microseconds and then pass that through to TimestampLiteral but that can overflow and does not actually represent a nanosecond timestamp!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW I still think this is correct but it's worth getting other's perspective on this since we are changing one of the assumptions of how value is interpreted when the type to convert to is nanoseconds.

CC @nastra @epgif @jacobmarble @rdblue thoughts?

case DATE:
if ((long) Integer.MAX_VALUE < value()) {
return aboveMax();
Expand Down
55 changes: 55 additions & 0 deletions api/src/test/java/org/apache/iceberg/expressions/TestTimestampLiteralConversions.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;

import java.time.LocalDate;
import java.time.format.DateTimeParseException;
import org.apache.iceberg.types.Types;
import org.apache.iceberg.util.DateTimeUtil;
Expand Down Expand Up @@ -107,6 +108,33 @@ public void testTimestampMicrosToDateConversion() {
assertThat(dateOrdinal).isEqualTo(-1);
}

@Test
public void testTimestampNanoWithLongLiteral() {
amogh-jahagirdar marked this conversation as resolved.
Show resolved Hide resolved
// verify round-trip between timestamp_ns and long
Literal<Long> timestampNano =
Literal.of("2017-11-16T14:31:08.000000001").to(Types.TimestampNanoType.withoutZone());
assertThat(timestampNano.value()).isEqualTo(1510842668000000001L);

Literal<Long> longLiteral =
Literal.of(1510842668000000001L).to(Types.TimestampNanoType.withoutZone());
assertThat(longLiteral).isEqualTo(timestampNano);

// cast long literal to temporal types
assertThat(longLiteral.to(Types.DateType.get()).value())
.isEqualTo((int) LocalDate.of(2017, 11, 16).toEpochDay());

assertThat(longLiteral.to(Types.TimestampType.withoutZone()).value())
.isEqualTo(1510842668000000L);

assertThat(longLiteral.to(Types.TimestampType.withZone()).value()).isEqualTo(1510842668000000L);

assertThat(longLiteral.to(Types.TimestampNanoType.withoutZone()).value())
.isEqualTo(1510842668000000001L);

assertThat(longLiteral.to(Types.TimestampNanoType.withZone()).value())
.isEqualTo(1510842668000000001L);
}

@Test
public void testTimestampNanoToTimestampConversion() {
Literal<Long> timestamp =
Expand Down Expand Up @@ -158,6 +186,33 @@ public void testTimestampNanosToDateConversion() {
assertThat(dateOrdinal).isEqualTo(-1);
}

@Test
public void testTimestampNanoWithZoneWithLongLiteral() {
// verify round-trip between timestamptz_ns and long
Literal<Long> timestampNanoWithZone =
Literal.of("2017-11-16T14:31:08.000000001+01:00").to(Types.TimestampNanoType.withZone());
assertThat(timestampNanoWithZone.value()).isEqualTo(1510839068000000001L);

Literal<Long> longLiteral =
Literal.of(1510839068000000001L).to(Types.TimestampNanoType.withZone());
assertThat(longLiteral).isEqualTo(timestampNanoWithZone);

// cast long literal to temporal types
assertThat(longLiteral.to(Types.DateType.get()).value())
.isEqualTo((int) LocalDate.of(2017, 11, 16).toEpochDay());

assertThat(longLiteral.to(Types.TimestampType.withoutZone()).value())
.isEqualTo(1510839068000000L);

assertThat(longLiteral.to(Types.TimestampType.withZone()).value()).isEqualTo(1510839068000000L);

assertThat(longLiteral.to(Types.TimestampNanoType.withoutZone()).value())
.isEqualTo(1510839068000000001L);

assertThat(longLiteral.to(Types.TimestampNanoType.withoutZone()).value())
.isEqualTo(1510839068000000001L);
}

@Test
public void testTimestampNanosWithZoneConversion() {
Literal<CharSequence> isoTimestampNanosWithZoneOffset =
Expand Down
10 changes: 4 additions & 6 deletions api/src/test/java/org/apache/iceberg/types/TestConversions.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,16 +104,14 @@ public void testByteBufferConversions() {
.isEqualTo(new byte[] {-128, 26, 6, 0, 0, 0, 0, 0});
assertThat(Literal.of(400000L).to(TimestampType.withZone()).toByteBuffer().array())
.isEqualTo(new byte[] {-128, 26, 6, 0, 0, 0, 0, 0});
// values passed to assertConversion and Literal.of differ because Literal.of(...) assumes
// the value is in micros, which gets converted when to(TimestampNanoType) is called
assertConversion(
400000000L, TimestampNanoType.withoutZone(), new byte[] {0, -124, -41, 23, 0, 0, 0, 0});
400000L, TimestampNanoType.withoutZone(), new byte[] {-128, 26, 6, 0, 0, 0, 0, 0});
assertConversion(
400000000L, TimestampNanoType.withZone(), new byte[] {0, -124, -41, 23, 0, 0, 0, 0});
400000L, TimestampNanoType.withZone(), new byte[] {-128, 26, 6, 0, 0, 0, 0, 0});
assertThat(Literal.of(400000L).to(TimestampNanoType.withoutZone()).toByteBuffer().array())
.isEqualTo(new byte[] {0, -124, -41, 23, 0, 0, 0, 0});
.isEqualTo(new byte[] {-128, 26, 6, 0, 0, 0, 0, 0});
assertThat(Literal.of(400000L).to(TimestampNanoType.withZone()).toByteBuffer().array())
Comment on lines 111 to 113
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit confused how the original assertion was passing? Shouldn't have this always been equivalent to {-128, 26, 6, 0, 0, 0, 0, 0}?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the cause is the original logic called DateTimeUtil.microsToNanos method which multiples the value by 1000:

iceberg/api/src/main/java/org/apache/iceberg/expressions/Literals.java

Lines 444 to 445 in b9b61b1

case TIMESTAMP_NANO:
return (Literal<T>) new TimestampNanoLiteral(DateTimeUtil.microsToNanos(value()));

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see the comment on line 107/108. Could we update the assertConversion to instead test against 400000L and then remove the comment. At this point we are no longer having to pass in different values since we Literal.of(someLong).to(TimestampNanos) will always interpret someLong as nanoseconds.

.isEqualTo(new byte[] {0, -124, -41, 23, 0, 0, 0, 0});
.isEqualTo(new byte[] {-128, 26, 6, 0, 0, 0, 0, 0});

// strings are stored as UTF-8 bytes (without length)
// 'A' -> 65, 'B' -> 66, 'C' -> 67
Expand Down