This change seems correct to me, the previous behavior for this was to assume the value was in microseconds and then pass that through to TimestampLiteral but that can overflow and does not actually represent a nanosecond timestamp!

FWIW I still think this is correct but it's worth getting other's perspective on this since we are changing one of the assumptions of how value is interpreted when the type to convert to is nanoseconds.

CC @nastra @epgif @jacobmarble @rdblue thoughts?

I think both before and after this change is correct. In Iceberg we had the assumption that everything is in microseconds. But this doesn't hold anymore now we have nano's. I do think the version after the change is more correct and more closely aligns with my expectations. If we can make sure that folks are not using this yet, I think this change is a good one 👍

I had a chat with @rdblue who reviewed the PR that introduced this, and it is actually on purpose. Spark always passes in microseconds, changing this would break this assumption with Spark. So I think we have to revert this line. That said, I do think we need to check (and raise an error) when it overflows. Easiest way of doing this is by converting it to nano's, and convert is back to micro's and check if it still the same value.

Thank you for sharing the context. What about other query engines? Actually, I found this issue when I was trying to support nanosecond precision in Trino Iceberg connector. As you may know, the max precision in Trino is picos (12).

@Fokko Could you confirm the above question? The current implementation is weird from other query engine's perspectives.

Sorry, the example I gave was not the best one. Previous to V3, Iceberg did not have nanoseconds, so therefore we have the assumption that every long we see, is in microseconds. Otherwise, the following query would change:

CREATE TABLE tbl(ts timestamp);

-- We want to have all the events from the future
SELECT * FROM tbl WHERE ts > 1739288553127 -- this is interpreted as microseconds

-- We need more precision
ALTER TABLE tbl MODIFY COLUMN ts timestamp_ns

SELECT * FROM tbl WHERE ts > 1739288553127t -- this is interpreted as nanoseconds, changing the result

The trick here is to let Trino construct TimestampNanoLiteral and push that into the evaluator, instead of a plain LongLiteral.

Please forgive my delayed participation here. I'm not familiar with either Spark or Trino, but @epgif and I did author the nanoseconds PR.

Originally, we did not allow conversion to long because the unit of the value was not known. When I implemented Spark filter pushdown, I added the conversion to timestamp because Spark internally uses the same microsecond representation. That set a precedent that longs are converted to timestamp using microseconds.

^^ @rdblue's comment #9008 (comment) is where we switched away from new TimestampNanoLiteral(value()).

we have the assumption that every long we see, is in microseconds

This is the core of the problem, right?

Here is an earlier comment from that same PR #9008 (comment)

Here is the problematic place in Trino:
https://github.com/trinodb/trino/blob/be9ae2f2d61aeee03352c34326416ea7e7fe1354/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/ExpressionConverter.java#L112-L121

            for (Range range : orderedRanges) {
                if (range.isSingleValue()) {
                    icebergValues.add(convertTrinoValueToIceberg(type, range.getLowBoundedValue()));
                }
                else {
                    rangeExpressions.add(toIcebergExpression(columnName, range));
                }
            }
            Expression ranges = or(rangeExpressions);
            Expression values = icebergValues.isEmpty() ? alwaysFalse() : in(columnName, icebergValues);

The convertTrinoValueToIceberg method returns Long (not literal) for timestamp(9) type. The bottom values expression is created with Iceberg's Expressions.in method.

let Trino construct TimestampNanoLiteral and push that into the evaluator, instead of a plain LongLiteral.

TimestampNanoLiteral and other literals' code are package-private. Let me know if there is a method we can use from other repositories.

I'm a bit confused how the original assertion was passing? Shouldn't have this always been equivalent to {-128, 26, 6, 0, 0, 0, 0, 0}?

I think the cause is the original logic called DateTimeUtil.microsToNanos method which multiples the value by 1000:

Ah I see the comment on line 107/108. Could we update the assertConversion to instead test against 400000L and then remove the comment. At this point we are no longer having to pass in different values since we Literal.of(someLong).to(TimestampNanos) will always interpret someLong as nanoseconds.