What type should logical timestamps be by default?

[frankmcsherry]

We currently use u64 as our internal timestamp type. This has some friction with others who use i64 as the timestamp type (Kafka) and those who use timestamp as the timestamp type (SQL). What are the benefits and problems of using each of the types?

Here are some first thoughts:

• u64: what we currently have. Changing it would probably introduce some bugs to track down.
• i64: more expressive range; a type that actually exists in SQL.
• timestamp: less friction with SQL queries, but more painful runtime compute (comparisons at the heart of DD).

Are there other candidates or considerations?

[aljoscha]

Forwarding my datapoint from slack: our current internal timestamp type of u64 won't fit into the timestamp type (Avro unsigned long, i64) of our CDCv2 schema: https://materialize.com/docs/connect/materialize-cdc/#main.

[frankmcsherry]

A related question is whether we should use integer milliseconds as our logical timestamps. This has the potential pain point that it limits the maximum number of distinct timestamps per second, and if one wanted high-throughput, high-resolution computations, we wouldn't be able to provide both I believe (i.e., we'd be limited to 1k distinguishable transactions per second).

[frankmcsherry]

Another option presented by @benesch is to have an opaque LogicalTimestamp type, whose valid range we would have to specify but which doesn't just magically typecheck as an existing type. We could write conversions to and from, but e.g. we could define its valid range to be at least positive integer milliseconds that are at most i64::MAX and then decide ourselves what to store it as. The question then becomes "what valid range should we commit to?"

[benesch]

I like the sound of this!