Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add RowBinary format for SELECT operations #216

Open
11 of 21 tasks
slvrtrn opened this issue Dec 15, 2023 · 2 comments · May be fixed by #257
Open
11 of 21 tasks

Add RowBinary format for SELECT operations #216

slvrtrn opened this issue Dec 15, 2023 · 2 comments · May be fixed by #257
Assignees
Labels
enhancement New feature or request

Comments

@slvrtrn
Copy link
Contributor

slvrtrn commented Dec 15, 2023

Work in progress: #257

If you have any feedback on the following, please don't hesitate to comment.

Streaming will be supported out of the box by design.

Target supported ClickHouse data types with SELECT operations for the initial release; certain types are already completed in the branch:

  • Boolean
  • (U)Int8-32 as number
  • (U)Int64-256 as a native BigInt
  • Float32, Float64 as a number
  • String as a string
  • Nullable, LowCardinality
  • Date, Date32 as either a JS Date, a string, or UNIX seconds as a number/BigInt by default (needs the decision to be consistent with DateTime family types), with a custom mapper support.
  • DateTime, DateTime64 - with a custom mapper support. DateTime could be either an ISO format string or BigInt UNIX seconds. DateTime64 should be either a string or UNIX nanoseconds as a BigInt.
  • Enum - needs a decision. Maybe as a string by default, with a custom mapper (index) -> T
  • UUID as a string
  • FixedString(N) as a string
  • IPv4/v6 as a string
  • Decimal as string by default, support custom mapper
  • Array(T) - should use custom mappers provided for the value type
  • Map(K, V) - should use custom mappers provided for K and V types
  • Tuple(T1, T2, ...) - should use custom mappers provided for T* types
  • Geo types - likely, should use [number, number] for Point as the base for the remaining Geo types.

The types that may be included as well, depending on the format complexity:

Related issues:

@slvrtrn slvrtrn added the enhancement New feature or request label Dec 15, 2023
@slvrtrn slvrtrn self-assigned this Dec 15, 2023
@slvrtrn slvrtrn added this to the 1.1.0 milestone Mar 11, 2024
@slvrtrn slvrtrn changed the title Add RowBinary format for inserts/selects Add RowBinary format for SELECT operations Mar 15, 2024
@slvrtrn slvrtrn pinned this issue Mar 16, 2024
@cjk
Copy link

cjk commented Mar 16, 2024

@slvrtrn Thanks for working on this! I'm watching this topic closely since it'll be useful for the work I do. My workload also contains Arrays and Maps (to better cope with semi-hierarchical data) and even some Nested types.

Guess I'll wait until Arrays and Maps are supported before jumping in. I also consider moving away from Nested, since it tends to make things more complicated and support for de-/serializing is often lacking / coming last.
Often I can use Maps to replace Nested data.

Again, thanks for working on this. I plan at least to support in testing once progress is enough for my usecases.
Clickhouse and this Client are both very awesome! 🥇

BTW I'm using both INSERTs and SELECTs.

@slvrtrn
Copy link
Contributor Author

slvrtrn commented Mar 16, 2024

@cjk, thanks for your feedback. I haven't looked into Nested yet, as the core part was to get the numeric types right first, but I believe it will be one of the trickier data types to implement there.

If you could provide a sample dataset for benchmarks or just table DDLs (I will just generate the random data there in the right shape), that would be great.

@slvrtrn slvrtrn linked a pull request Apr 2, 2024 that will close this issue
3 tasks
@slvrtrn slvrtrn removed this from the 1.1.0 milestone Jun 6, 2024
@slvrtrn slvrtrn unpinned this issue Nov 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants