Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to serialize Decimal and still keep the results in order (bitwise comparison) #610

Closed
KKould opened this issue Sep 25, 2023 · 3 comments · May be fixed by #676
Closed

How to serialize Decimal and still keep the results in order (bitwise comparison) #610

KKould opened this issue Sep 25, 2023 · 3 comments · May be fixed by #676

Comments

@KKould
Copy link

KKould commented Sep 25, 2023

I will store Decimal as a Key in RocksDB, but how can I keep Decimal in order after serialization and arrange Decimal in a more reasonable manner?

@paupino
Copy link
Owner

paupino commented Sep 26, 2023

Hi @KKould,

I'm not sure I quite understand the question here - but admittedly, I'm unfamiliar with RocksDB. If this is about key clustering, this may be more suitable to ask on the RocksDB project.

That said - I'll try to provide a couple of answers:

  • Rust Decimal has a c-repr feature if you wanted to align structure across 128 bits consistently. This would allow you to perform memory translation if needed.
  • There is a serialize function to consistently serialize into a collection of bytes. This is in a well-defined order.
  • You could also use something like Borsh to perform binary serialization.

I'm not sure if this helps at all. If not, if you could please provide some further context it may help direct me a bit more.

@paupino paupino closed this as completed Sep 26, 2023
@KKould
Copy link
Author

KKould commented Sep 28, 2023

Hi @paupino

Implement Memory Comparable: After serialization, it can still be sorted (same order as before serialization)
e.g. TIDB: https://github.com/pingcap/tidb/blob/48f6b35d442289721af6fd5753b4b9c151e1d384/util/codec/decimal.go#L25C22-L25C22

I tried serialize but it's not what I want

@paupino
Copy link
Owner

paupino commented Sep 28, 2023

Another thing you could try is to implement this manually. For example:

Then recombine them for your requirements. The code you've linked seems to write precision, scale then mantissa. You could emulate this by calculating the precision (i.e. length of the mantissa), followed by scale, followed by mantissa. Perhaps that would do the trick?

i.e. in pseudo like code

let mantissa = number.mantissa();
let scale = number.scale();
let precision = mantissa.ilog10() + 1;
// This is not space efficient
let mut bytes = Vec::new();
bytes.extend(precision.to_be_bytes());
bytes.extend(scale.to_be_bytes());
bytes.extend(mantissa.to_be_bytes());

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants