Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add rykv support for Serialization, Archive and Deserialization #32

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

Miaxos
Copy link

@Miaxos Miaxos commented Dec 17, 2021

Hello 👋 !

Just a small PR to add https://rkyv.org/ integration behind the rkyv feature!

@droundy
Copy link
Owner

droundy commented Dec 18, 2021

Looks like you'll need to disable this feature for the MSRV test.

I'd also like to see documentation of the feature, maybe a section on features in the readme?

@droundy
Copy link
Owner

droundy commented Dec 18, 2021

It also just occurred to me that there might be a possibility of a much more compact and efficient storage, if your archive format were able to store just one copy of each interned item. I'd you could store the values in an array and then for each Intern you would just store an index.

Not sure how you'd engineer this, but if sounds like rykv is performance oriented, so thought it might be interesting.

@djkoloski
Copy link

There is a preliminary implementation of archived string interning available here. It's still a bit rough for a release, but should get cleaned up soon.

@Miaxos
Copy link
Author

Miaxos commented Dec 20, 2021

Looks like you'll need to disable this feature for the MSRV test.

I'd also like to see documentation of the feature, maybe a section on features in the readme?

Okay, will do it!

It also just occurred to me that there might be a possibility of a much more compact and efficient storage, if your archive format were able to store just one copy of each interned item. I'd you could store the values in an array and then for each Intern you would just store an index.

Not sure how you'd engineer this, but if sounds like rykv is performance oriented, so thought it might be interesting.

Yes you're right about that, I tried doing something similar but I did not succeed right now 😅, so I kept this storage right now and added pass-through rkyv serialization/archive/deserialization to allow us using some optimisations but yeah, there are improvements to be made.

There is a preliminary implementation of archived string interning available here. It's still a bit rough for a release, but should get cleaned up soon.

Yes, I'm using this to do interning but I got troubles doing what I wanted: I'm deserializing a serialized rkyv hashmap with interning and I want to load it into the memory with interning while using the provided interning from rkyv to hydrate the Intern container and not creating it and checking against every value with Intern::new. I'll provide an isolated exemple as soon as possible for this part.

@droundy
Copy link
Owner

droundy commented Feb 11, 2022

Just checking in. It looks like this still fails tests?

@Miaxos
Copy link
Author

Miaxos commented Feb 11, 2022

Yes, I didn't advance on this subject yet, I'll take a look next week

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants