-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Resynchronization #18
Comments
@davebenson how about using a list as a root object (similar to muon chaining), and applying a |
Having a random delimiter could be a good idea, however, I believe a 16-byte delimiter is an overkill. |
It might be a good idea to also allocate 1 byte for the counter (increases with each delimiter). In this case: 5 random bytes gives us |
@davebenson please provide some real-world use cases or elaborate on the motivation of this request. |
Here is an idea for supporting random delimiters without actually having to change the current format. Not sure if that is actually something to worry about (it's not like there are a ton of muon parsers out there that would break, nor that muon is considered a stable format yet), but it might help to keep the core format itself smaller. Let's assume that we're using the chaining approach suggested in the docs for concatenating muon objects. The main idea would be to encode the delimiter in one or more valid muon objects. Then one can inject this (sequence of) object(s) as the delimiter between the muon objects we actually care about during concatenation. A parser that knows we are using this approach could avoid allocating the delimiter objects, improving performance. Otherwise they would have to be removed after parsing. Since we're creating a list, that would mean removing every other entry. To signal that we're using delimiter objects we could do something similar to JavaScript's Next question: which object would be most suitable for this? Hidden: boring analysis of typed arrays, base-64 strings and LEB128 encoded values, before I realized all of which them are utterly inferior to using one or two U64 valuesFirst, we could try a typed array. Doing so would add a storage overhead of three bytes - One could store the separator as a base64-encoded string. That would increase the size by 1/3 since it encodes only six bits per byte, plus a final zero-terminator byte. Rounding up encoding 16 bytes would take up 23 bytes this way, and 8 bytes would take 12. A bit bigger, but on the other hand string creation and memory overhead are both a lot more optimized in JS engines. Another option is LEB128 encoding (or whatever varint encoding Muon will settle on - see #8). That would imply an overhead of The most efficient option is to use Again, just some thoughts on how to support this with relatively low complexity without actually having to change the Muon format. |
i'm not opposed to adding a specialized tag for this. we have more than enough unallocated markers. just need to understand the rationale |
@davebenson waiting for your inputs here |
I think it's probably fine to not implement this stuff in core muon.
Probably searching, indexing, and resynchronization are beyond the core
standard.
(JSONL isn't really an extension of JSON; it's a way of framing (aka
packetizing) json that provides certain conveniences that raw concatenated
json doesn't have. Assuming that your stream is objects and arrays, JSON
doesn't need end-of-record delimiters either)
But, JSONL is binary-searchable -- assuming the lines are sorted. And
files can be broken apart without reading the whole file - which is
important if you want to get parallelism in processing a single file. And
it is resynchronizable as well, meaning that you can seek midway into a
huge file, and also that corrupt entries can be skipped.
…On Fri, Aug 19, 2022 at 5:15 AM Volodymyr Shymanskyy < ***@***.***> wrote:
@davebenson <https://github.com/davebenson> waiting for your inputs here
—
Reply to this email directly, view it on GitHub
<#18 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AACNSBSF64GG26ABCM7ZW63VZ53H7ANCNFSM55U3VHCQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
muon is awesome, but the one thing it is lacking versus line-by-line json is ability to seek around randomly in the stream.
the jsonl separator \n is hard to beat, but support for raw data makes that impossible.
I have found the AVRO strategy of picking a random 16-byte delimiter is nice.
I think some byte (maybe 0xf0) should indicate a delimiter follows. the first time will define the delimiter (which is chosen randomly); subsequent times should just be validated.
The text was updated successfully, but these errors were encountered: