Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stack usage #286

Open
laurgarn opened this issue Jul 24, 2024 · 4 comments
Open

Stack usage #286

laurgarn opened this issue Jul 24, 2024 · 4 comments

Comments

@laurgarn
Copy link
Contributor

With a big network of data, json-io may take a lot of stack space both at writing and reading.
For example, I have to allocate 2 Mio of stack in order to read and/or write a json.gz of 50 Mio on disk.
This minimum amount varies impredictably.
I have crafted a compatible writer on a branch that is happy with just 256 kio of stack space, by trying to limit depth first strategy.
It is noticeably slower but still usable (17s vs 10s, and 18s to parse, no change) .
Topping on the cake, it can output a far more human friendly json when formatted.
This is not an advocacy to use my solution, but allowing a "least deep strategy" would be a nice plus.

@jdereg
Copy link
Owner

jdereg commented Aug 13, 2024

I'm interested in seeing what you have created.

@laurgarn
Copy link
Contributor Author

https://github.com/laurgarn/json-io/tree/write_less_deep_over_master
Kinda POC, especially because it adds a ObjectWriter totally aside JsonWriter.
I hope it will help you design something more, hum, polished than what I ended up with.

@jdereg
Copy link
Owner

jdereg commented Sep 11, 2024

The “stack” issue is relevant to both reading and writing. The parsing attempts to minimize stack depth, however, it does use recursion. I plan to release a version that eliminates the recursion using a graph traversal with a ‘visited’ set for cycle detection. I don’t have an ETA, but that is on my radar.

The same is true for writing JSON, and the same approach would eliminate the stack usage.

@laurgarn
Copy link
Contributor Author

You're right. I forgot to insist on the gain at reading side too.
Nice to know it is on your radar :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants