Limit number of selected record in metadata / header

[julie-bsx]

My colleague, David, opened an issue in regards to viewing selected records. This is a slightly different request that is focused more on debugging than developing, though it may also be useful for development when the simple hack I've used doesn't work so well.

At present the only way to limit the number of records which are selected is to use the hooks filter described in #50 . This works reasonably well in that it prevents the web IDE from blowing up when too many objects are created, though it can require a bit of trial and error.

I have a nodeFilter function which deselects the records with formats / definitions that I've debugged. As each format is debugged, I alter the repeat-until condition I'm using to include more of the file.

What I really want is something a lot closer to this expression, except I want records to be the number of nodes which were selected by hooks.nodeFilter and not the number which were read from the input file --

    repeat: until
    repeat-until: (((_io.pos + 20) > _io.size) or (records.size > 1000))

With the above code, the parser reads up to 1000 records. If there are no desired records, based on the nodeFilter expression, nothing shows up. And then I have to go back and change the 1000 value to something bigger. Lather, rinse, repeat.

[GreyCat]

Looks like the main culprit in your case is somewhat deeper, i.e. you are investigating something like a network packet stream, which can be gigabytes long, and there are several aspects of parsing a huge stream of lookalike layered packets like these:

• WebIDE must be able to load them all in memory (as JS typically works with in-memory arrays)
  • Actually, we might mitigate loading the whole file somewhat, if we'll decouple parsing and visualization (see Decoupling visualizer vs parser-running engine (visualizer protocol) kaitai_struct#143)
• All generated parser code needs to be as lazy as possible, i.e. to avoid creating too many parsed packet objects in memory at once. "Layered" lazy parsing is possible nowadays using lazy instances, and more work should be done on that by allowing seq attributes to be lazy (see Lazy parsing of sequence attributes kaitai_struct#133)
• And, finally, all generated parser code needs to be able to run in a stream-capable manner, so we won't need to resort to these tricks fixing .ksy file in order for it to be debuggable. This means that nodeFilter function would not actually filter things which are already created, but a generated JS parser code should emit not just an array, but a stream of records, which are to be created on the fly, and filtered on the fly.

[julie-bsx]

@GreyCat - I'm not disagreeing with you, but I am saying that you're wrong in this particular instance. For Kaitai Struct, as a thing, it obviously needs to do all of that. But web browsers -- and the WebIDE runs in a web browser -- have finite memory. The example file I used wasn't particularly huge, but it did repeatedly crash the browser because of the large number of DOM elements that were being generated. And I couldn't get to where I could use the nodeFilter hack until I made sure I was parsing any of the records correctly.

The reason I suggested using the file meta-data is so that the WebIDE could process the tag and KSC (or whatever else) could just plain ignore it.