Architecture thoughts on serialize/deserialize capability

[j-simmons-phd]

With work wrapping up on the project create feature and our moving into work to define a standard set schema queries, I have been thinking we might benefit from having the ability to serialize our queries when we are creating them.

Here are some updated thoughts on what serializing might look like. Note, we will need to add the dendron notes to this thread shortly to consolidate things.

General approach

• I would like to have a library that feels a lot like the js yaml library
• two main functions: dump and load
• dump takes a js object and a model navigation path and instead of returning a string it puts the data onto the fs at the location that matches the model navigation path
• load takes a model navigation path and returns a js object based on fs contents at that location

[j-simmons-phd]

Some additional thought:

• I greatly prefer an implementation that does not need to know the schema in order for the library to function properly
• The dump and load functions must be one-to-one and onto (e.g. for a given js object there is only one way it will serialize, and deserializing the fs representation must return the original js object exactly)
• I think it is critical to be able to lazy load and lazy dump (e.g. only load/dump one level of the model at a time, instead of having to load/dump the full model every time)
• I think it is essential that the implementation create a representation on the fs that is easily understood by users when viewed directly
• What we called building the full model from the fs in older implementations of this tooling would be as simple loading the model from fs for the top level model element (with lazy loading disabled) and then writing the js object to a yaml file
• The implementation should isolate changes to the element that was actually modified (e.g. if an object representing a pizza party has a list of invitees, changes to the list of invitees are specifically about the list of invitees, not the fundamentals of the pizza party like the start and end times; so adding an invitee should only modify the list of invitees on the fs). This is why we want to do things like make list contents a separate file from the object that owns the list. This way adding or subtracting elements (or rearranging their order) only requires committing a new version of the list file

[j-simmons-phd]

• What we called building the full model from the fs in older implementations of this tooling would be as simple loading the model from fs for the top level model element (with lazy loading disabled) and then writing the js object to a yaml file

This statement may not fully cover building the full model, as we have to account for how we will include content from sub-projects and projects brought in via npm install in top level builds. Still, this will at least be a part of the implementation and it will simplify things compared to earlier work.

[j-simmons-phd]

Oh, one other thing...

I want to be able to do is push "big" strings (things like the njk templates) to separate files when we dump to fs (so that we can edit them as stand alone files) and keep basic strings directly in object files.

In past discussions we have established the idea of scalar values are stored in line and vector/object values are stored in separate files. So, I propose a string be considered scalar of it contains no newlines and vector if it contains any newlines, to enable strings to work as described above and also follow the convention we already established.

[j-simmons-phd]

I think this might be useful to create a short GUID like ID for unnamed elements (e.g. elements of lists). https://shortunique.id/