Refactor NbBlock

[pietroppeter]

this is a major change, implements #168 (and also #117). Very WIP in a sandbox folder (see sandbox/notes.md)

[pietroppeter]

milestone reached, now in sandbox/minib3.nim both a minimal html and json backend work. (your json work was great @HugoGranstrom). Still lots of details to work on, but I am feeling more confident now!

A big realization was that in the end we might not need after all nim mustache anymore (we might still provide it for legacy reason and we should be able to provide a somewhat backward compatible NbLegacyDoc object that implements the default theme as before with partials), in a sense nimib itself is some code based (and block based) templating system with superpowers...

[pietroppeter]

(moved this note in sandbox/notes.md)

[HugoGranstrom]

Wow! Great work! 🥳 Looking good, it definitely seems way easier to define custom blocks now that the mustache backend is gone 😄 But I'm wondering how we will solve the problem of customizing an existing block type without it 🤔 But maybe that is what this idea is about?:

• can I also use a super method or something to wrap the rendered block in a NbBlock?
  • for example it could be used to add a class name inside a div
    • this could be important for example to customize code block appearance
  • or to add an optional id to a block
  • (this could anyway be added later)
  • and it could be added as something that by default a block does during rendering
  • it might have a different signature (takes rendering of content and outputs new rendering)

Mmh, no to customize an existing block the idea is to replace the function that renders it in the NbRenderobject here:

nimib/sandbox/minib3/minib.nim

Line 26 in 5e6727e

var nbToHtml: NbRender # since we need it for json, let's make it also for html

This sounds magical! Being able to modify the rendered code would be so cool and useful. The custom block code could stay very simple while we add all sort of features behind the scenes. Would we be able to do this with raw HTML though? Parse the HTML to and AST and modify the AST and then rewrite it as HTML?

The part above is still an idea and will need more fleshing out. I would definitely start with bare html and would allow customization only in the sense of: give me the rendered output and I can put things before or after. Any kind of refinement to that idea would need a use case that we think makes it worth it. And I would not probably push this during this refactor, maybe for now we do not even do what is sketched above (just to keep the scope manageable).

there is a tension between what you want to be able to serialize (content and specificities of a page) and what you want to be controlled (later) by the SSG (like the theme) and should not be serialized (also because it is redundant). Tension also with a third element which is the fact that rendering does need a theme

Is there really a tension? Couldn't we theoretically serialize the theme as well and just ignore/overwrite it in the SSG?

Yes, that is an option (serializing also the theme), but I guess it is kind of wasteful and also not "clean". I would like if possible to have a somewhat readable json. Especially if it is a low cost thing and a simple rule such as: everything that is in theme field of NbBlock is not serialized, the rest will.

Also, while we are refactoring the code, one part that has always confused me is nb.blk. I don't really see the point of it. Can't we just use nb.blocks[^1] instead (behind a template for easy access)? What value does adding it as its own variable (that you need to remember to set) bring?

Yeah, that is a good observation and it might not be needed after all. In principle you should not remember to set it since it should be in the NbBlock generating sugar, but if we can avoid needing it I could probably remove it. It will get trickier once I introduce real custom container blocks. My idea there at the moment is that NbDoc would have a add or similar field that represent what is used to add the block (the container could have multiple blocks fields, or maybe add, adds first to field first then to field second and that is it...). Then it could still be useful to have a reference to last block processed (which is needed for post processing, for the moment we have only the nbClearOutput in nimib, but I guess there could be other reasonable use cases).

Thanks for the feedback on the work-in-progress, it is useful :) nothing here is yet set in stone but I think a cleaner implementation is finally coming out.

[pietroppeter]

note, I just pushed a possible implementation of a NbContainer object (and NbDoc would inherit from this):

  NbContainer = ref object of NbBlock
    blocks: seq[NbBlock]
    parent: NbContainer

while the html backend works fine with it the json backend fails to serialize because of the circular reference. This is an issue with jsony that does not support this out of the box. It does not seem easy to fix. I tried to use skipHook (with a dev version of jsony), but since ref objects delegate to the non ref version, it does not work (I do not know if there is a way to get the type of the non ref version). I also tried with a custom dumpHook (which would not be nice to automate) but it also does not work. This issues should be probably better documented (maybe in a jsony issue?).

My current plan is to actually skip the parent field in NbContainer and have a containers: seq[NbContainer] field in Nb object (and add(nb: var Nb, blk: NbBlock) as a generic api that might support a different implementation.

[pietroppeter]

ok the new NbContainer implementation is indeed simpler, easier to implement and... it actually works! :)
I have also "hidden" the nb.blk api which is something that we might get rid of in the future.

Next steps:

• implement nbCode in minib
• implement nbCodeFromJs in minib
• think if there are essentially other types of blocks we need to test with the new refactoring
• if not go ahead and implement sugar to implement custom blocks (and refactor)
• clean up minib implementation
• reimplement all of nimib starting with the new minib poc

[HugoGranstrom]

Nice work! 🤩

I really like the withContainer and nb.add. Now that nb.blk is hidden, I don't have that much against it anymore tbh.

[pietroppeter]

notes from another discussion. in this PR we should also:

• reduce the number of global variables created (see the gotchas section of nimislides)
• have a non-global api where many blocks could be called without creating any globals (e.g. nb.text("hi"))

[HugoGranstrom]

So, I've started working on implementing nbJsFromCode. The part that I'm not sure about is where we should do the Nim->JS compilation. Should it happen before or after the transformation to JSON? It feels like it should happen before so that the JSON->HTML can happen without a Nim-compiler present + that steps become faster (important for SSG?).

Another thing that I'm wondering about is whether we should do the compilation the same way we are doing it now. Now we do it in nbSave: code
The thing is that we compile all of the nbJsFromCode blocks in the same file at the end, so we can't do the compilation before we have created all of them. I think the way we do it should be fine but I'm open to other ideas.

[pietroppeter]

Agree it should happen before Json creation and it could be fine to have it in the nbSave node as before (I do not see at the moment advantages for the refactoring for this part).

[HugoGranstrom]

Great, thanks for your input :D (does that sounds like an AI or is it just me? 🤣).

I agree, why change something that works. It is if/when we get plugin support (#209) that we could reconsider how we do things.

[HugoGranstrom]

A first (hacky) implementation of nbJsFromCode is now implemented. Just have to add back a few fields to the NbDoc to make it work fully. The one problem is that we will probably need to define the types in types.nim so that they can be accessed in jsutils.nim without circular imports.

One thing I realized we will need now with container blocks is an iterator that walks through them all and returns a flattened list of blocks.

[pietroppeter]

we will probably need to define the types in types.nim

ok, for the moment I would probably only move the js block types there (maybe even in a jstypes), which do have a reason to be special, not all types definitions for blocks

[HugoGranstrom]

Sounds like a good plan 👍

[HugoGranstrom]

I've started thinking about the sugar for defining a block now and we have for example:

type
  NbImage = ref object of NbBlock
    url: string
template nbImage*(turl: string) =
  let blk = NbImage(url: turl, kind: "NbImage")
  nb.add blk
func nbImageToHtml*(blk: NbBlock, nb: Nb): string =
  let blk = blk.NbImage
  "<img src= '" & blk.url & "'>"
nbToHtml.funcs["NbImage"] = nbImageToHtml
addNbBlockToJson(NbImage)

Which should be shortened to:

newNbBlock(nbImage):
  url: string
  toHtml:
    "<img src= '" & blk.url & "'>"

I'm not sure how the template could be automated with this sugar. It feels like it should be created manually unless we can come up with a nice way of abstracting it. So currently I think this is feasible:

newNbBlock(nbImage):
  url: string
  toHtml:
    "<img src= '" & blk.url & "'>"

template nbImage*(turl: string) =
  let blk = NbImage(url: turl, kind: "NbImage")
  nb.add blk

Something else to consider is inheritance. By default the block inherits from NbBlock but you could also write:

newNbBlock(nbImage of NbContainer):
  ...

to signify that it should inherit another block?

[HugoGranstrom]

Something we could do though to prevent the error of writing the wrong kind (wrong casing probably) in the template when creating the block, is to generate a newNbImage proc which has all the inputs except kind which is already prefilled. So the example would become:

newNbBlock(nbImage):
  url: string
  toHtml:
    "<img src= '" & blk.url & "'>"

template nbImage*(turl: string) =
  let blk = newNbImage(url=turl)
  nb.add blk

[pietroppeter]

Why the template cannot be created from the sugar macro? Is there some macro limitation I am not aware of (I know I am not aware of many things regarding macros... :))

[pietroppeter]

Btw I forgot to add them but I think it might worth adding also a new nb. api that is not template based, e.g.:

proc image*(nb: var NbContext, url: string) =
  nb.blk.add NbImage(url: url)

template nbImage*(url: string) =
  nb.image(url)

The new procedural api would also help to reduce the number of templates that nimib needs and incidentally should reduce the number of globals (of course for nbCode and similar we still would have a template in order to have untyped arguments)

[pietroppeter]

Also agree on the additional sugar for inheriting from other blocks

[HugoGranstrom]

Because we don't know the inputs and what to do with them 😄 Say we have two fields on the object and we only want the user to provide one of them. So the problem is that we don't know what the input to the template should be from just the information given right now. We could of course add the template body as a new section in the sugar syntax with the inputs as well. But then we have basically written the template already. So I think it is more flexible if we write the template manually. For example when you have a block that you want to create a few different templates with different defaults for.

[HugoGranstrom]

Yes it's a good idea to write the functional api as well 😄 I feel like I have regained some coding motivation now so I'll just need to find some time the coming weeks 😁

[pietroppeter]

Because we don't know the inputs and what to do with them

Ah right. I think we could provide maybe a default with all fields or indeed think of additional sugar but I guess it might be wiser for the moment to limit ourselves to leave it out of the sugar for now and see later if there is some good pattern that emerges

[pietroppeter]

And yeah probably we should just provide the functional api as default (with all the fields) and leave all the templates as a customization (indeed nbCode has a lot of variants, the skip one, the block one, ... and they all refer to the same type)

[pietroppeter]

In that case I would suggest the templates to use the functional api in their implementation (and this could be general advice).

[HugoGranstrom]

Because we don't know the inputs and what to do with them

Ah right. I think we could provide maybe a default with all fields or indeed think of additional sugar but I guess it might be wiser for the moment to limit ourselves to leave it out of the sugar for now and see later if there is some good pattern that emerges

The problem with creating a default template is that it can cause conflict with a manually created one.

[HugoGranstrom]

And yeah probably we should just provide the functional api as default (with all the fields) and leave all the templates as a customization (indeed nbCode has a lot of variants, the skip one, the block one, ... and they all refer to the same type)

So we automatically generate the functional api? How would we do that for blocks that require untyped arguments? Or do you mean we just create a constructor automatically?

[HugoGranstrom]

In that case I would suggest the templates to use the functional api in their implementation (and this could be general advice).

Agreed, when possible, the templates should be sugar on top of the functional api 👍

[HugoGranstrom]

Looking at this again, I'll try to summarize the last conversation we had:

• The sugar should generate a functional initializer with the kind prefilled
• The sugar should not generate the template (as we don't know what the inputs to it as they could be untyped)
• The sugar should not generate the functional API for the same reason, at least not by default. This is because it would create problems if we want to create a custom template/proc.

This means that for now, when defining a new block. We would have to create a functional api proc/template for each block type. And then create a "old-style" template using that (e.g. nbCode using nb.code)

This could be included in the sugar, but then we are basically defining the proc/template in there instead. Which could be a good thing as stuff would be nicely contained.

[HugoGranstrom]

Sooo, now the sugar is finally finished. So instead of this:

type
  NbImage = ref object of NbBlock
    url: string
template nbImage*(turl: string) =
  let blk = NbImage(url: turl, kind: "NbImage")
  nb.add blk
func nbImageToHtml*(blk: NbBlock, nb: Nb): string =
  let blk = blk.NbImage
  "<img src= '" & blk.url & "'>"
nbToHtml.funcs["NbImage"] = nbImageToHtml
addNbBlockToJson(NbImage)

We now do this:

newNbBlock(nbImage):
  url: string
  toHtml:
    "<img src= '" & blk.url & "'>"

func image*(nb: var Nb, url: string) =
  let blk = newNbImage(url=url)
  nb.add blk

template nbImage*(url: string) =
  nb.image(url)

Note that nbImage must be a template as it needs to access nb which isn't defined until we run nbInit