nimpretty

[haxscramper]

nimpretty itself is a fairly simple tool. Implementation leaves a lot to be desired - currently it is hacked into parser.nim and lexer.nim directly, with multiple (total of ~37 and ~23 respectively) conditional compilation checks like

  if p.tok.tokType in {tkCurlyDotRi, tkCurlyRi}:
    when defined(nimpretty):
      if p.tok.tokType == tkCurlyRi: curlyRiWasPragma(p.em)
    getTok(p)

This makes it really hard to provide a global code layout, leading to bad formatting and sometimes even broken code. Instead, the parser should be refactored to avoid information loss - the best idea so far was to simply keep the underlying tokens and then put them back in the generated code on formatting.

Engine for code layout can use an algorithm from "A New Approach to Optimal Code Formatting" (has a reference python implementation which I rewrote in nim), but tooling like dartfmt (accompanied by the article) or scalafmt (research paper) can be investigated as well.

[haxscramper]

This can be done as a three-part refactor. First we remove nimpretty, it is kind of useless, then make sure parser does not lose information, and finally implement the layout engine.

[haxscramper]

What about punctuation tokens and comments? They are not in the AST, but in order for code formatter to work they are necessary.

    tkParLe = "(", tkParRi = ")", tkBracketLe = "[",
    tkBracketRi = "]", tkCurlyLe = "{", tkCurlyRi = "}",
    tkBracketDotLe = "[.", tkBracketDotRi = ".]",
    tkCurlyDotLe = "{.", tkCurlyDotRi = ".}",
    tkParDotLe = "(.", tkParDotRi = ".)",
    tkComma = ",", tkSemiColon = ";",

[saem]

Ugh, this half baked implementation.

Ok, we need a source syntax tree or concrete syntax tree. It needs to refer to the actual source via the tokens, it's this thing that we run the prettyfier on.

Hilarious, this would be a relatively easy to extend mechanism and would be the back bone of source filters and all that, too.

[haxscramper]

ConfigRef stores a module graph, each module is in there, IIRC

Registered in the modules.partialInitModule

    packageSyms*: TStrTable
    modulesPerPackage*: Table[ItemId, TStrTable]

[haxscramper]

@saem If I understand correctly, your idea for token representation is pretty close to how it is done in zig, right? I just watched https://media.handmade-seattle.com/practical-data-oriented-design/, and they have pretty much identical representation.

[saem]

Yup, it is.

[haxscramper]

I have cleaned up the implementation from hmisc - https://github.com/haxscramper/hack/blob/master/testing/nim/implementation_ideas/blockfmt_generic.nim in conjunction with chunking approach from https://journal.stuffwithstuff.com/2015/09/08/the-hardest-program-ive-ever-written/ it can be used to implement pretty-printer once we have a concrete syntax tree.

Example of usage

  proc lytProc(
      args: openarray[LytBlock[Rune]], body: LytBlock[Rune]): LytBlock[Rune] =
    let
      h = T["proc ("]
      t = T[") = "]
      hsep = makeHSeparated[Rune](@args, T[", "])
      vsep = makeVSeparated[Rune](@args, T[", "])


    result = C[
      H[h, hsep, t, body],
      V[H[h, hsep, t, I[2, body]]],
      V[h, I[4, vsep], t, I[2, body]]
    ]

  for args in [1, 2]:
    for body in [20, 60]:
      echo toString lytProc(
        args = mapIt(0 ..< args, T[&"arg{it}: arg{it}_type"]),
        body = T[repeat("?", body)]
      )

Printed optimized layouts

proc (arg0: arg0_type) = ????????????????????
proc (
    arg0: arg0_type
) = 
  ????????????????????????????????????????????????????????????
proc (arg0: arg0_type, arg1: arg1_type) = ????????????????????
proc (
    arg0: arg0_type, 
    arg1: arg1_type
) = 
  ????????????????????????????????????????????????????????????

[haxscramper]

One thing that is still not obvious to me when it comes to using this library for code formatting, is mapping more complicated formatting opportunities to the code block tree. The library operates by first building a tree of nested blocks - line, stack, choice, wrap, text and then chooses the most optimal layout given configuration constraints (like right margin).

  echo toString(H[
    T["stmtPragmas* = "],
    V[
      T["{ wChecks,"],
      T["  wOverflowChecks,"],
      T["  wNilChecks }"]
  ]])


#  stmtPragmas* = { wChecks,
#                   wOverflowChecks,
#                   wNilChecks }

  echo toString(V[
    T["stmtPragmas* = {"],
    I[2, V[
      T["wChecks,      wObjChecks,"],
      T["wBoundChecks, wOverflowChecks, wNilChecks"]
    ]],
    T["}"]
  ])

# stmtPragmas* = {
#  wChecks,      wObjChecks,
#  wBoundChecks, wOverflowChecks, wNilChecks
# }

The code block tree might simply follow the AST structure, but it becomes problematic if we want to support both layout options - the first one is rather simple, and generally follows the AST structure of the identifier declaration. But the second one is more complicated, since it can only be built as a vertical stack of the name = {, then body, followed by }. Negative indentation is not implemented (there is no explicit "indentation" block, it is a H[T[" " * n], <main block>] wrapper).

I don't know how to fix this issue right now, even though I've spent quite some time thinking about it, but maybe the solution is obvious, and I just couldn't see it.

We can of course write a little more elaborate patterns, I might be overthinking this problem.

[saem]

@alaviss and I have occasionally discussed this idea of auto-inserting things like end, or semicolons etc.

This has a number of benefits, specifically it forces a certain amount of precision and specification. But I believe it could potentially help in this scenario.

[haxscramper]

https://github.com/haxscramper/hack/blob/6040096bd369104597578528ce102450f054822c/testing/nim/implementation_ideas/blockfmt_generic.nim updated block formatter implementation - now the layout itself is done using only minimally necessary information, and then resulting text is written by user that transformsm id into concrete chunk of text.

  LytStr* = object
    id*: LytStrId
    len*: int

Example implementation of the layouter that uses simple strings

type
  StrStore = ref object
    strings: seq[string]

proc str(store: var StrStore, str: string): LytStr =
  result = lytStrIdx(store.strings.len, str.len)
  store.strings.add str

proc str(s: StrStore, str: LytStr): string =
  s.strings[str.id.toIndex()]

proc toString(s: StrStore, opts: LytOptions, blc: LytBlock): string =
  let lyt = blc.toLayout(opts)
  for event in formatEvents(lyt):
    case event.kind:
      of layEvNewline:
        result.add "\n"

      of layEvSpaces:
        result.add repeat(" ", event.spaces)

      of layEvStr:
        result.add s.str(event.str)

[haxscramper]

With proper code formatter implemented, we can also remove renderer.nim, since it usually provides an absolutely abysmal output that, in a lot of cases, can't even be properly compiled as a nim code due to formatting errors.

[haxscramper]

Mass-formatting compiler source certainly will break most of the git history. For local git blame, this can be mitigated with https://www.moxio.com/blog/43/ignoring-bulk-change-commits-with-git-blame - --ignore-revs-file .git-blame-ignore-rev.

Support for this is requested in GitHub as well community/community#5033 and https://github.community/t/support-ignore-revs-file-in-githubs-blame-view/3256/27

[haxscramper]

One problem with nimpretty rendering is support for macro code - while it is next to impossible to provide good rendering for all known macros (could theoretically be solved by allowing user-provided pattern expressions with custom formatting rules, but I highly doubt this is worth it) the printer must support the unusual AST combinations to some extent