Clojure Notebooks

[Cyrik]

This is related to the #1824 feature so we can have a public discussion where to take this and what decisions we made.

[Cyrik]

My thinking for next steps:

• @djblue is currently extracting portal views for us so that output cells can use the views
• LSP/nrepl tooling needs to be added to the cells. The LSP protocol does have notebook integration in its latest version but it's not takes a bit of infrastructure work.
• the users configured pretty printer should be used instead of using Calva for every value
• we probably want an optional way of saving the evaled output so that you don't have to run everything. this is probably more important for actual datasciencey use-cases. this isn't super difficult but we have to choose a format and decide where to save the extra data.
• there are probably a lot of quality of life features we can add to make working inside notebooks easier
• we should look into interactive outputs, where the cells can call back into Calva

[PEZ]

Is it possible to embed markdown in comments and have that work in both regular file editors and notebooks? I'm thinking that when a file is opened as a notebook it would treat all line comment content as markdown. And when saving notebooks, any markdown blocks would be prepended with line comment markers (semicolons).

[Cyrik]

That's my current idea yes. It'll probably be a bit more complicated than I'm imagining, but it should be very doable since we do have full power of transformation in both directions.

[seancorfield]

As mentioned on Slack, to use a notebook to edit and test production code, you need some way to have evaluable cells that are not just top-level forms -- akin to how RCFs (comment blocks) can be used today, where Calva treats the contents as a series of top-level forms.

I don't know how that maps to the notebook "mindset" but for me it's really important that all the exploratory code, examples, and REPL-based "tests" stay in the production code files (as comment blocks) so none of that context is lost.

[Cyrik]

There's kind of a big foundational decision that we need to make for notebooks:
Do we show everything that's in the Clojure file in the notebook or do we try to present a clean notebook by removing some things? This is mostly a question around whitespace but also impacts things like rich comments.
Some examples:

(defn foo []
  (println "bar"))

(defn bar []
  (println "bar"))

This Clojure file can produce these two cells if we keep all the whitespace (\n is in there so github shows the lines):

(defn foo []
  (println "bar"))

\n
\n
(defn bar []
  (println "bar"))

Or we could clean it up and remember the whitespace as metadata on the cells:

(defn foo []
  (println "bar"))

(defn bar []
  (println "bar"))

As for rich comments:

(comment
  (+ 40 2)

  42)

With everything version:

(comment

\n
  (+ 40 2)

\n
\n
  42

)

Or without:

(+ 40 2)

42

As you can see, idents make it even more difficult.

I've now built the latest PRs in the "clean" style and saving does work correctly. It does get a lot more difficult when you create new cells that don't have inherent metadata so I have to make some decisions around adding whitespace on save.
IMHO this is a fair tradeoff. You get cleaner-looking notebooks but it's harder to control the layout from inside the notebook.

The reason I'm trying to make a decision now is that it's hard to change to something different later. This is pretty much baked into the representation and will also have to be built into clojure-lsp handling of notebooks since it will also get the cells and will have to translate them into a file representation.
So @ericdallo @mainej you might also have something to say here since I'm potentially making decisions for other LSP clients as well.

[mainej]

tldr: I'd say in .clj files, treat each top-level form—and each second-level form inside a top-level comment block—as a cell. And from a LSP server developer perspective, I'd say clojure-lsp, when concatenating and analyzing a notebook, should interpose newline characters between cells. Or alternatively, clojure-lsp should not interpose anything, and clients should append newlines to cells unless they have some specific reason not to.

I have to admit I haven't thought a lot about notebooks, either in the data-science way or the literate-programming way. They've always brought to mind .md files with code blocks in them, with clear indicators for what language each code block is in. But it sounds like that's the easy case, because in .md the text itself specifies which ranges are code and which aren't. This conversation is about a harder case—treating .clj files as notebooks. And in that case we, as LSP client and server developers, have to decide which ranges are code vs. not-code. Am I following so far, or am I off in some other rabbit hole?

Naively, I'd expect this to be a client-side problem, primarily.

I see why a client has several decisions to make. First, the client has to decide how to translate edits in a rendered wysiwyg notebook back into text and vice versa, keeping in mind whether the file is .md or .clj. I'm sure that's tricky to standardize, and I see why you'd want to coordinate with other clients. You wouldn't want files to be reformatted every time they're read by one client or another.

Second, in .clj files the client has to decide how to fence off some ranges as code and others as not-code. To me the natural answer is what you @Cyrik and @PEZ seem to be describing—each top-level form is a code block, and everything in between is non-code. The one exception is top-level comment blocks, in which each second-level form is a code block, and again, everything between those is non-code. The nesting stops there. That's roughly what I think most Clojure developers have come to expect from editors and tools. (And therefore it's an acceptable lowest common denominator from a tool-developer perspective. It's nice if we can write tooling that understands a def inside a defn inside a loop inside a macro inside a rich comment. But if we throw our hands up at some point, nobody blames us.)

What I don't follow is exactly how this affects servers.

From my meager knowledge about LSP's notebook specification, a server has to understand only two things. A) a "file" is no longer one chunk of text but is now a list of chunks of text, that should be concatenated and analyzed together as code. And B) it can't expect to be able to read that code from disk, because whatever's on disk is markdown, not code.

The one unspecified thing is how the server should do the concatenation. I think you're saying we have 3 choices.

Consider the following two cells (with | indicating the cell range):

|3 ;; comment about 3|

|4|

Option 1: concatenate with (string/join "" cell-texts)

3 ;; comment about 34

Option 2: concatenate with (string/join "\n" cell-texts)

3 ;; comment about 3
4

Option 3: communicate whitespace via LSP's cell metadata field, something like

(->> cells
     (map (fn [{:keys [text cell-metadata]}]
            (str (:leading-ws cell-metadata) text (:trailing-ws cell-metadata))))
     (string/join ""))

If this is right, I'd vote for option 2.

Option 1 might be what the LSP spec writers were expecting, although I'm not sure. As the example concatenation shows, a trailing comment in one cell would comment out the first line of the next cell. That'd be true in any language that allows trailing comments, which is most of them. That seems very broken to me and makes me wonder—can we look at what VS Code does with JavaScript or Python notebooks? Does it always append newlines to cells? If so, perhaps Calva should do the same thing, in which case I'd change my vote to option 1.

Option 3 is a lot to explain to other clients who are implementing notebooks.

Option 2 seems like a reasonable middle ground. It's easy to explain. It wouldn't matter if some clients add their own newline while others don't. Would a client ever need to send two cells which shouldn't have a newline interposed? I can't imagine when that would happen.

Are there other implicit assumptions a client makes that a server would have to understand?

As a final piece of clarification, have I understood what you mean when you talk about "cell metadata"—that you're talking about the LSP field? Or are you talking about a Calva internal thing? You've said "new cells ... don't have inherent metadata". Are you saying that a cell interpreted from an on-disk file could be assigned whitespace metadata, but a cell created when a user chooses "Notebook -> Insert Cell" in their editor would have to be given that metadata?

[Cyrik]

Thanks for having a look at this @mainej !
You are partially correct that this is more of a client problem.
The interesting bit for the server is the concatenation of the cells. It's not really true however that the server can't look at the file directly, at least not in our case. Since we are using normal .clj files as notebooks it's fine to treat them as normal files. It would however mean that the server has to respect any extra translation stuff we do. This is kind of beside the point however since the LSP notebook cell calls will provide you with the contents anyway.

I'm currently hoping that option 2 will be ok on the LSP side for the most part, but I'm not certain that there aren't any corner cases if the Client does option 3 itself. The difficulty is usually in mapping cell/line/coll info back and forth.
It does get more complicated once we get into rich comments since we'd want to not show the (comment ) part of it and only put the "this is inside a comment" information into the metadata. If the server ignores this metadata it will not use any special comment rules that might be set for clj-kondo.
The bigger reason to at least partially understand the metadata on the server side is comments. The current plan is to treat any comment line starting with ;; outside a top-level cell as markdown, the same way clerk does. This means that we'd drop the ;; part before putting the comment into a markdown cell. The server would have to understand that this should not be treated as real code.
I'm guessing there might be more special cases in the future, but I'm hoping to avoid too many cases on the server side.

As for your questions:
As far as I know, all the other notebooks either don't use LSP or are still relying on special client code that translates notebooks into file representations and the LSP doesn't know anything about the notebook itself. The addition of notebooks to LSP is new in the latest version.
Yes, when I'm saying metadata I mean the LSP field. This is also an actual parameter on the vscode cell representation. I'm storing extra information in there, like the leading/trailing whitespace you mentioned. This metadata is filled when we parse a .clj file into a cell representation, but as far as I can tell there's no way to add any data onto cells created by "Insert Cell". This is especially annoying in the case of rich comment blocks because I have no good way of marking a new cell as being inside a comment block. My current solution to this is to specially mark the first and last cell inside a comment block so that any cells in between are put back into that surrounding comment.

Once rich comment and markdown comment handling are done inside Calva I'm hoping to add some basic handling into clojure-lsp. This will validate if we can actually stick with a mix of options 2 and 3 without running into any weirdness.

[mainej]

we'd drop the ;; part before putting the comment into a markdown cell. The server would have to understand that this should not be treated as real code.

I think you're saying you intend to sync all the cells to the server, including these cells which are sort of markdown but also sort of Clojure comments.

My interpretation of the spec was that a server can tell a client which cells it's interested in. I'd expect clojure-lsp to subscribe to Clojure cells, but not to markdown cells. After all, it isn't a markdown language server. That is, it's fine if the client wants to split up a .clj file, converting some ;; comments into markdown cells, leaving other parts as Clojure code cells. The client can do that, as long as it's confident it can put those cells back together into a file. But the details of splitting up the file and reassembling it should be opaque to the server. The server should live in a simpler world where it doesn't have to know anything about "markdown cells that can be converted to Clojure comments" or "groups of cells that are surrounded by a comment block". The server doesn't care whether the Clojure code it's fed has had some or all of the comments stripped out, it just wants to be able to analyze the code after it's all concatenated back together, via Option 1 or Option 2. As long as it can do that, it's happy. It might not be exactly the same analysis as you'd get if you opened the file as a Clojure file instead of a Notebook file, but that's the price you pay for getting to see the document as a notebook; and hopefully the analysis is not that different.

I think it all needs to work that way so that clojure-lsp can support real notebooks (i.e. .md notebooks) with several languages in them (e.g markdown, Clojure, and python). I think that was one of the spec designer's goals of attaching language to each cell.

Does that interpretation make sense? The part of the spec that talks about subscribing to cells of certain types seems to cover scenarios I don't understand, so it's very possible I'm not understanding how this is all supposed to work.

With that interpretation, I think that if it wants, Calva can treat .clj files as a weird markdown format, but that's its decision/problem. When handling a .clj notebook perhaps the server could subscribe to all the cells, including the markdown cells, and do a similar translation of markdown back into ;; comments (Option 3). But I don't think it should do that. Similarly for comment blocks. If Calva strips off (comment and ), that's fine, but the server shouldn't know that's happening. Again, you might get slightly different analysis, but that's the price you pay.

Or to put it another way, given the 3 options, my strongest vote is: the server should not have to do option 3, even if the client is doing something like that.

From a wider perspective, I'd like to express my skepticism of using metadata fields. The LSP spec definitely has deficiencies. I think subtleties about notebooks (like, for example that a comment block is sort of a notebook within another notebook) are one such deficiency. Having some deficiencies is not so surprising—after all this is the first draft of the spec that has included notebooks. Hopefully the spec will evolve to address these problems. In the meantime it's tempting to use metadata fields for out-of-band client-server communication that works around the deficiencies. (clojure-lsp does something similar with the data field on completion items, IMO, unfortunately.) But I, personally, would rather wait for the spec to evolve than create a temporary solution that clojure-lsp ends up supporting forever even if it contradicts the evolved spec. This is doubly true if the temporary solution was crafted for one particular editor.

Those are my opinions on comments. There's another aspect to this, which you've mentioned briefly @Cyrik, which is line numbers. I've been thinking about that more, and wow... that's a hard problem. It helps me understand why you've been considering syncing the whole document, including comments. If the server has only part of a document, constructed by concatenating (Option 1 or 2) only some of the cells, then the row/col data info from clj-kondo and rewrite-clj can't be translated directly back to positions in the .clj document. But don't you have a coordinate transformation problem anyway? The file/row/col coordinate system that clj-kondo and rewrite-clj understand is different than the cell/row/col coordinate system that the client understands. I believe that's true regardless of whether you sync all the cells vs just some of the cells. Am I right that the coordinates in a cell are relative to the cell? That is, do the clients assume the origin of a cell is 0,0? I think they must, because I don't see anywhere in the spec where the client could tell the server about a different origin. Assuming they do, I think a server that handles notebooks has to be responsible for transforming coordinate systems, the coordinate system it gets by concatenating several cells together, and the coordinate systems (plural!) established by the cells.

There's lots of subtlety to this problem. After parsing a partial document (concatenated via option 1 or 2), you have to go back through it, identifying the locations which start or end a cell. Then you have to translate their file/row/col metadata, which is relative to the start of the document, into cell/row/col metadata, which is relative to the starts of the cells. If you edit the parse tree things get even trickier. Could edits cross cell boundaries? I think that's unlikely, but if it's possible then the server should really know how to split an edit into several smaller per-cell edits.

Yikes. Sounds hard. But I think it's all necessary to say that clojure-lsp truly understands notebooks. That said, I'd love to find out I'm over-complicating things—I can definitely be guilty of that.

Taking one giant step back... maybe notebooks aren't the right model. That is, Calva may want to treat .clj files as notebooks internally, rendering comments as markdown, splitting top-level forms up into cells, etc. But perhaps it should hide that from the server. That is, the server already understands how to process .clj files, as long as they're one big blob. Calva could ask the server to treat the file as a file, not a notebook. Calva could still ask the server to give it navigation data, or refactoring edits, but using the file's coordinate system. Or to put it another way, perhaps this Calva feature doesn't have to tackle the complexity of LSP notebooks. That can happen when Calva really needs to understand .md notebooks that happen to have a few Clojure code blocks in them.

[Cyrik]

You're completely right, I totally forgot that you don't need to send all of the cells. I still had my mental model from work, where the whole file content is relevant and newlines can potentially make a difference. Not sending markup cells sounds like a great solution to that problem.
It somehow also felt like the simpler solution, by pushing the complexity of notebook weirdness to the edges and running on the same file representation the client would produce when saving. I see that this might be wrong in this case, but we'll see once we get into the details of the implementation.

I slightly disagree with you on the metadata field in the general, but you are probably right in this case. In my case I "control" both the client and the server so it feels natural to add any extras I'd need to do what I want, but clojure-lsp does serve a broader range of clients and has to be backward compatible so metadata is a bigger problem. In this case, the different handling of comment blocks is also a pretty small problem, so we should try to get by with the base protocol here. If we ever really really need some special handling we should probably try to come up with a more generic way of handling that case.

Coordinate systems:
Yes, this is a big source of complexity, which was the major reason I was trying to have the same underlying representation on the client and the server. Each cell is kind of its own document and starts as 0,0. So even easy calls like hover get more complicated. This is would be the request:

[Trace - 12:14:02 AM] Sending request 'textDocument/hover - (36)'.
Params: {
    "textDocument": {
        "uri": "vscode-notebook-cell:/Users/lukas/Workspace/testing/test.clj#W1sZmlsZQ%3D%3D"
    },
    "position": {
        "line": 0,
        "character": 3
    }
}

You can see that you now need special handling for the URI and a way to translate the cell coordinates into the coordinates of your internal representation. When sending back anything the server also has to translate it back and find the correct cell to show it on.
The good news is that I already have code to do this on the server side and unless I missed something major it's not too crazy, but there might be surprises coming. As you mentioned, doing rewrites will be even more difficult and I didn't have to do that yet.

As for your last point: I do get where you're coming from, trying to get back into a simple representation and pretending notebooks don't really exist. This is how the notebook extensions in vscode health with LSP interactions until the latest version (well they still do since none of them changed it yet I think).
The problem with this way of doing it is that you still have to do all the translation between representations, you're just doing it all in the client. The next client that wants to do notebooks is going to write the same exact code to do it and so on. So now we're moving away from the LSP idea of having the difficult, common stuff in the server.
Adding to this, we'd basically have to proxy all the LSP calls that the built-in client sends, translate them into file-based requests and send those to the actual server. That seems like a lot of brittle work.
So my vote would be for having the difficult translation layer on the server but trying to not add any client-specific stuff to it.

Overall I'm really thankful for your perspective on this. I've been thinking of clojure-lsp and Calva too much as a package when in reality it needs to have clear boundaries. It's nice to know you're going to push back when I try to add stuff that's too specific to Calva's needs 😄

[mainej]

The good news is that I already have code to do this on the server side and unless I missed something major it's not too crazy.

That's amazing, impressive, and means you're already pretty close to a solution that can accommodate notebooks that use several languages. I'm really looking forward to seeing how it turns out.

As you mentioned, doing rewrites will be even more difficult and I didn't have to do that yet.

Yeah, still sounds pretty hairy. But less so if you've got some tools for coordinate system transformations.

my vote would be for having the difficult translation layer on the server but trying to not add any client-specific stuff to it.

Agreed, that sounds like a reasonable balance. I definitely understand that the idea is to have the LSP server do as much of the hard stuff as possible. And at the same time, staying within the spec means clojure-lsp has fewer client-specific bugs to fight.

I've been thinking of clojure-lsp and Calva too much as a package when in reality it needs to have clear boundaries. It's nice to know you're going to push back when I try to add stuff that's too specific to Calva's needs

I've been guilty of this too—although I tend to over-focus on Emac's needs and quirks instead of Calva's.

Overall I'm really thankful for your perspective on this.

My pleasure, although I'm personally grateful for the Calva maintainers' and community's involvement in clojure-lsp. That feedback loop is critical to making sure clojure-lsp works not just in Calva, but also in other editors that I and the other clojure-lsp maintainers don't use as much daily.

[orestis]

Having used the current implementation of Clojure Notebooks (as of Calva v2.0.321), I have some feedback. This is based on me watching the sci-clj presentation and then trying it out for the first time in a real use-case I have.

The use case

We have a namespace called fiddle.clj that contains bits and pieces of useful little functions that we use when we connect to the production REPL. Together with these functions, we have a ton of RCF where we actually do exploratory programming, and the occasional data fix.

The current workflow

1. Connect an SSH tunnel to prod
2. Connect Calva to the remote nREPL
3. Evaluate the whole file (load all the functions)
4. Either find an existing comment block that does something similar, or create a new one
5. REPL away

How do notebooks come into play
Some times, the values we get back need to be filtered, inspected, or presented in nicer ways. I have tried to setup Portal for this, but it is a bit clunky (I have a few Joyride functions to help me, but it's not ideal). I was hoping I could evaluate code bit by bit, and then use the mini-portals to explore the data, then iterate.

First impressions

1. I was thrown off by the fact that there's no grouping of cells. In the CLJ editor, I use the comment block as a kind of fence, but in the editor the expressions inside the RCF carry the same visual weight as the top-level functions.
2. This means that when I make changes to the cells, I'm terrorized about accidentally putting an expression at the top-level, where it would be evaluated when I load the whole file again in the future, potentially causing data-loss in production. There's no visual indication that I could see about this.
3. In the same vein, Notebooks have auto-execute buttons (above/below/all). AFAICT, pressing one of them would mean code in my RCF will get executed, again with catastrophic results when connected to a production REPL.
4. Making each top-level expression inside a RCF its own cell can be a bit too fine-grained. A lot of my RCF expressions are a few defs where I grab a database connection, setup a few IDs, perhaps require something etc. Then I start the exploration, where I constantly edit an expression, review the results, perhaps then def the results and move on.
5. I was glad to find that I can evaluate code inside a Notebook cell, using my usual cmd-enter shortcut, as if I had it open in a plain editor. However, the output of the evaluation is still inline, i.e. it doesn't get rendered in a mini-portal.

I have a few ideas about how the UI could work differently, perhaps with a few specific metadata (e.g. annotate namespaces or comment blocks with something like ^:calva.notebook/cell etc), or perhaps by having cells be bigger (the entire RCF), where you still execute the expressions manually (via the usual keyboard shortcuts) but the output appears in the mini-portal below the cell - but at this point it's just random thoughts and nothing concrete.

In any case, many thanks to everyone involved in this, it has the feeling of something really big in the Clojure tooling space!

[Cyrik]

1 + 2: I'm also not super happy that I can't somehow mark the RCF by coloring its cells or somehow grouping them, but sadly notebooks don't allow for this right now. I should be able to put markdown blocks around it that indicate that this is a grouped cell though.

4. This is a bit difficult since running a comment would produce nothing 😄
5. This is on purpose, so you can run parts of it if you want, but the main button still sends it to the cell output. There's probably also a shortcut for that.

I did not really think about multiple RCFs being a major thing when I designed this.
In your case, it would probably make sense to have these in separate namespaces until we can come up with a better UI. This will give you the same possibility as something like ^:calva.notebook/cell.

[orestis]

Thanks @Cyrik for your reply. I'm curious to hear from other people who might have a use-case for this. In my case, perhaps for now I will be creating a new ephemeral namespace for notebook-based exploration, connected to a production REPL, for now.

[seancorfield]

FWIW, my workflow, when I connect to QA or production for some debugging is essentially the same (steps 1-5 above), so the RCF issue is a big obstacle for me to use the Notebook format.

In development, it's less of a worry since my RCFs wouldn't be too problematic if they accidentally got evaluated (in general -- but not always!). I also have multiple RCFs in a file quite often.

Currently, for debugging QA/production, I'm starting a headless Portal web server in one of our processes (and an nREPL server), then setting up two ssh tunnels (one for nREPL, one for Portal), and then opening a Simple Browser in VS Code pointing at the remote web server and connecting Calva to the remote nREPL.

[orestis]

@seancorfield by the RCF issue do you mean the danger of accidentally evaluating the inner-expressions of RCFs? Or the expansion of one top-level RCF into multiple notebook cells?

[seancorfield]

The former. You can mitigate the latter by grouping expressions in a (do ..) inside the RCF.

[orestis]

Good to know there's more use-cases. Regarding the (do ...), it's always an option for manual grouping, but you need to do it before you open the namespace in the Notebook editor. At least, I can see the "join cell with above" but it doesn't get persisted (I see that there's some whitespace changes in the text file).

[orestis]

VSCode seems to be exposing two kinds of UI for cells:

• Slide-type (presumably to make the whole notebook a slideshow)
• Cell tags (https://twitter.com/code/status/1578086012462432263?s=20&t=A-2aZiCS2VEZoAO0zNGfKg)

Currently the Calva notebooks show the UI but do nothing, but perhaps it could be one possible solution for cell customisation (the cell tag could be stored as metadata on the expression).