Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

📄 Text Transform #6

Draft
wants to merge 15 commits into
base: main
Choose a base branch
from
Draft

📄 Text Transform #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
79cf209
Transform Markdown data back to text
zampino Sep 2, 2022
ae09abe
Add notebook
zampino Sep 28, 2022
10d01b4
Fix sidenotes / Add test
zampino Sep 28, 2022
4f97dd7
Transform tables
zampino Sep 28, 2022
c7ce7f6
Add transformation of a Markdown notebook to Clerk clojure namespace
zampino Sep 29, 2022
617716c
Better title
zampino Sep 29, 2022
273bfc3
Merge remote-tracking branch 'origin/main' into text-transform
zampino Feb 1, 2023
6c97b5e
Merge remote-tracking branch 'origin/main' into text-transform
zampino Mar 10, 2023
22758ba
Bump Clerk
zampino Mar 10, 2023
deaa4b0
Reorg / improve tests
zampino Mar 10, 2023
ca3d8a0
Bump graal JS
zampino Apr 14, 2023
3b152ae
Keep markup from markdown tokens
zampino Apr 14, 2023
eb7f8a2
A first step toward preserving markup
zampino Apr 14, 2023
bc356f5
Support hard line breaks
zampino Apr 14, 2023
c1b60ca
Typing
zampino Feb 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,14 +93,14 @@ We've built hiccup transformation in for convenience, but the same approach can
This library is one of the building blocks of [Clerk](https://github.com/nextjournal/clerk) where it is used for rendering _literate fragments_.

```clojure
^{:nextjournal.clerk/viewer :markdown}
^{:nextjournal.clerk/viewer 'nextjournal.clerk.viewer/markdown-viewer}
data
```

The transformation of markdown node types can be customised like this:

```clojure
^{:nextjournal.clerk/viewer :html}
^{:nextjournal.clerk/viewer 'nextjournal.clerk.viewer/html-viewer}
(md.transform/->hiccup
(assoc md.transform/default-hiccup-renderers
;; :doc specify a custom container for the whole doc
Expand Down
7 changes: 4 additions & 3 deletions deps.edn
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
{:paths ["src" "resources"]
:deps {applied-science/js-interop {:mvn/version "0.3.3"}
org.clojure/data.json {:mvn/version "2.4.0"}
org.graalvm.js/js {:mvn/version "21.3.2.1"}}
org.graalvm.js/js {:mvn/version "22.3.1"}}

:aliases
{:nextjournal/clerk
{:extra-paths ["notebooks"]
:extra-deps {io.github.nextjournal/clerk {:git/sha "60f2399dcdaf39f5d7bc2c213a8dde9a6e0081a9"
:extra-deps {io.github.nextjournal/clerk {:git/sha "711a1b2fae3c212d2c8c2323cf4d53c178766114"
:exclusions [io.github.nextjournal/markdown]}}
:jvm-opts ["-Dclojure.main.report=stderr"
"-Dclerk.resource_manifest={\"/js/viewer.js\" \"js/viewer.js\"}"] ;;
Expand All @@ -18,6 +18,7 @@
"notebooks/pandoc.clj"
"notebooks/parsing_extensibility.clj"
"notebooks/benchmarks.clj"
"notebooks/clerk_to_markdown.clj"
"notebooks/tight_lists.clj"]}}

:quiet
Expand All @@ -39,7 +40,7 @@
:main-opts ["-m" "shadow.cljs.devtools.cli"]
:jvm-opts ["-Dclerk.resource_manifest={\"/js/viewer.js\" \"http://localhost:8021/viewer.js\"}"]
:extra-deps {io.github.nextjournal/clerk.render {:git/url "https://github.com/nextjournal/clerk"
:git/sha "bc137cb51ce98236f48d8dfcb420f6e9df3342c1"
:git/sha "711a1b2fae3c212d2c8c2323cf4d53c178766114"
:deps/root "render"}}}

:build
Expand Down
49 changes: 49 additions & 0 deletions notebooks/clerk_to_markdown.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
;; # 🔄 Clerk to Markdown and Back
(ns clerk_to_markdown
{:nextjournal.clerk/no-cache true}
(:require [clojure.string :as str]
[nextjournal.clerk :as clerk]
[nextjournal.clerk.parser :as clerk.parser]
[nextjournal.markdown.transform :as md.transform]))

;; This notebook shows how to transform a Clojure namespace with Clerk style _literal fragments_ into a [markdown](https://daringfireball.net/projects/markdown/) file.

^{::clerk/visibility {:result :hide}}
(def this-notebook "notebooks/clerk_to_markdown.clj")

;; A clerk notebook is composed of blocks, of type `:markdown` and `:code`. With recent additions to the markdown library we can turn our markdown AST data back into markdown text.

;; This function turns a Clerk block into a markdown string

^{::clerk/visibility {:result :hide}}
(defn block->md [{:as block :keys [type text doc]}]
(case type
:code (str "```clojure\n" text "\n```\n\n")
:markdown (md.transform/->md doc)))

;; now, to put everything together, parse this notebook with Clerk and emit markdown as follows.

^{::clerk/viewer '(fn [s _] (nextjournal.clerk.viewer/html [:pre s]))}
(def as-markdown
(->> this-notebook
(clerk.parser/parse-file {:doc? true})
:blocks
(map block->md)
(apply str)))

;; We can go back to a clojure namespace as follows
(defn block->clj [{:as block :keys [type text doc]}]
(case type
:code (str "\n" text "\n")
:markdown (apply str
(interleave (repeat ";; ")
(str/split-lines (md.transform/->md doc))
(repeat "\n")))))

^{::clerk/viewer '(fn [s _] (nextjournal.clerk.viewer/html [:pre s]))}
(->> (clerk.parser/parse-markdown-string {:doc? true} as-markdown)
:blocks
(map block->clj)
(apply str))

;; To learn more about clerk visit our [github page](https://github.com/nextjournal/clerk).
8 changes: 4 additions & 4 deletions notebooks/try.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@
^{::clerk/width :full
::clerk/visibility {:code :fold}
::clerk/viewer {:render-fn '(fn [_]
(v/html
(reagent/with-let
(nextjournal.clerk.viewer/html
(reagent.core/with-let
[init-text "# 👋 Hello Markdown

```clojure id=xxyyzzww
Expand All @@ -19,13 +19,13 @@
- [ ] _stuff_ here"
text->state (fn [text] (as-> (md/parse text) parsed {:parsed parsed
:hiccup (md.transform/->hiccup md.demo/renderers parsed)}))
!state (reagent/atom (text->state init-text))
!state (reagent.core/atom (text->state init-text))
text-update! (fn [text] (reset! !state (text->state text)))]
[:div.grid.grid-cols-2.m-10
[:div.m-2.p-2.text-xl.border-2.overflow-y-scroll.bg-slate-100 {:style {:height "20rem"}} [md.demo/editor {:doc-update text-update! :doc init-text}]]
[:div.m-2.p-2.font-medium.overflow-y-scroll {:style {:height "20rem"}} [md.demo/inspect-expanded (:parsed @!state)]]
[:div.m-2.p-2.overflow-x-scroll [md.demo/inspect-expanded (:hiccup @!state)]]
[:div.m-2.p-2.bg-slate-50.viewer-markdown [v/html (:hiccup @!state)]]])))}}
[:div.m-2.p-2.bg-slate-50.viewer-markdown [nextjournal.clerk.viewer/html (:hiccup @!state)]]])))}}
(Object.)


Expand Down
4 changes: 3 additions & 1 deletion src/nextjournal/markdown.cljs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@

(extend-type Token
ILookup
(-lookup [this key] (j/get this key)))
(-lookup
([this key] (j/get this key))
([this key not-found] (j/get this key not-found))))

(def tokenize md/tokenize)

Expand Down
62 changes: 35 additions & 27 deletions src/nextjournal/markdown/parser.cljc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,10 +135,9 @@

;; node constructors
(defn node
[type content attrs top-level]
[type content keyvals]
(cond-> {:type type :content content}
(seq attrs) (assoc :attrs attrs)
(seq top-level) (merge top-level)))
(seq keyvals) (merge keyvals)))

(defn empty-text-node? [{text :text t :type}] (and (= :text t) (empty? text)))

Expand All @@ -158,10 +157,9 @@

(defn open-node
([doc type] (open-node doc type {}))
([doc type attrs] (open-node doc type attrs {}))
([doc type attrs top-level]
([doc type keyvals]
(-> doc
(push-node (node type [] attrs top-level))
(push-node (node type [] keyvals))
(update ::path into [:content -1]))))

;; after closing a node, document ::path will point at it
Expand Down Expand Up @@ -305,7 +303,7 @@ end"
doc)

;; blocks
(defmethod apply-token "heading_open" [doc token] (open-node doc :heading {} {:heading-level (hlevel token)}))
(defmethod apply-token "heading_open" [doc token] (open-node doc :heading (assoc (select-keys token [:markup]) :heading-level (hlevel token))))
(defmethod apply-token "heading_close" [doc {doc-level :level}]
(let [{:as doc ::keys [path]} (close-node doc)
doc' (assign-node-id+emoji doc)
Expand All @@ -320,13 +318,13 @@ end"
(defmethod apply-token "paragraph_open" [doc {:as _token :keys [hidden]}] (open-node doc (if hidden :plain :paragraph)))
(defmethod apply-token "paragraph_close" [doc _token] (close-node doc))

(defmethod apply-token "bullet_list_open" [doc {{:as attrs :keys [has-todos]} :attrs}] (open-node doc (if has-todos :todo-list :bullet-list) attrs))
(defmethod apply-token "bullet_list_open" [doc {:as token :keys [attrs]}] (open-node doc (if (:has-todos attrs) :todo-list :bullet-list) (select-keys token [:attrs])))
(defmethod apply-token "bullet_list_close" [doc _token] (close-node doc))

(defmethod apply-token "ordered_list_open" [doc {:keys [attrs]}] (open-node doc :numbered-list attrs))
(defmethod apply-token "ordered_list_open" [doc token] (open-node doc :numbered-list (select-keys token [:attrs])))
(defmethod apply-token "ordered_list_close" [doc _token] (close-node doc))

(defmethod apply-token "list_item_open" [doc {{:as attrs :keys [todo]} :attrs}] (open-node doc (if todo :todo-item :list-item) attrs))
(defmethod apply-token "list_item_open" [doc {:as token :keys [attrs]}] (open-node doc (if (:todo attrs) :todo-item :list-item) (select-keys token [:attrs :markup])))
(defmethod apply-token "list_item_close" [doc _token] (close-node doc))

(defmethod apply-token "math_block" [doc {text :content}] (push-node doc (block-formula text)))
Expand All @@ -341,15 +339,17 @@ end"
(defmethod apply-token "tocBody" [doc _token] doc) ;; ignore body
(defmethod apply-token "tocClose" [doc _token] (-> doc close-node (update-current dissoc :content)))

(defmethod apply-token "code_block" [doc {:as _token c :content}]
(defmethod apply-token "code_block" [doc {:as token :keys [content]}]
(-> doc
(open-node :code)
(push-node (text-node c))
(open-node :code (select-keys token [:markup]))
(push-node (text-node content))
close-node))
(defmethod apply-token "fence" [doc {:as _token i :info c :content}]
(defmethod apply-token "fence" [doc {:as token :keys [info content]}]
(-> doc
(open-node :code {} (assoc (parse-fence-info i) :info i))
(push-node (text-node c))
(open-node :code (-> (select-keys token [:markup])
(assoc :info info)
(merge (parse-fence-info info))))
(push-node (text-node content))
close-node))

;; footnotes
Expand All @@ -363,7 +363,7 @@ end"
;; consider an offset in case we're parsing multiple inputs into the same context
(let [ref (+ (get-in* token [:meta :id]) footnote-offset)
label (get-in* token [:meta :label])]
(open-node doc :footnote nil (cond-> {:ref ref} label (assoc :label label)))))
(open-node doc :footnote (cond-> {:ref ref} label (assoc :label label)))))

(defmethod apply-token "footnote_close" [doc token] (close-node doc))

Expand All @@ -384,7 +384,7 @@ end"

(defn footnote->sidenote [{:keys [ref label content]}]
;; this assumes the footnote container is a paragraph, won't work for lists
(node :sidenote (-> content first :content) nil (cond-> {:ref ref} label (assoc :label label))))
(node :sidenote (-> content first :content) (cond-> {:ref ref} label (assoc :label label))))

(defn node-with-sidenote-refs [p-node]
(loop [l (->zip p-node) refs []]
Expand Down Expand Up @@ -457,11 +457,11 @@ And what.
(defmethod apply-token "thead_close" [doc _token] (close-node doc))
(defmethod apply-token "tr_open" [doc _token] (open-node doc :table-row))
(defmethod apply-token "tr_close" [doc _token] (close-node doc))
(defmethod apply-token "th_open" [doc token] (open-node doc :table-header (:attrs token)))
(defmethod apply-token "th_open" [doc token] (open-node doc :table-header (select-keys token [:attrs])))
(defmethod apply-token "th_close" [doc _token] (close-node doc))
(defmethod apply-token "tbody_open" [doc _token] (open-node doc :table-body))
(defmethod apply-token "tbody_close" [doc _token] (close-node doc))
(defmethod apply-token "td_open" [doc token] (open-node doc :table-data (:attrs token)))
(defmethod apply-token "td_open" [doc token] (open-node doc :table-data (select-keys token [:attrs])))
(defmethod apply-token "td_close" [doc _token] (close-node doc))

(comment
Expand Down Expand Up @@ -577,18 +577,25 @@ _this #should be a tag_, but this [_actually #foo shouldnt_](/bar/) is not."
(defmethod apply-token "hardbreak" [doc _token] (push-node doc {:type :hardbreak}))

;; images
(defmethod apply-token "image" [doc {:keys [attrs children]}] (-> doc (open-node :image attrs) (apply-tokens children) close-node))
(defmethod apply-token "image" [doc {:as token :keys [children]}]
(-> doc
(open-node :image (select-keys token [:attrs]))
(apply-tokens children)
close-node))

;; marks
(defmethod apply-token "em_open" [doc _token] (open-node doc :em))
(defmethod apply-token "em_open" [doc token] (open-node doc :em (select-keys token [:markup])))
(defmethod apply-token "em_close" [doc _token] (close-node doc))
(defmethod apply-token "strong_open" [doc _token] (open-node doc :strong))
(defmethod apply-token "strong_open" [doc token] (open-node doc :strong (select-keys token [:markup])))
(defmethod apply-token "strong_close" [doc _token] (close-node doc))
(defmethod apply-token "s_open" [doc _token] (open-node doc :strikethrough))
(defmethod apply-token "s_open" [doc token] (open-node doc :strikethrough (select-keys token [:markup])))
(defmethod apply-token "s_close" [doc _token] (close-node doc))
(defmethod apply-token "link_open" [doc token] (open-node doc :link (:attrs token)))
(defmethod apply-token "link_open" [doc token] (open-node doc :link (select-keys token [:attrs])))
(defmethod apply-token "link_close" [doc _token] (close-node doc))
(defmethod apply-token "code_inline" [doc {text :content}] (-> doc (open-node :monospace) (push-node (text-node text)) close-node))
(defmethod apply-token "code_inline" [doc {:as token :keys [content]}]
(-> doc (open-node :monospace (select-keys token [:markup]))
(push-node (text-node content))
close-node))

;; html (ignored)
(defmethod apply-token "html_inline" [doc _] doc)
Expand Down Expand Up @@ -737,5 +744,6 @@ some final par"
(into []
(comp
(mapcat (partial tree-seq (comp seq :children) :children))
(map #(select-keys % [:type :content :hidden :level :info :meta])))
(map (comp #(into {} (filter (comp some? val)) %)
#(select-keys % [:type :content :hidden :level :info :attrs :markup :meta]))))
tokens))
Loading
Loading