Integrate 1.3 and 1.4 LUA/AST changes

docs/advanced/latex-raw.qmd

@@ -0,0 +1,19 @@
+---
+title: Include Quarto Markdown in LaTeX Raw Blocks
+summary: Allows library authors to emit raw blocks that include Quarto features like cross-references or shortcodes.
+---
+
+Use the syntax `\QuartoMarkdownBase64{<<base64-encoded-markdown-content>>}` 
+to include Quarto Markdown in LaTeX raw blocks. 
+When this syntax is detected by Quarto, the contents will be decoded, 
+processed in Quarto (including user filters), and then inserted back into the LaTeX raw block.
+This allows libraries that emit raw blocks to benefit
+from Quarto features such as cross-reference resolution and shortcodes.
+
+This is useful for third-party libraries that seek to emit LaTeX content that nevertheless 
+can have "quarto content". Note that, unlike the [equivalent HTML feature](/docs/authoring/tables.qmd#html-tables), 
+Quarto currently only supports base-64 encoded content in LaTeX blocks.
+
+This LaTeX feature cannot currently be disabled.
+We expect this to not be necessary because `QuartoMarkdownBase64` is unlikely to conflict with
+existing LaTeX environments.

docs/advanced/quarto-ast.qmd

@@ -0,0 +1,165 @@
+---
+title: Quarto AST 
+summary: Quarto extends Pandoc's AST processing to allow more flexible customization in filters
+aliases: 
+  - /docs/prerelease/1.3/ast.html
+  - /docs/prerelease/1.4/lua_changes.html
+---
+
+## Overview
+
+Quarto extends Pandoc's AST processing to allow more flexible customization in filters: 
+
+* [Custom nodes](#custom-nodes): Quarto defines custom AST node types for Quarto specific types of content like Callouts, Tabsets etc. This allows filters to target, modify or create these elements.
+
+* [Custom renderers](#custom-formats-and-custom-renderers): Add custom renderers for Quarto's custom node types to facilitate handling Quarto specific elements in custom formats.
+
+* [Targeting AST Processing Phases](#targeting-of-ast-processing-phases): Apply filters at precise points in the AST processing.
+
+## Custom Nodes
+
+Quarto defines some custom AST nodes for use in Pandoc filters. 
+This allows more flexibility in defining and using Lua filters. 
+For example, by using the `Callout` custom node this filter forces every callout to be of type "caution":
+
+``` lua
+function Callout(callout)
+  -- do something with the callout
+  callout.type = "caution"
+
+  -- note that custom AST nodes are passed by reference. You can
+  -- return the value if you choose, but you do not need to.
+end
+```
+
+Finally, custom AST node constructors are available in the `quarto` object: `quarto.Callout`, `quarto.Tabset`, etc. See the sections below for details.
+
+### Callouts
+
+You can create callout AST nodes in Lua filters with the `quarto.Callout` constructor. The constructor takes a single parameter, a table with entries `type`, `title`, and `content`, as described below. In Lua filters, callouts are represented as a table with the following fields:
+
+- `type`: the type of callout: `note`, `caution`, `warning`, etc (optional in the constructor).
+- `title`: The callout title (if any) (optional in the constructor),
+- `icon`: the callout icon (or `false` if none) (optional in the constructor)
+- `appearance`: `"minimal"`, `"simple"`, or `"default"` (optional in the constructor)
+- `collapse`: whether to render the callout as collapsible (optional in the constructor, default `false`)
+- `content`: the content of the callout (a `pandoc.Blocks` object, or a plain list in the constructor)
+
+### Tabsets
+
+You can create conditional blocks in Lua filters with the `quarto.Tabset` constructor, with parameters `tabs`, `level` and `attr` as described above. In
+addition, you can use `quarto.Tab` to create the tab objects for the `tabs` field. `quarto.Tab` is more lenient with parameter types, converting strings to `Blocks` and `Inlines` as needed. In Lua filters, tabsets are represented as a table with the following fields:
+
+- `tabs`: a table containing the content for each tab. Each entry is a table with two entries: `title` (a `pandoc.Inlines`) and `content` (a `pandoc.Blocks`) (optional in the contructor, default value `{}`)
+- `level`: the level of the tab headings to be used in rendering the tabset (optional in the constructor, default value `2`)
+- `attr`: the `Attr` object for the resulting tabset div (optional in the constructor)
+
+### Conditional Blocks
+
+You can create conditional block AST nodes in Lua filters with the `quarto.ConditionalBlock` constructor. The constructor takes a single parameter, a table with entries `node`, `behavior`, and `condition`, as described below.
+
+In Lua filters, conditional blocks are represented as a table with the following fields:
+
+- `node`: the div containing the content
+- `behavior`: either `content-visible` or `content-hidden`
+- `condition`: a list of 2-element lists, such as `{ { "unless-format", "html" } }` (optional in the constructor, default value `{}`). The first element of each sublist must be one of `when-format`, `unless-format`, `when-profile`, and `unless-profile`. The second element is the relevant format or profile.
+
+### Cross-referenceable Elements
+
+Crossreferenceable elements all have a single generic type, `FloatRefTarget`. 
+This element can be constructed explicitly in a Lua filter.
+It can also be used as the element to be processed in a Lua filter directly.
+
+```{.lua}
+-- A filter targeting FloatRefTarget nodes
+return {
+  FloatRefTarget = function(float)
+    if float.caption_long then
+      float.caption_long.content:insert(pandoc.Str("[This will appear at the beginning of every caption]"))
+      return float
+    end
+  end
+}
+```
+
+`FloatRefTarget` nodes have the following fields:
+
+- `type`: The type of element: `Figure`, `Table`, `Listing`, etc. Quarto v1.4 supports
+  custom node types that can be declared at the document or project level.
+- `content`: The content of the Figure, Table, etc. Quarto v1.4
+  accepts any content in any `FloatRefTarget` type (so if your tables are better displayed
+  as an image, you can use that.).
+- `caption_long`: The caption that appears in the main body of the document
+- `caption_short`: The caption that appears in the element collations (such as a list of tables, 
+  list of figures, etc.)
+- `identifier`, `attributes`, `classes`: these are analogous to `Attr` fields in Pandoc AST elements like spans and divs. 
+  `identifier`, in addition, needs to be the string that is used in a crossref (`fig-cars`, `tbl-votes`, `lst-query`, and so on).
+- `parent_id`: if a `FloatRefTarget` is a subfloat of a parent multiple-element float, then `parent_id` will hold the identifier
+  of the parent float.
+
+## Custom Formats and Custom Renderers
+
+Quarto has support for extensible renderers of quarto AST nodes such as `FloatRefTarget`, `Callout` etc.
+In order to declare a custom renderer, add the following to a Lua filter:
+
+```lua
+local predicate = function(float)
+  -- return true if this renderer should be used;
+  -- typically, this will return true if the custom format is active.
+end
+local renderer = function(float)
+  -- returns a plain Pandoc representation of the rendered figure.
+end
+quarto._quarto.ast.add_renderer(
+  "FloatRefTarget", 
+  predicate, 
+  renderer)
+```
+
+## Targeting of AST Processing Phases
+
+Quarto's AST processing phase is split into three parts: `ast`, `quarto`, and `render`.
+
+- `ast`: normalizes the input syntax from Pandoc, recognizing constructs such as `Callout`, `FloatRefTarget`, and so on.
+- `quarto`: processes the normalized syntax, for example by resolving cross-references.
+- `render`: produces format-specific output from the processed input.
+
+Lua filters can be inserted before or after any of these stages:
+
+```yaml
+filters:
+  - at: pre-ast
+    path: filter1.lua
+  - at: post-quarto
+    path: filter2.lua
+  - at: post-render
+    path: filter3.lua
+```
+
+Any of the stages can be prefixed by `pre-` or `post-`.
+Currently `pre-quarto` and `post-ast` correspond to the same insertion location in the filter chain, as do `post-quarto` and `pre-render`.
+
+You can also use JSON filters with this syntax.
+Either use `type: json` explicitly, or use a path that doesn't end in `.lua`.
+Conversely, `type: lua` will force the file to be treated as a Lua filter.
+
+Prior to Quarto 1.4, Lua filters were either "pre" filters (the default setting), or "post" filters.
+Those filters are specified like this:
+
+```yaml
+# "pre" filters:
+filters:
+  - pre_filter_1.lua
+  - pre_filter_2.lua  
+  # ...
+# "post" filters:
+filters:
+  - quarto
+  - post_filter_1.lua
+  - post_filter_2.lua  
+  # ...
+```
+
+This syntax continues to work.
+"Pre" filters are injected at the `pre-quarto` entry point, and "post" filters are injected at the `post-render` entry point.
+

docs/extensions/filters.qmd

@@ -45,6 +45,9 @@ filters:
   - fontawesome
 ```
 
+Beyond a running filters before or after Quarto's filters, you can also
+precisely target the location in the AST processing. Read more in [Targeting of AST Processing Phases](/docs/advanced/quarto-ast.qmd#targeting-of-ast-processing-phases).
+
 You'll notice that one of the extensions (`spellcheck.lua`) has a file extension and the other (`fontawesome`) does not. This difference stems from how the extensions are distributed: an extension distributed as a plain Lua file uses `.lua` whereas a filter distributed as a [Quarto Extension](index.qmd) does not. The next section explores how to create filters as extensions.
 
 

docs/extensions/lua-api.qmd

@@ -56,11 +56,13 @@ Complete documentation for the Pandoc Lua API can be found in the [Lua Filters](
 
 Various utility functions are provided:
 
-| Function                          | Description                                                                                                                                                                                    |
+| Function                                     | Description                                                                                                                                                                                    |
 |-------------------|-----------------------------------------------------|
-| `quarto.version`                | Return the current Quarto version as a `pandoc.Version` object.                                                                                                                                |
-| `quarto.log.output(obj)`          | Dump a text representation of the passed object to stdout.                                                                                                                                     |
-| `quarto.utils.resolve_path(path)` | Compute the full path to a file that is installed alongside your extension's Lua script. This is useful for *internal* resources that your filter needs but should not be visible to the user. |
+| `quarto.version`                             | Return the current Quarto version as a `pandoc.Version` object.                                                                                                                                |
+| `quarto.log.output(obj)`                     | Dump a text representation of the passed object to stdout.                                                                                                                                     |
+| `quarto.utils.resolve_path(path)`            | Compute the full path to a file that is installed alongside your extension's Lua script. This is useful for *internal* resources that your filter needs but should not be visible to the user. |
+| `quarto.utils.string_to_inlines(path, sep)`  | Converts a string to a list of Pandoc Inlines, processing any Quarto custom syntax in the string.                                                                                              |
+| `quarto.utils.string_to_blocks(path)`        | Converts a string to a list of Pandoc Blocks, processing any Quarto custom syntax in the string.                                                                                               |
 
 Quarto includes the [pandoc-lua-logging](https://github.com/wlupton/pandoc-lua-logging) library, which should be used in preference to the dump function. For example, you can examine an element passed to a filter function as follows:
 
@@ -70,6 +72,61 @@ function Div(el)
 end
 ```
 
+### `require()`
+
+In larger, more complex filters, it becomes useful to structure your Lua code in modules.
+Quarto overwrites the standard LUA `require()` to support the use of relative paths, 
+so that small modules can be easily created and reused.
+
+For example:
+
+```{.lua filename="filter.lua"}
+local utility = require('./utils')
+function Pandoc(doc)
+  -- process 
+end
+```
+
+Using relative paths makes it harder for multiple filters to accidentally
+create conflicting module names (as would eventually happen when using the standard Lua
+`require('utils')` syntax). It's possible to refer to subdirectories and parent directories as well:
+
+```{.lua filename="filter2.lua"}
+local parsing = require('./utils/parsing')
+function Pandoc(doc)
+  -- process 
+end
+```
+
+```{.lua filename="utils/parsing.lua"}
+local utils = require("../utils")
+-- ...
+```
+
+### Document Metadata
+
+Extensions may need to access the path for the input and output files. 
+Quarto makes these variables available:
+
+| Function                 | Description                                      |
+|--------------------------|--------------------------------------------------|
+| `quarto.doc.input_file`  | Full path to input file for the current render.  |
+| `quarto.doc.output_file` | Full path to output file for the current render. |
+
+### Project Metadata
+
+Extensions may need to know about paths relevant to the project.
+Quarto makes these variables available:
+
+
+| Function                          | Description                                                                                                     |
+|-----------------------------------|-----------------------------------------------------------------------------------------------------------------|        
+| `quarto.project.directory`        | Full path to current project directory (`nil` if no project).                                                   |
+| `quarto.project.output_directory` | Full path to current project output directory (`nil` if no project).                                            |
+| `quarto.project.offset`           | Offset (relative path) from the directory of the current file to the root of the project (`nil` if no project). |
+| `quarto.project.profile`          | List of currently active project profiles.                                                                      |
+
+
 ### Format Detection
 
 Extensions will often need to detect the current format to create custom content depending on the target output medium. The `quarto.doc.is_format()` function
@@ -135,6 +192,8 @@ Extensions will sometimes want to add external dependencies (for example, a Java
 | `quarto.doc.attach_to_dependency(name, attach)` | Attach a file to an existing dependency. `attach` is a file path relative to the Lua filter or table with \`path\` and \`name\` for renaming the file as its copied.                                                                                      |
 | `quarto.doc.use_latex_package(pkg, opt)`        | Adds a `\usepackage` statement to the LaTeX output (along an options string specified in `opt`)                                                                                                                                                           |
 | `quarto.doc.add_format_resource(path)`          | Add a format resource to the document. Format resources will be copied into the directory next to the rendered output. This is useful, for example, if your format references a `bst` or `cls` file which must be copied into the LaTeX output directory. |
+| `quarto.doc.add_resource(path)`                 | Add a resource file to the current render, copying that file to the same relative location in the output directory.                                                                                                                                       | 
+| `quarto.doc.add_supporting(path)`               | Add a supporting file to the current render, moving that file file to the same relative location in the output directory.                                                                                                                                 |
 
 For example, here we add a LaTeX package dependency:
 
@@ -224,6 +283,17 @@ quarto.doc.add_html_dependency({
 })
 ```
 
+### Custom Nodes
+
+Quarto adds some custom AST node types. You can read more about them in [Quarto AST](/docs/advanced/quarto-ast.qmd#custom-nodes). You can create them with the following constructors:
+
+| Function                        | Node                                                                                                              |
+|--------------------------------|-----------------------------------------|
+| `quarto.Callout(tbl)`          | Callout: `tbl` is a table with entries `type`, `title`, and `content`.                                    |
+| `quarto.Tabset(tbl)`           | Tabset: `tbl` is a table with entries `tabs`, `level` and `attr`.                                         |
+| `quarto.ConditionalBlock(tbl)` | Conditional block: `tbl` is a table with entries `node`, `behavior`, and `condition`.                     |
+| `quarto.FloatRefTarget(tbl)`   | Cross-referencable element: `tbl` is a table with entries `content`, `caption_long`, and `caption_short`. |
+
 ### JSON Encoding
 
 Quarto includes a copy of [json.lua](https://github.com/rxi/json.lua). a lightweight JSON library for Lua. You can access the JSON functions as follows:

docs/prerelease/1.3/ast.qmd

@@ -1,6 +1,6 @@
 ---
 title: Custom AST Nodes
-search: true
+search: false
 ---
 
 {{< include /docs/_require-1.3.qmd >}}

docs/prerelease/1.4/ast.qmd

@@ -1,5 +1,6 @@
 ---
 title: "AST processing changes in v1.4"
+search: false
 ---
 
 In Quarto v1.3, we added support for parsing HTML tables as native Pandoc elements, so that sophisticated table layouts are available in more formats. Quarto v1.4 extends this in a few ways.

docs/prerelease/1.4/lua_changes.qmd

@@ -1,5 +1,6 @@
 ---
 title: "Lua filter changes"
+search: false
 ---
 
 {{< include /docs/_require-1.4.qmd >}}