tutorial/tutorial-preview.html

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
   <head>
      <meta http-equiv="Content-Type" content="application/xml; charset=UTF-8">
      <title>TUTORIAL PREVIEW</title>
      <style xml:space="preserve" type="text/css">
.toc { font-size: 70%; padding: 0.4em; outline: thin solid black; margin: 0.4em 0em; width: fit-content; counter-reset: lessonNo 0; font-family: sans-serif }
.toc * { margin: 0em; font-weight: normal }
.toc div { margin: 0.2em; margin-left: 1em; outline: thin solid grey; height: fit-content }
.toc .lesson { display: grid; grid-template-columns: 1fr repeat(3,3fr); counter-increment: lessonNo 1; padding: 0.8em }
.toc .lesson:nth-child(even) { background-color: lightsteelblue }
.toc div.lesson:before { content: attr(class) ' ' counter(lessonNo) ': ' attr(name); background-color: lavender; color: midnightblue; padding: 0.2em; font-family: sans-serif; display: inline-block; height: fit-content }
.toc .unit { clamp(12vw, 100%, 24vw) }
section section section { margin: 0.2em; margin-left: 1em; padding-left: 0.6em; border-left: medium solid grey }

table { width: 80vw; resize: horizontal; padding: 0.8em; background-color: whitesmoke; position: relative; border: thin solid grey; overflow: auto; display: inherit }
tr:nth-child(even) { background-color: gainsboro }


th { width: clamp(10em, auto, 40em) }
td { width: clamp(10em, auto, 40em); border-top: thin solid grey }

section.unit   { width: clamp(45ch, 100%, 75ch); padding: 0.8em; outline: thin solid black; margin: 0.6em 0em }
section.unit h1:first-child { margin-top: 0em }
.observer { background-color: honeydew ; grid-column: 2 }
.maker    { background-color: seashell ; grid-column: 3 }
.learner  { background-color: aliceblue; grid-column: 4 }                 
           
span.wordcount { font-size: smaller; font-weight: bolder; font-style: italic; break-inside: avoid }
span.wordcount.over { color: darkred }

</style>
   </head>
   <body>
      <h1>XProc in Action: an Immersive Introduction</h1>
      <h3>version 1.0_draft PREVIEW December 20 2024</h3>
      <div class="toc">
         <div class="lesson"
              id="toc-acquire"
              name="acquire">
            <div class="unit observer"
                 id="toc-acquire_101"
                 data-track="observer">
               <h1>101: Project setup and installation<span class="wordcount okay"> (~1714)</span>
               </h1>
               <div>
                  <h2>Goals</h2>
               </div>
               <div>
                  <h2>Prerequisites</h2>
               </div>
               <div>
                  <h2>Resources</h2>
               </div>
               <div>
                  <h2>Step One: Setup</h2>
                  <div>
                     <h3>Shortcut</h3>
                  </div>
               </div>
               <div>
                  <h2>Step Two: Confirm</h2>
               </div>
               <div>
                  <h2>Comments / review</h2>
                  <div>
                     <h3>When running from a command line</h3>
                  </div>
               </div>
            </div>
            <div class="unit maker"
                 id="toc-acquire_102"
                 data-track="maker">
               <h1>102: Examining the setup<span class="wordcount okay"> (~885)</span>
               </h1>
               <div>
                  <h2>Goals</h2>
               </div>
               <div>
                  <h2>Prerequisites</h2>
               </div>
               <div>
                  <h2>Resources</h2>
               </div>
               <div>
                  <h2>Step One: Inspect the pipelines</h2>
               </div>
               <div>
                  <h2>Step Two: Modify the pipelines</h2>
               </div>
               <div>
                  <h2>For consideration</h2>
               </div>
            </div>
            <div class="unit learner"
                 id="toc-acquire_599"
                 data-track="learner">
               <h1>599: Meeting XProc<span class="wordcount okay"> (~698)</span>
               </h1>
               <div>
                  <h2>Goals</h2>
               </div>
               <div>
                  <h2>Resources</h2>
               </div>
               <div>
                  <h2>Some observations</h2>
               </div>
               <div>
                  <h2>Declarative markup in action</h2>
                  <div>
                     <h3>Standards for documents</h3>
                  </div>
               </div>
            </div>
         </div>
         <div class="lesson"
              id="toc-walkthrough"
              name="walkthrough">
            <div class="unit observer"
                 id="toc-walkthrough_101"
                 data-track="observer">
               <h1>101: Unpacking XProc 3.0<span class="wordcount over"> (~2752)</span>
               </h1>
               <div>
                  <h2>Goals</h2>
               </div>
               <div>
                  <h2>Prerequisites</h2>
               </div>
               <div>
                  <h2>Resources</h2>
                  <div>
                     <h3>For reference</h3>
                  </div>
               </div>
               <div>
                  <h2>A closer look</h2>
               </div>
               <div>
                  <h2>Survey</h2>
                  <div>
                     <h3>
                        <a href="../smoketest/TEST-XPROC3.xpl">TEST-XPROC3</a>
                     </h3>
                  </div>
                  <div>
                     <h3>
                        <a href="../smoketest/TEST-XSLT.xpl">TEST-XSLT</a>
                     </h3>
                  </div>
                  <div>
                     <h3>
                        <a href="../smoketest/TEST-SCHEMATRON.xpl">TEST-SCHEMATRON</a>
                     </h3>
                  </div>
                  <div>
                     <h3>
                        <a href="../smoketest/TEST-XSPEC.xpl">TEST-XSPEC</a>
                     </h3>
                  </div>
               </div>
               <div>
                  <h2>A not-so-simple pipeline</h2>
                  <div>
                     <h3>PRODUCE-PROJECTS-ELEMENTLIST</h3>
                  </div>
               </div>
               <div>
                  <h2>Respecting XML syntax, XPath and XProc</h2>
               </div>
               <div>
                  <h2>Learning more about XProc</h2>
               </div>
            </div>
            <div class="unit maker"
                 id="toc-walkthrough_102"
                 data-track="maker">
               <h1>102: XProc fundamentals<span class="wordcount okay"> (~1526)</span>
               </h1>
               <div>
                  <h2>Goals</h2>
               </div>
               <div>
                  <h2>Prerequisites</h2>
               </div>
               <div>
                  <h2>Resources</h2>
               </div>
               <div>
                  <h2>Learning more about XProc</h2>
               </div>
               <div>
                  <h2>Details details!</h2>
                  <div>
                     <h3>TEST-XSPEC</h3>
                  </div>
                  <div>
                     <h3>PRODUCE-PROJECTS-ELEMENTLIST</h3>
                  </div>
               </div>
               <div>
                  <h2>Messing around</h2>
                  <div>
                     <h3>Disabling your code</h3>
                  </div>
               </div>
               <div>
                  <h2>Take note</h2>
                  <div>
                     <h3>Where are these downloads coming from?</h3>
                  </div>
                  <div>
                     <h3>Syntax tips</h3>
                  </div>
               </div>
            </div>
            <div class="unit learner"
                 id="toc-walkthrough_219"
                 data-track="learner">
               <h1>219: XProc, XML and XDM  (the XML Data Model)<span class="wordcount over"> (~2214)</span>
               </h1>
               <div>
                  <h2>Goals</h2>
               </div>
               <div>
                  <h2>Resources</h2>
               </div>
               <div>
                  <h2>XProc as XML</h2>
                  <div>
                     <h3>Survey of XProc elements</h3>
                  </div>
               </div>
               <div>
                  <h2>XML and the XML Data Model (XDM): context and rationale</h2>
               </div>
               <div>
                  <h2>Snapshot history: an XML time line</h2>
               </div>
               <div>
                  <h2>XPath</h2>
                  <div>
                     <h3>Documents and data</h3>
                  </div>
                  <div>
                     <h3>XPath illustrative examples</h3>
                  </div>
               </div>
               <div>
                  <h2>Exercise: Discussion board</h2>
               </div>
            </div>
            <div class="unit observer"
                 id="toc-walkthrough_301"
                 data-track="observer">
               <h1>301: Automated XProc<span class="wordcount okay"> (~954)</span>
               </h1>
               <div>
                  <h2>Goals</h2>
               </div>
               <div>
                  <h2>Prerequisites</h2>
               </div>
               <div>
                  <h2>Resources</h2>
               </div>
               <div>
                  <h2>XProc for quality testing</h2>
                  <div>
                     <h3>Pipelines useful for the developer:</h3>
                  </div>
                  <div>
                     <h3>Pipelines run under CI/CD</h3>
                  </div>
                  <div>
                     <h3>File set listings as step declarations</h3>
                  </div>
                  <div>
                     <h3>About the XProc House Rules</h3>
                  </div>
                  <div>
                     <h3>About XSpec testing in this repository</h3>
                  </div>
               </div>
               <div>
                  <h2>XProc running under continuous integration and development (CI/CD)</h2>
               </div>
            </div>
            <div class="unit learner"
                 id="toc-walkthrough_401"
                 data-track="learner">
               <h1>401: The XSLT Factor<span class="wordcount over"> (~2667)</span>
               </h1>
               <div>
                  <h2>Goals</h2>
               </div>
               <div>
                  <h2>Prerequisites</h2>
               </div>
               <div>
                  <h2>Resources</h2>
                  <div>
                     <h3>XSLT 1.0 and XPath 1.0</h3>
                  </div>
                  <div>
                     <h3>XSLT 2.0 and XQuery 1.0</h3>
                  </div>
                  <div>
                     <h3>XSLT 3.0, XQuery 3.0, XPath 3.1</h3>
                  </div>
               </div>
               <div>
                  <h2>XSLT: XSL (XML Stylesheet Language) Transformations</h2>
                  <div>
                     <h3>Reflecting on XSLT</h3>
                  </div>
                  <div>
                     <h3>Running XSLT without XProc</h3>
                  </div>
               </div>
               <div>
                  <h2>Using XSLT in XProc: avoiding annoyances</h2>
                  <div>
                     <h3>Namespaces in and for your XSLT</h3>
                  </div>
                  <div>
                     <h3>Text and attribute value syntax in embedded XSLT</h3>
                  </div>
               </div>
               <div>
                  <h2>Learning XSLT the safer way</h2>
               </div>
               <div>
                  <h2>XProc without XSLT?</h2>
               </div>
               <div>
                  <h2>XProc, XDM (the XML data model) and the standards stack</h2>
               </div>
            </div>
         </div>
         <div class="lesson"
              id="toc-oscal-convert"
              name="oscal-convert">
            <div class="unit observer"
                 id="toc-oscal-convert_101"
                 data-track="observer">
               <h1>101: Converting OSCAL – XML to JSON and JSON to XML<span class="wordcount over"> (~2170)</span>
               </h1>
               <div>
                  <h2>Goals</h2>
               </div>
               <div>
                  <h2>Prerequisites</h2>
               </div>
               <div>
                  <h2>Resources</h2>
               </div>
               <div>
                  <h2>Pipeline rundown</h2>
                  <div>
                     <h3>
                        <a href="../projects/oscal-convert/GRAB-RESOURCES.xpl">GRAB-RESOURCES</a>
                     </h3>
                  </div>
                  <div>
                     <h3>
                        <a href="../projects/oscal-convert/BATCH_JSON-TO-XML.xpl">BATCH-JSON-TO-XML</a>
                     </h3>
                  </div>
                  <div>
                     <h3>
                        <a href="../projects/oscal-convert/BATCH_XML-TO-JSON.xpl">BATCH-XML-TO-JSON</a>
                     </h3>
                  </div>
                  <div>
                     <h3>
                        <a href="../projects/oscal-convert/CONVERT-OSCAL-XML-DATA.xpl">CONVERT-OSCAL-XML-DATA</a>
                     </h3>
                  </div>
                  <div>
                     <h3>
                        <a href="../projects/oscal-convert/CONVERT-OSCAL-XML-FOLDER.xpl">CONVERT-OSCAL-XML-FOLDER</a>
                     </h3>
                  </div>
               </div>
               <div>
                  <h2>Working concept: return trip</h2>
               </div>
               <div>
                  <h2>What is this XSLT?</h2>
               </div>
               <div>
                  <h2>What could possibly go wrong?</h2>
                  <div>
                     <h3>The playing field is the Internet</h3>
                  </div>
               </div>
               <div>
                  <h2>More catalogs needed!</h2>
               </div>
            </div>
            <div class="unit maker"
                 id="toc-oscal-convert_102"
                 data-track="maker">
               <h1>102: Hands on data conversions<span class="wordcount over"> (~3348)</span>
               </h1>
               <div>
                  <h2>Goals</h2>
               </div>
               <div>
                  <h2>Prerequisites</h2>
               </div>
               <div>
                  <h2>Resources</h2>
               </div>
               <div>
                  <h2>Some breaking and making</h2>
               </div>
               <div>
                  <h2>Value templates in attributes and text: { XPath-expr }</h2>
               </div>
               <div>
                  <h2>Designating inputs</h2>
                  <div>
                     <h3>Lightening the <code>p:load</code>
                     </h3>
                  </div>
               </div>
               <div>
                  <h2>Warning: do you know where your source files are?</h2>
               </div>
               <div>
                  <h2>Probing error space – data conversions</h2>
                  <div>
                     <h3>Converting broken XML or JSON</h3>
                  </div>
                  <div>
                     <h3>Converting not-OSCAL</h3>
                  </div>
                  <div>
                     <h3>Converting broken OSCAL</h3>
                  </div>
               </div>
               <div>
                  <h2>XProc diagnostic how-to</h2>
                  <div>
                     <h3>Emitting runtime messages</h3>
                  </div>
                  <div>
                     <h3>Saving out interim results</h3>
                  </div>
               </div>
               <div>
                  <h2>Validate early and often</h2>
               </div>
            </div>
            <div class="unit learner"
                 id="toc-oscal-convert_201"
                 data-track="learner">
               <h1>201: Anatomy of an XProc pipeline<span class="wordcount over"> (~2779)</span>
               </h1>
               <div>
                  <h2>Goals</h2>
               </div>
               <div>
                  <h2>Prerequisites</h2>
               </div>
               <div>
                  <h2>Resources</h2>
               </div>
               <div>
                  <h2>XProc as XML (redux)</h2>
                  <div>
                     <h3>XProc at the top</h3>
                  </div>
                  <div>
                     <h3>Namespaces</h3>
                  </div>
                  <div>
                     <h3>@name and @type</h3>
                  </div>
               </div>
               <div>
                  <h2>Prologue and body</h2>
               </div>
               <div>
                  <h2>XProc steps</h2>
               </div>
               <div>
                  <h2>Atomic and compound steps</h2>
               </div>
               <div>
                  <h2>Namespaces and extension steps</h2>
               </div>
               <div>
                  <h2>Schema for XProc 3.0</h2>
               </div>
            </div>
            <div class="unit maker"
                 id="toc-oscal-convert_350"
                 data-track="maker">
               <h1>350: Namespaces in XML and XProc<span class="wordcount okay"> (~724)</span>
               </h1>
               <div>
                  <h2>Goals</h2>
               </div>
               <div>
                  <h2>Prerequisites</h2>
               </div>
               <div>
                  <h2>Resources</h2>
               </div>
               <div>
                  <h2>XML and namespaces</h2>
               </div>
               <div>
                  <h2>Namespace fixup and namespace cleanup steps</h2>
               </div>
               <div>
                  <h2>Namespace tips and tricks</h2>
                  <div>
                     <h3>Coining new namespaces</h3>
                  </div>
                  <div>
                     <h3>On-the-fly namespace declarations</h3>
                  </div>
                  <div>
                     <h3>Overloading prefixes</h3>
                  </div>
                  <div>
                     <h3>Matching with namespace wildcard</h3>
                  </div>
               </div>
            </div>
            <div class="unit learner"
                 id="toc-oscal-convert_400"
                 data-track="learner">
               <h1>400: Documents in XProc<span class="wordcount okay"> (~1750)</span>
               </h1>
               <div>
                  <h2>Goals</h2>
               </div>
               <div>
                  <h2>Resources</h2>
               </div>
               <div>
                  <h2>What is an XProc document</h2>
               </div>
               <div>
                  <h2>A universe of content types</h2>
                  <div>
                     <h3>XML and XML-like content</h3>
                  </div>
                  <div>
                     <h3>JSON and other text-based formats</h3>
                  </div>
                  <div>
                     <h3>Binaries and what-have-you</h3>
                  </div>
               </div>
            </div>
            <div class="unit maker"
                 id="toc-oscal-convert_401"
                 data-track="maker">
               <h1>401: XProc, XML, JSON and content types<span class="wordcount okay"> (~274)</span>
               </h1>
               <div>
                  <h2>Goals</h2>
               </div>
               <div>
                  <h2>Resources</h2>
               </div>
               <div>
                  <h2>Exercise some options</h2>
               </div>
            </div>
         </div>
         <div class="lesson"
              id="toc-courseware"
              name="courseware">
            <div class="unit observer"
                 id="toc-courseware_101"
                 data-track="observer">
               <h1>Courseware 101: Producing this tutorial<span class="wordcount okay"> (~433)</span>
               </h1>
               <div>
                  <h2>Goals</h2>
               </div>
               <div>
                  <h2>Prerequisites</h2>
               </div>
               <div>
                  <h2>Resources</h2>
               </div>
               <div>
                  <h2>Tutorial production pipelines</h2>
                  <div>
                     <h3>
                        <a href="../tutorial/PRODUCE-TUTORIAL-PREVIEW.xpl">PRODUCE-TUTORIAL-PREVIEW</a>
                     </h3>
                  </div>
                  <div>
                     <h3>
                        <a href="../tutorial/PRODUCE-TUTORIAL-MARKDOWN.xpl">PRODUCE-TUTORIAL-MARKDOWN</a>
                     </h3>
                  </div>
                  <div>
                     <h3>
                        <a href="../tutorial/PRODUCE-TUTORIAL-TOC.xpl">PRODUCE-TUTORIAL-TOC</a>
                     </h3>
                  </div>
                  <div>
                     <h3>
                        <a href="../tutorial/PRODUCE-PROJECTS-ELEMENTLIST.xpl">PRODUCE-PROJECTS-ELEMENTLIST</a>
                     </h3>
                  </div>
               </div>
            </div>
            <div class="unit maker"
                 id="toc-courseware_219"
                 data-track="maker">
               <h1>Courseware 219: Learn by Teaching<span class="wordcount okay"> (~754)</span>
               </h1>
               <div>
                  <h2>Goals</h2>
               </div>
               <div>
                  <h2>Prerequisites</h2>
               </div>
               <div>
                  <h2>Resources</h2>
               </div>
               <div>
                  <h2>Improve or enhance a lesson or lesson unit</h2>
                  <div>
                     <h3>Apply Schematron to your edits</h3>
                  </div>
               </div>
               <div>
                  <h2>Create a new set of lessons</h2>
               </div>
               <div>
                  <h2>Produce a new project and document it with a tutorial</h2>
               </div>
            </div>
         </div>
      </div>
      <section class="lesson"
               id="acquire"
               name="acquire">
         <section class="unit observer"
                  id="acquire_101"
                  data-track="observer">
            <h1>101: Project setup and installation</h1>
            <section>
               <h2>Goals</h2>
               <p>Set up and run an XProc 3.0 pipeline in an XProc 3.0 engine.</p>
               <p>Get some results. See them in the console (message tracebacks), the file system (new files acquired or
            produced), or both.</p>
               <p>With a little practice, become comfortable running XProc pipelines.</p>
               <p>After the first script to get the XProc engine, we use XProc for subsequent downloads. Finishing the setup
            gets you started practicing with the pipelines.</p>
            </section>
            <section>
               <h2>Prerequisites</h2>
               <p>If you have not done so, scan the <a href="../tutorial/readme.md">tutorial readme file</a> for some helpful
            background and ideas on how tutorial materials are arranged.</p>
               <p>If ready to proceed, <b>you have a system with Java installed</b> offering a JVM (Java Virtual Machine)
            available on the command line (a JRE or JDK), version 8 (and later).</p>
               <p>
                  <b>Tip:</b> check your Java version from the console using <code>java --version</code>.</p>
               <p>Also, <b>you have an Internet connection available</b> and the capability to download and save resources
            (binaries and code libraries) for local use. (There are no runtime dependencies on connecting, but some
            XProc pipelines make requests over <code>http/s</code>.)</p>
               <p>
                  <b>You are comfortable entering commands on the command line</b> (i.e. terminal or console window). For
            installation, you want a <code>bash</code> shell if available. On Windows, both WSL (Ubuntu) and Git Bash
            have been found to work. If you cannot use <code>bash</code>, the setup can be done by hand (downloading and
            unpacking a package from SourceForge).</p>
               <p>After installation, subsequent work on Windows does not require <code>bash</code> unless you choose to use
            it – a Windows <code>CMD</code> or Powershell can serve as your environment and the processor invoked with a
            Windows <code>bat</code> file (as described in the documentation). Mac and Linux (and WSL) users can
            continue to use <code>bash</code>.</p>
               <p>If you have already performed the setup as described in <a href="../README.md">README</a> and <a href="../setup-notes.md">setup notes</a>, this lesson unit will be a breeze.</p>
               <p>Prior knowledge of XProc, XSLT or XML is <i>not</i> a prerequisite (for this or any lesson unit). If you are
            learning as we go – at any level – welcome and please seek us out for help and feedback.</p>
            </section>
            <section>
               <h2>Resources</h2>
               <p>The setup script is a <code>bash</code> script: <a href="../setup.sh">./setup.sh</a>, to be run with
            no arguments. See <a href="../setup-notes.md">top-level documentation</a> if you can't use this script
            or if you prefer to download and unzip the dependencies by hand.</p>
               <p>For XProc runtime — to execute pipelines — use either of the scripts <a href="../xp3.sh">./xp3.sh</a>
            (under <code>bash</code>) or <a href="../xp3.bat">./xp3.bat</a> (for Windows). These scripts are used
            for all pipelines (basically, for everything) unless otherwise noted.</p>
               <p>To perform the setup: first, you download an XProc engine; then you complete setup and testing by running
            these pipelines. They are described in top-level <a href="../README.md">README</a> documentation and
            the expected places.</p>
               <ul>
                  <li>
                     <a href="../lib/GRAB-SAXON.xpl">lib/GRAB-SAXON.xpl</a>
                  </li>
                  <li>
                     <a href="../lib/GRAB-SCHXSLT.xpl">lib/GRAB-SCHXSLT.xpl</a>
                  </li>
                  <li>
                     <a href="../lib/GRAB-XSPEC.xpl">lib/GRAB-XSPEC.xpl</a>
                  </li>
                  <li>
                     <a href="../smoketest/TEST-XPROC3.xpl">smoketest/TEST-XPROC3.xpl</a>
                  </li>
                  <li>
                     <a href="../smoketest/TEST-XSLT.xpl">smoketest/TEST-XSLT.xpl</a>
                  </li>
                  <li>
                     <a href="../smoketest/TEST-SCHEMATRON.xpl">smoketest/TEST-SCHEMATRON.xpl</a>
                  </li>
                  <li>
                     <a href="../smoketest/TEST-XSPEC.xpl">smoketest/TEST-XSPEC.xpl</a>
                  </li>
               </ul>
            </section>
            <section>
               <h2>Step One: Setup</h2>
               <p>Find setup instructions for the repository in the <a href="../README.md">Project README</a> and in the
            linked <a href="../setup-notes.md">Setup Notes</a>.</p>
               <p>After reading and reviewing these documents, perform the setup on your system as instructed. To do this you
            can either fork or clone the repository in GitHub or simply download and decompress a zip of the <a href="../">current
            distribution</a>.</p>
               <p>After running the setup script, or performing the installation by hand, make sure you can run all the smoke
            tests successfully.</p>
               <p>As noted in the docs, if you happen already to have an XProc 3.0 processor, you do not need to download <a href="../">Morgana XProc III</a> here. At time of writing
            (December 2024) this notably includes <a href="../">XML Calabash
               3</a> (newly out in an alpha release). In any case, equipped with <i>any conformant XProc 3.0/3.1
               implemenentation</i>. try skipping straight to the smoke tests. You can use a runtime script
               <code>xp3.sh</code> or <code>xp3.bat</code> as a model for your own, and adjust.</p>
               <section>
                  <h3>Shortcut</h3>
                  <p>If you want to run through the tutorial exercises but you are unsure of how deeply you will delve, you
               can postpone two of the installations until later:</p>
                  <ul>
                     <li>You will need XSpec only when you want to run tests of stylesheets or queries using the <a href="../">XSpec</a> testing framework</li>
                     <li>You will need SchXSLT only when you want to run Schematron (or XSpec tests of Schematron)</li>
                  </ul>
                  <p>When you see tracebacks suggesting one of these is not supported, you can return to setup.</p>
                  <p>Since almost any pipeline will use XSLT and since we do use the latest version (XSLT 3.0 with XPath 3.1),
               consider the Saxon installation an essential requirement.</p>
               </section>
            </section>
            <section>
               <h2>Step Two: Confirm</h2>
               <p>The top-level README and setup notes also describe testing your installation. Do this next.</p>
               <p>You know things are working in your XProc when two things are happening:</p>
               <ul>
                  <li>On the console, notifications show up with reassuring messages announcing progress</li>
                  <li>When you expect files to be produced for you, they appear, or are updated, as expected</li>
               </ul>
               <p>Both of those will occur with this lesson. The files produced by downloading pipelines are written into the
            project <code>lib</code> directory, as documented. Refresh or restore by deleting the downloaded files and
            running the pipelines to acquire them again.</p>
               <p>Note: you need a live Internet connection for your <code>http</code> requests to go through.</p>
               <p>When you can run all the smoke tests without ugly tracebacks, this lesson is complete.</p>
            </section>
            <section>
               <h2>Comments / review</h2>
               <p>Within this project as a whole, and within its subprojects, everything is done with XProc 3.0. The aim is to
            make it possible to do anything needed with XProc, regarded as a general-purpose 'scripting' solution for
            the choreography of arbitrarily complex jobs, tasks and workflows. To support arbitrary complexity and
            scalability together, it must be very simple. This simplicity, with the composability that goes with it, is
            at the center of the argument for XProc.</p>
               <p>You will see this simplicity at the level of <q>top-level</q>
                  <em>invocation</em> XProc pipelines designed to serve as entry points. If things are done right, these will
            be fairly simple, well encapsulated <q>subroutines</q> in potentially elegant arrangements. They in turn may
            call on libraries of XProc pipelines for well-defined tasks.</p>
               <p>Effectively (and much more could be said about the processing stack, dependency management and so forth)
            what this means is that XProc promises the user and the developer (in either or both roles) with focused and
            concentrated points of control or points of adjustment. In the field – where software is deployed and used –
            things almost never just <q>drop in</q>. User interfaces, APIs, dependencies and platform quirks: all these
            constrain what users can do, and even developers are rarely as free as they would like to experiment and
            explore.</p>
               <p>What is offered here is therefore <i>both</i> an example of a deployment of a demonstration solution set
            using an open-source tool (an XProc engine capable of running the pipelines we offer), doing things that are
            actually or potential useful (with OSCAL data), <i>and</i> a set of pipelines that should in principle work
            as well in any other tool or software deployment supporting XProc 3.0.</p>
               <p>But to the extent this imposes a requirement for both abstract and testable conformance (or at any rate for
            interoperability as a proxy for that), this project only works if things are actually simple enough to pick
            up, use, learn and adapt. <code>xp3.sh</code> and <code>xp3.bat</code> represent attempts at making a simple
            deployment, easy to emulate but better yet, improve.</p>
               <p>Each of these scripts (on its execution platform) enables a user to run, without further configuration, the
               <a href="../">Morgana XProcIIIse</a> processor on any
            XProc 3.0 pipeline, assuming the appropriate platform for each (<code>bash</code> in the case of the shell
            script, Windows batch command syntax for the <code>bat</code> file). Providing a similar script for XML
            Calabash remains (with apologies to NDW) a <i>desideratum</i> for this project as we post this version of
            the tutorial. Stay tuned!</p>
               <p>In any case such a script itself must be <q>vanilla</q> and generic: it will simply invoke the processor
            with the designated pipeline, and stand back. (Yes, runtime arguments and settings can be provided.) The
            logic of operations is entirely encapsulated in the XProc pipeline designated. XProc 3.0 is both scalable
            and flexible enough to open a wide range of possibilities for data processing, both XML-based and using
            other formats such as JSON and plain text. It is the intent of this project not to explore and map this
            space – which is vast – but to show off enough XProc and related logic (XSLT, XSpec) to show how this
            exploration can be done. We are an outfitter at the beginning of what we hope will be many profitable
            voyages to places we have never been.</p>
               <section>
                  <h3>When running from a command line</h3>
                  <p>As simple examples, these scripts show only one way of running XProc. Keep in mind that even simple
               scripts can be used in more than one way.</p>
                  <p>For example, a pipeline can be executed from the project root:</p>
                  <pre>$ ./xp3.sh smoketest/TEST-XPROC3.xpl</pre>
                  <p>Alternatively, a pipeline can be executed from its home directory, for example if currently in the
                  <code>smoketest</code> directory (note the path to the script):</p>
                  <pre>$ ../xp3.sh TEST-XPROC3.xpl</pre>
                  <p>This works the same ways on Windows, with adjustments:</p>
                  <pre>&gt; ..\xp3 TEST-XPROC3.xpl </pre>
                  <p>(On Windows a <code>bat</code> file suffix marks it as executable and does not have to be given
               explicitly when called.)</p>
                  <p>Windows users (and others to varying degrees) can set up <a href="../">a drag-and-drop based workflow</a> –
               using your mouse or pointer, select an XProc pipeline file and drag it to a shortcut for the executable
               (Windows batch file). A command window opens to show the operation of the pipeline. See the <a href="../tutorial/README.md">README</a> for more information.</p>
                  <p>It is important to try things out since any of these methods can be the basis of a workflow.</p>
                  <p>For the big picture, keep in mind that while the command line is useful for development and demonstration
               – and however familiar XProc itself may become to the developer – to a great number of people it remains,
               like XProc, obscure, cryptic and intimidating if not forbidding.</p>
                  <p>This is a pity because (among other reasons) the kind of layered system we will see and build here is not
               endless or infinitely complex. Begin by making yourself comfortable at the command line. See how the
               pieces fit together by working them.</p>
                  <p>Then too, if you have something better, by all means use it. XProc-based systems, when integrated into
               tools or developer editors and environments, can look much nicer than tracebacks in a console window. The
               elegance and power we are trying to cultivate are at a deeper level. First and last, the focus must be on
               the data.</p>
               </section>
            </section>
         </section>
         <section class="unit maker"
                  id="acquire_102"
                  data-track="maker">
            <h1>102: Examining the setup</h1>
            <section>
               <h2>Goals</h2>
               <ul>
                  <li>Look at some pipeline organization and syntax on the inside</li>
                  <li>Success and failure invoking XProc pipelines: making friends with tracebacks</li>
               </ul>
            </section>
            <section>
               <h2>Prerequisites</h2>
               <p>Please complete the repository setup and smoke tests as described in the <a href="../tutorial/source/acquire/acquire_101_src.html"
                     class="LessonUnit">101 lesson</a>. In this lesson, we will run these pipelines with adjustments, or
            similar pipelines.</p>
               <p>This discussion assumes basic knowledge of coding, the Internet (including retrieving resources via
               <code>file</code> and <code>http</code> protocols), and web-based technologies including HTML.</p>
               <p>XML knowledge is <i>not</i> assumed. This poses a special challenge since in addition to its XML-based
            syntax, XProc uses the <a href="../">XML Data Model (XDM)</a> along with
               <a href="../">XPath 3.1</a>, the query language for XML: together, a deep
            topic. We make the assumption that if you already know XML, XPath, XSLT or XQuery, much will be familiar,
            but you will be tolerant of some restatement for the sake of those who do not. (As we all start somewhere,
            why not here.)</p>
               <p>You will also need a programmer's plain text editor, XML/XSLT editor or IDE (integrated development
            environment) for more interactive testing of the code.</p>
            </section>
            <section>
               <h2>Resources</h2>
               <p>Same as <a href="../tutorial/source/acquire/acquire_101_src.html"
                     class="LessonUnit">Setup 101</a>.</p>
            </section>
            <section>
               <h2>Step One: Inspect the pipelines</h2>
               <p>The two groupings of pipelines used in setup and testing can be considered separately.</p>
               <p>The key to understanding both groups is to know that once the initial <a href="../setup.sh">Setup
               script</a> is run, your processor or <q>engine</q> (such as Morgana) can be invoked directly, as paths
            and scripts are already in place. In doing so – before extension libraries are in place – it can use only
            basic XProc steps, but those are enough to start with.</p>
               <p>Specifically, the pipelines can acquire resources from the Internet, save them locally, and perform
            unarchiving (unzipping). Having been downloaded, each library provides software that the pipeline engine
            (Morgana) can use to do more.</p>
               <p>Accordingly, the first group of pipelines (in the <a href="../lib/readme.md">lib</a> directory has
            a single purpose, namely (together and separately) to download software to augment Morgana's feature
            set.</p>
               <p>If not using the open-source Morgana distribution, you can skip to smoke tests below, and see how far you
            get.</p>
               <ul>
                  <li>
                     <a href="../lib/GRAB-SAXON.xpl">lib/GRAB-SAXON.xpl</a>
                  </li>
                  <li>
                     <a href="../lib/GRAB-SCHXSLT.xpl">lib/GRAB-SCHXSLT.xpl</a>
                  </li>
                  <li>
                     <a href="../lib/GRAB-XSPEC.xpl">lib/GRAB-XSPEC.xpl</a>
                  </li>
               </ul>
               <p>Pipelines in a second group work similarly in that each one exercises and tests capabilities provided by
            software downloaded by a member of the first group.</p>
               <ul>
                  <li>
                     <a href="../smoketest/TEST-XPROC3.xpl">smoketest/TEST-XPROC3.xpl</a> tests the execution runtime
               (MorganaXProc-III or other engine)</li>
                  <li>
                     <a href="../smoketest/TEST-XSLT.xpl">smoketest/TEST-XSLT.xpl</a> tests Saxon</li>
                  <li>
                     <a href="../smoketest/TEST-SCHEMATRON.xpl">smoketest/TEST-SCHEMATRON.xpl</a> tests SchXSLT
               (using <code>p:validate-with-schematron</code> step)</li>
                  <li>
                     <a href="../smoketest/TEST-XSPEC.xpl">smoketest/TEST-XSPEC.xpl</a> tests XSpec (using locally
               defined steps)</li>
               </ul>
               <p>Take a look at these files. It may be helpful (for those getting used to it) to envision the XML syntax as a
            set of nested frames with labels and connectors.</p>
               <p>Try more than one way of looking at the XProc source code: in the Github repository, on your file system, in
            a plain text editor, in an XML editor.</p>
            </section>
            <section>
               <h2>Step Two: Modify the pipelines</h2>
               <p>Use a text editor or software development application for this exercise.</p>
               <p>If you have any concepts for improvements to the pipelines, or other resources that might be acquired this way, copy and modify one of the pipelines given to achieve those results.</p>
               <p>Even if not: be sure to break the pipelines given – or copies under new names – in any of several ways. Then
            run the modified pipelines, as a <i>safe way</i> to familiarize yourself with error messages:</p>
               <ul>
                  <li>Break the XML syntax of a pipeline and try to run it</li>
                  <li>Leave XML syntax intact (well-formed), but break something in the XProc <ul>
                        <li>An element name, attribute or attribute setting</li>
                        <li>A namespace</li>
                     </ul>
                  </li>
                  <li>Try to retrieve something from a broken link</li>
               </ul>
               <p>Having introduced an error, reverse the damage. Make sure your pipelines are back in working order when this
            exercise is complete.</p>
            </section>
            <section>
               <h2>For consideration</h2>
               <p>Developers coming to this technology need to consider who would use it, and whether it is useful mainly at
            the back end, or also <q>on the shop floor</q>, directly in the hands of professionals who must work with
            the data, bringing expertise in subject matter (such as, for OSCAL, systems security documentation) but not
            in data processing as such.</p>
               <p>Key to this question is not only whether attractive and capable user interfaces (or other mediators) can be
            developed (this is a known problem) but more importantly whether the systems themselves are adaptable enough
            so they can be deployed, used, refitted and maintained not just for repetitive generic tasks, but for
               <i>particular</i>, <i>special</i> and <i>local</i> problems, especially those discoverable only at the
            points where information is gathered and codified.</p>
               <p>This larger fitting of solutions to problems is a responsibility for both SMEs (subject matter experts) and
            software developers together, who must define problems to be solved before approaches to them can be
            found.</p>
               <p>The open questions are: who can use XProc pipelines; and how can they be made more useful? The questions
            come up in an OSCAL context or any context where XML is demonstrably capable, or indeed anywhere we find the
            necessity of handling data with digital tools has become inescapable.</p>
               <p>In order to help answer this question, actual experience will be invaluable – part of our motive here.
            Unless we can make the demonstration pipelines in this repository accessible, they cannot be reasoned about.
            That accessibility requires not only open publication, but also use cases and  user bases ready to take
            advantage.</p>
               <p>Having completed and tested the setup you are ready for work with XProc: proceed to the next lesson.</p>
            </section>
         </section>
         <section class="unit learner"
                  id="acquire_599"
                  data-track="learner">
            <h1>599: Meeting XProc</h1>
            <section>
               <h2>Goals</h2>
               <p>Gain some more sense of context.</p>
               <p>XProc is not a simple thing, with only one way in. The territory is vast, but it has also been well charted.
            And here we have a pathway marked in front of us.</p>
            </section>
            <section>
               <h2>Resources</h2>
               <p>
                  <a href="../">A Declarative Markup Bibliography</a> is
            available on line for future reference on this interesting topic.</p>
            </section>
            <section>
               <h2>Some observations</h2>
               <p>Because it is now centered on <em>pipelines</em> built out of combining capabilities of <em>steps</em>
            (which may be black boxes), as much as on files and software packages, dependency management when using
            XProc is different from other technologies including Java and NodeJS – how so?</p>
               <p>MorganaXProc-III is implemented in Scala, and Saxon is built in Java, but otherwise distributions including
            the SchXSLT and XSpec distributions consist mainly of XSLT. This is either very good (with development and
            maintenance requirements in view), or not good at all.</p>
               <p>If not using Morgana but another XProc engine (at time of writing, XML Calabash 3 has been published in
            alpha), there will presumably be analogous arrangements: contracts between the tool and its dependencies,
            software or components and capabilities bundled and unbundled.</p>
               <p>So does this work well, on balance, and what are the determining variables that tell you XProc is a good fit
            for data processing, whether high touch, or at scale? How much of this is due to the high-level, abstracted
            nature of <a href="../">4GLs</a> including
            both XSLT 3.1 and XProc 3.0? Prior experience with XML-based systems and the problem domains in which they
            work well is probably a consideration. But maybe the more important blockers have to do with culture, states
            of knowledge, incorrect assumptions and outdated perceptions.</p>
               <p>Will it always be that a developer determined to use XSLT will find a way, whereas a developer determined
            not to, will find a way to refuse it? XProc in 2024 seems slow in adoption – maybe because everyone who
            would want it, already has a functional equivalent in place.</p>
               <p>In any case, it might also be that such neglect creates a market opportunity. Those who use these
            technologies without advertising the fact may have the most to gain. But building the commons is also a
            common responsibility.</p>
               <p>It's all about the tools.  Find ways to support your open-source developer and the software development
            operations who offer free tools and services.</p>
            </section>
            <section>
               <h2>Declarative markup in action</h2>
               <p>Considerable care is taken in developing these demonstrations to see to it that the technologies on which we
            depend, notably XProc and XSLT but not limited to these, are both nominally and actually conformant to
            externally specified standard technologies, i.e. XProc and XSLT respectively (as well as others), and
            reliant to the greatest possible extent on well-documented and accessible runtimes.</p>
               <p>Is it too much to expect that any code base should be both easy to integrate and use with others, and at the
            same time, functionally complete and self-sufficient? Of these two, we are lucky to get one, even if we are
            thoughtful enough to limit ourselves to building blocks. Because the world is complex, we are always
            throwing in one or another new dependency, along with new rule sets. The approach enabled by XML and
            openly-specified supporting specifications is to work by making everything transparent as possible. We seek
            for clarity and transparency at all levels (so nothing is downloaded behind the scenes, for example) while
            also documenting as thoroughly as we can, including with code comments.</p>
               <p>Can any code base be fully self-explanatory and self-disclosing? This may be doubtful even if we can agree
            what those terms mean. At the same time, the attempt can be made: we can try and leave tracks and markers,
            at least. We call it <q>code</q> with the hope and intent that it should be amenable to and rewarding of
            interpretation.</p>
               <section>
                  <h3>Standards for documents</h3>
                  <p>In addition to the web itself (in HTML and CSS), a number of important initiatives in recent decades have
               capitalized on the core principles of declarative markup:</p>
                  <ul>
                     <li>
                        <a href="../">Text Encoding Initiative
                  </a>(TEI) since 1989</li>
                     <li>OASIS <a href="../">DocBook</a> since 1991</li>
                     <li>
                        <a href="../">Darwin Information
                     Typing Architecture</a> (DITA) since 2001</li>
                     <li>
                        <a href="../">Journal Article Tag Suite</a> (JATS, since 2003) with <a href="../">Standards Tag Suite</a> (STS, since 2017)</li>
                     <li>
                        <a href="../">Open Security Controls Assessment Language</a> (OSCAL), since
                  2018</li>
                  </ul>
               </section>
            </section>
         </section>
      </section>
      <section class="lesson"
               id="walkthrough"
               name="walkthrough">
         <section class="unit observer"
                  id="walkthrough_101"
                  data-track="observer">
            <h1>101: Unpacking XProc 3.0</h1>
            <section>
               <h2>Goals</h2>
               <ul>
                  <li>More familiarity with XProc 3.0, including some syntax</li>
                  <li>Some awareness of XProc's capabilities, along with context including background history, concepts and
               resources</li>
               </ul>
               <p>If you read only a single page from this tutorial, and you know nothing about XProc, this page could offer
            the most rapid introduction.</p>
               <p>But a further goal is to stimulate a reader's curiosity along with their awareness.</p>
            </section>
            <section>
               <h2>Prerequisites</h2>
               <p>
                  <b>No prerequisites.</b> The commentary assumes, however:</p>
               <ul>
                  <li>You have run the acquisition, setup and test pipelines <a href="../tutorial/source/acquire/acquire_101_src.html"
                        class="LessonUnit">described in the first lesson set</a>, or you are willing to pretend you have done
               so for our purposes here</li>
                  <li>
                     <i>Either</i>, you have some prior experience with XML technologies, web publishing technologies, data
               modeling and Internet-based data interchange</li>
                  <li>
                     <i>Or</i> (again) you are open to learning or at least thinking about it</li>
               </ul>
               <p>If you fall into neither of these categories – welcome, and congratulations on your perseverence at least.
            These pages are certainly available to be read and referenced even if you are not running any of the
            software: as part of the resource provided (the repository as a whole), they are open to anyone who finds
            them to be useful, including for specialized purposes.</p>
            </section>
            <section>
               <h2>Resources</h2>
               <p>This lesson discusses, in more depth, the same pipelines you ran in setup: <a href="../tutorial/source/acquire/acquire_101_src.html"
                     class="LessonUnit">Setup 101</a>, in particular <a href="../smoketest/readme.md">the smoke test pipelines</a>.</p>
               <p>Also, this lesson discusses a pipeline used for producing this tutorial: <a href="../tutorial/PRODUCE-PROJECTS-ELEMENTLIST.xpl">PRODUCE-PROJECTS-ELEMENTLIST.xpl</a> generates an <a href="../tutorial/sequence/element-directory.md">index to XProc in this repository</a>. It is offered as a
            first demonstration (among other things) of XProc applied to XProc.</p>
               <section>
                  <h3>For reference</h3>
                  <p>The <a href="../">XProc.org 3.0 dashboard page</a> is maintained by XProc community
               organizers. It offers a hub for reference materials and community contributions.</p>
                  <p>The last section of this lesson unit describes more reference materials as well.</p>
               </section>
            </section>
            <section>
               <h2>A closer look</h2>
               <p>If you have completed <a href="../tutorial/source/acquire/acquire_101_src.html"
                     class="LessonUnit">Acquire 102</a> you have
            already inspected the <a href="../lib/readme.md">lib</a> and <a href="../smoketest/readme.md">smoketest</a> folders, and run the pipelines you have found there. If you haven't, now is a moment to
            catch up. Even if you have decided not to install any software, please look at the pipelines we are using
            for the purpose here.</p>
               <p>Routine code inspection can also be done <a href="../">on Github</a>
            as well (not a bad idea in any case), not just in a copy of the distribution. If you are reading this in the
            repository, the pipelines files (generally, suffixed <code>xpl</code> by convention) can be inspected in
            another browser window.</p>
               <p>A quick summary of what these pipelines do:</p>
               <ul>
                  <li>
                     <a href="../lib/GRAB-SAXON.xpl">lib/GRAB-SAXON.xpl</a> downloads a zip file from a <a href="../">Saxonica download site</a>, saves it, and extracts a
                  <code>jar</code> (Java library) file, which it places in the Morgana library directory</li>
                  <li>
                     <a href="../lib/GRAB-SCHXSLT.xpl">lib/GRAB-SCHXSLT.xpl</a> downloads a zip file from Github and
               unzips it into a directory where Morgana can find it.</li>
                  <li>
                     <a href="../lib/GRAB-XSPEC.xpl">lib/GRAB-XSPEC.xpl</a> also downloads and <q>unarchives</q> a zip
               file resource, this time a copy of <a href="../">an XSpec
               distribution</a>.</li>
               </ul>
               <p>Essentially, these all replicate and capture the work a developer must do to identify and acquire libraries.
            Maintaining our dependencies this way – not quite, but almost <q>by hand</q> – appears to have benefits for
            system transparency and robustness.</p>
               <p>The second group of pipelines is a bit more interesting. Each of the utilities provided for in packages just
            downloaded is tested by running a <b>smoke test</b>.</p>
               <p>Each smoke test performs a minor task, with the aim of determining whether a simple representative process
            will complete successfully. (When we plug the unit in and switch on the power, can we see and smell smoke?
            Do tracebacks burn your retinas?)</p>
               <ul>
                  <li>
                     <a href="../smoketest/TEST-XPROC3.xpl">smoketest/TEST-XPROC3.xpl</a> amounts to an XProc <q>Hello
                  World</q>. In that spirit, feel free to write your own version. (You get another chance to do this <a href="../tutorial/source/walkthrough/walkthrough_102_src.html"
                        class="LessonUnit">real soon now</a>.)</li>
                  <li>
                     <a href="../smoketest/TEST-XSLT.xpl">smoketest/TEST-XSLT.xpl</a> tests Saxon, an XSLT/XQuery
               transformation engine. XSLT and XQuery are related technologies (different languages, same data model)
               developed with XML processing in mind, but in recent years generalized to a wider range of data
               structures.</li>
                  <li>
                     <a href="../smoketest/TEST-SCHEMATRON.xpl">smoketest/TEST-SCHEMATRON.xpl</a> tests SchXSLT.
               SchXSLT is an implementation of Schematron, an ISO-standard validation and reporting technology. As this
               implementation relies on XSLT, this library also requires Saxon.</li>
                  <li>
                     <a href="../smoketest/TEST-XSPEC.xpl">smoketest/TEST-XSPEC.xpl</a> tests XSpec, an XSLT-based
               testing framework useful for testing deployments of XSLT, XQuery and Schematron.</li>
               </ul>
               <p>Any and each of these can be used as a <q>black box</q> by any competent operator, even without
            understanding the internals. But this simplicity masks and manages complexity. XProc is XProc but never just
            that, since its capabilities are also extended by XSLT, XQuery, Schematron, XSpec and others, an open-ended
            set of compatible and complimentary technologies that are even more powerful together than they are in
            particular.</p>
               <p>At the same time, common foundations make it possible to learn these technologies together and in
            tandem.</p>
            </section>
            <section>
               <h2>Survey</h2>
               <p>Each of the test pipelines exercises a simple sequence of operations. Open any XProc file in an editor or
            viewer where you can see the tags. Skim this section to get only a high-level view.</p>
               <p>The aim here is demystification. Understand the parts to understand the whole. Reading the element names
            also inscribes them in (metaphorical) memory circuits where they will resonate later.</p>
               <section>
                  <h3>
                     <a href="../smoketest/TEST-XPROC3.xpl">TEST-XPROC3</a>
                  </h3>
                  <p>Examine the pipeline <a href="../smoketest/TEST-XPROC3.xpl">TEST-XPROC3.xpl</a>. It is a short
               chain of two steps, with one output offered. It breaks down as follows:</p>
                  <ul>
                     <li>
                        <code>p:output</code> – An <em>output port</em> is defined. It specifies that when the process
                  results are delivered, a couple of serialization rules are followed: the text is indented and written
                  without an XML declaration at the top. With this port, the process outputs can be captured by the
                  calling process (such as your script), or simply echoed to the console.</li>
                     <li>
                        <code>p:identity</code> – An <em>identity step</em> does nothing with its input except pass it along.
                  This one has its input given as a literal (XML) fragment in the pipeline. Essentially, because this
                  pipeline has this step, it does not need to load or rely on any inputs, because its inputs are given
                  here. The input is a single line of XML.</li>
                     <li>
                        <code>p:namespace-delete</code> – A <em>namespace-delete</em> step is used to strip an XML namespace
                  definition from the document coming back (resulting from) the prior identity step. When nothing else
                  is specifically designated as such, the input of any step is assumed to be the last step's results. In
                  this case, our XML inherits namespaces from the pipeline itself (where it is embedded), but it has no
                  elements or attributes that use it, so the namespace is unneeded and its declaration comes through as
                  noise. With this step the pipeline results are clean and simple.</li>
                  </ul>
                  <p>When you run this pipeline, the <code>CONGRATULATIONS</code> document given in line will be echoed to the
               console, where designated outputs will appear if not otherwise directed.</p>
               </section>
               <section>
                  <h3>
                     <a href="../smoketest/TEST-XSLT.xpl">TEST-XSLT</a>
                  </h3>
                  <p>
                     <a href="../smoketest/TEST-XSLT.xpl">This pipeline</a> executes a simple XSLT transformation, in
               order to test that XSLT transformations can be successfully executed.</p>
                  <ul>
                     <li>
                        <code>p:output</code> – An output port is designated with <code>p:output</code> as in the <a href="../smoketest/TEST-XPROC3.xpl">TEST-XPROC3 pipeline</a>.</li>
                     <li>
                        <code>p:xslt</code> – Instead of providing a literal document in an <em>identity</em> step, this
                  pipeline performs an XSLT transformation. The input to this transformation is given as a literal XML
                  in the same way, except this time it is provided as input to a transformation process defined by an <a href="../smoketest/src/congratulations.xsl">XSLT stylesheet</a> called in by the
                  pipeline.</li>
                     <li>
                        <code>p:namespace-delete</code> – The <code>ox</code> namespace is stripped from the result as in <a href="../smoketest/TEST-XPROC3.xpl">TEST-XPROC3 pipeline</a>. This could have been done in
                  the XSLT as well, but this way the transformation has one less thing to do or go wrong. More simpler
                  steps prove more legible and tractable than fewer complicated ones.</li>
                  </ul>
                  <p>Like the <a href="../smoketest/TEST-XPROC3.xpl">TEST-XPROC3 pipeline</a> this pipeline shows its
               results in the console. This time the result is not just the XML given in the pipeline, but that XML as
               modified by the transformation.</p>
                  <p>If your pipeline execution can't process the XSLT (perhaps Saxon is not installed, or the XSLT itself has
               a problem) you will get an error saying so.</p>
                  <p>Errors in XProc are reported by the XProc engine, typically using XML syntax. (The exact format of errors
               depends on the processor, and they are very much worth comparing.) Among other things, this means that
               XML results (for example showing errors trapped in try/catch) can be captured and processed in
               pipelines.</p>
               </section>
               <section>
                  <h3>
                     <a href="../smoketest/TEST-SCHEMATRON.xpl">TEST-SCHEMATRON</a>
                  </h3>
                  <p>Schematron is a language used to specify rules to apply to XML documents. In this case a small Schematron
               is applied to a small XML. This flexible technology enables easy testing of XML against rule sets defined
               either for particular cases in particular workflows, or for entire classes or sets of documents whose
               rules are defined for standards and across systems.</p>
                  <ul>
                     <li>
                        <code>p:output</code> – An output port is designated for the results with the same settings.</li>
                     <li>
                        <code>p:validate-with-schematron</code> – This is an XProc step specifically for evaluating an XML
                  document against the rules of a given Schematron. Like the TEST-XPROC3 and TEST-XSLT` pipelines, this
                  one presents its own input, given as a literal XML document given in the pipeline document (using
                     <code>p:inline</code>). A setting on this step provides for it to throw an error if the document
                  does not conform to the rules. The Schematron file provided as input to this step, <a href="../smoketest/src/doing-well.sch">src/doing-well.sch</a>, gives the rules.</li>
                     <li>
                        <code>p:namespace-delete</code> – This step is used here as in the other tests for final cleanup of
                  the information set produced (as a namespace-qualified XML document).</li>
                  </ul>
               </section>
               <section>
                  <h3>
                     <a href="../smoketest/TEST-XSPEC.xpl">TEST-XSPEC</a>
                  </h3>
                  <p>
                     <a href="../">XSpec</a> is a testing framework for XSLT, XQuery and
               Schematron. It takes the form of a vocabulary and a process (inevitably implemented in XSLT and XQuery)
               for executing queries, transformations, and validations, by running them over known inputs, comparing the
               results to expected results, and reporting the results of this comparison. XProc, built to orchestrate
               manipulations of XML contents, is well suited for running XSpec.</p>
                  <p>An XSpec instance (as a document in itself) defines a set of tests for a transformation or query module
               using the XSpec vocabulary. An XSpec implementation executes the tests and delivers the results. Since
               XSpec, like Schematron, reports its findings in XML, XProc can be useful both to manage the inputs and
               outputs, and to process the XSpec reports.</p>
                  <ul>
                     <li>
                        <code>p:import</code> – calls to an external XProc file to make its step definitions available.</li>
                     <li>
                        <code>p:input</code> – works as it does elsewhere, to declare inputs for the pipeline. In this case,
                  the inputs must be XSpec documents using the XSpec vocabulary. You can expect errors when they are
                  not. This testing pipeline offers three different XSpecs to be run, one each for XSLT, XQuery and
                  Schematron.</li>
                     <li>
                        <code>p:for-each</code> – defines a step or sequence of steps to be applied to each input,
                  separately.</li>
                     <li>
                        <code>p:sink</code> – Having performed  the validations, we are done. While this step is optional
                  here and a <q>no-op</q> in the XProc itself, it provides an occasion for a message to help trace the
                  XProc execution.</li>
                  </ul>
                  <p>
                     <a href="../tutorial/source/walkthrough/walkthrough_102_src.html"
                        class="LessonUnit">The next lesson</a> offers more detail about this
               pipeline.</p>
               </section>
            </section>
            <section>
               <h2>A not-so-simple pipeline</h2>
               <p>Examine the Markdown file presenting an <a href="../tutorial/sequence/element-directory.md">XProc Element
               directory</a>. It is generated by <a href="../tutorial/PRODUCE-PROJECTS-ELEMENTLIST.xpl">a pipeline</a>.
            Examine that pipeline to see XProc with real-world complexity.</p>
               <p>The simple pipelines examined so far show how useful things can be done simply, while the pipeline
            architecture allows for great flexibility.</p>
               <p>Simplicity and flexibility together enable complexity. Once it is factored out, a complex operation can be
            managed and deployed just like a simple one, with its internal complexities masked by a simple and
            predictable interface.</p>
               <p>Next, take a look at a more complex example, the prototype pipeline <a href="../tutorial/PRODUCE-PROJECTS-ELEMENTLIST.xpl">PRODUCE-PROJECTS-ELEMENTLIST.xpl</a>. Like the setup and
            smoke-test pipelines, this is a standalone pipeline (requiring no runtime bindings or settings): when this
            plan (<q>step</q> or pipeline) is executed, the processor acquires inputs, produces results for its
            operations, and write those results to the file system. In this case the output it generates is stored as <a href="../tutorial/sequence/element-directory.md">element-directory.md</a>, a Markdown file (find the
               <code>p:store</code> step).</p>
               <p>The result is a reference resource encoded in Markdown: an index of XProc elements used in pipelines in this
            repository. As Markdown, once reposted back into the repository, it can be viewed with any Markdown viewing
            application. The index lists XProc elements, i.e., the core of the XProc vocabulary: for any XProc element
            used anywhere among the projects listed, the listing shows the pipelines where it appears. Following the
            index, the resource also shows a list of (repository) project folders in a prescribed order, with their
            XProc files and whatever XProc elements appear <i>first</i> (within the entire sequence up to to point)
            within that file. Among other uses this is helpful for assessing coverage of tutorial lessons as it offers a
            (semi) <i>ordered</i> survey of their use of XProc (and other) elements.</p>
               <p>For example, looking up <code>p:store</code> you can see all the pipelines that contain this common step. Or
            looking at the <code>oscal-convert</code> listing you can see the XProc steps appearing first in that
            project folder.</p>
               <p>To confirm proper functioning, run the pipeline again after deleting or renaming the Markdown result
            file.</p>
               <p>Consider also what other kinds of indexing might be useful. When you modify XProc or add new XProc pipelines
            to the project folders, consider running this pipeline again to update the indexes.</p>
               <p>Open the file and inspect it to get a sense of how it works. The XML syntax is verbose, but not really all
            that frightening. Practice helps! The pipeline is also considered in the <a href="../tutorial/source/walkthrough/walkthrough_102_src.html"
                     class="LessonUnit">102 Lesson unit</a> segment.</p>
               <section>
                  <h3>PRODUCE-PROJECTS-ELEMENTLIST</h3>
                  <p>The pipeline <a href="../tutorial/PRODUCE-PROJECTS-ELEMENTLIST.xpl">PRODUCE-PROJECTS-ELEMENTLIST.xpl</a>
               contains XML comments (<code>&lt;!-- using comment syntax --&gt;</code>) that should help explain its
               operations.</p>
                  <p>There are some significant differences between this pipeline and the small ones we have looked at so
               far.</p>
                  <ul>
                     <li>Of course, it is longer and more complex, reflecting the complexity of the operations it
                  performs.</li>
                     <li>Part of the complexity is due to a two-step process here. First, the file system is surveyed in
                  locations named in an input configuration. Then all those resources (which happen to be XML using the
                  XProc vocabulary) are indexed again, this time showing only first occurrences of elements within files
                  given in the stipulated order. Both indexes are written into the results, showing how a single survey
                  can support more than one analysis.</li>
                     <li>In particular, the index to <q>first use</q> is not simple. A great deal of the complexity of
                  detailed operations has been off-loaded into XSLT transformation code, which in this pipeline can be
                  seen embedded in the XProc, indeed occupying the greater part of the XML in the file. (About two
                  thirds of the element count: you can usually recognize XSLT by the conventional <code>xsl:</code>
                  element prefix.) This pipeline also has an XSLT called from an external file (toward the end). XProc
                  can provide XSLT either way, and each has its advantages. In this case, XSLT is left in place as
                  literal embedded code, partly to show a borderline case. (It helps to use an editor or viewer with
                  code folding.)</li>
                     <li>One good thing about seeing the XSLT here is you can get a good sense of what it looks like, whether
                  embedded or kept externally. XSLT is <a href="../tutorial/source/walkthrough/walkthrough_401_src.html"
                           class="LessonUnit">not
                     essential to XProc</a>, but it very much expands its practical capabilities.</li>
                  </ul>
               </section>
            </section>
            <section>
               <h2>Respecting XML syntax, XPath and XProc</h2>
               <p>Newcomers to XML may feel they are in the deep water with XML syntax.</p>
               <p>In the context of XProc, this is actually not as hard as it looks:</p>
               <ul>
                  <li>All XML files follow the same syntax rules with respect to tags, elements and attributes (names and
               syntax), namespaces, comments etc.</li>
               </ul>
               <pre>&lt;tagged attribute="value"&gt;Information goes here&lt;/tagged&gt;
&lt;!-- comment goes here --&gt; </pre>
               <ul>
                  <li>XML vocabularies are typically qualified with <b>namespaces</b> to show, and to disambiguate, which XML
               application or language they belong to. The namespaces are indicated by name <em>prefixes</em>. So in
               this repository (and conventionally for XProc), any element prefixed <code>p:</code> is an XProc element,
               and another prefix or none indicates an extension or another vocabulary, such as may appear in XML being
               processed.</li>
               </ul>
               <pre>&lt;p:output port="result" serialization="map{'indent' : true(), 'omit-xml-declaration': true() }" /&gt;</pre>
               <ul>
                  <li>Embedded in the syntax is another syntax, <em>XPath</em>. This lightweight but powerful query language
               is a formal subset of XQuery. XPath is ubiquitous in XProc, XSLT, Schematron, XSpec etc. In XProc, the
               XPath will ordinarily be given in attributes.</li>
               </ul>
               <pre>&lt;p:output port="result" serialization="<b>map{'indent' : true(), 'omit-xml-declaration': true() }</b>" /&gt;</pre>
               <ul>
                  <li>Learn more about this in the <a href="../tutorial/source/walkthrough/walkthrough_102_src.html"
                        class="LessonUnit">102 Lesson unit</a> –
               or plunge on, and pick up what you need as you go.</li>
               </ul>
            </section>
            <section>
               <h2>Learning more about XProc</h2>
               <p>This tutorial has a handmade <a href="../tutorial/xproc-links.md">XProc links page</a> with links.</p>
               <p>Also, see the official <a href="../">XProc.org dashboard page</a>.</p>
               <p>Also, check out indexing logic offered in the <a href="../projects/xproc-doc/readme.md">xproc-doc
               project folder</a>. It has pipelines producing useful indexes to XProc in this repository and in general,
            including one producing an <a href="../projects/xproc-doc/XPROC-STEP-INDEX-HTML.xpl">index to XProc
               simple steps</a> in HTML, with code snips.</p>
               <p>There is <a href="../">a book, Erik Siegel's <i>XProc 3.0
                  Programmer's Reference</i>
                  </a> (2020) and an <a href="../">excellent
               reference site</a> by the same author. Like this resource, it is generated using XProc.</p>
            </section>
         </section>
         <section class="unit maker"
                  id="walkthrough_102"
                  data-track="maker">
            <h1>102: XProc fundamentals</h1>
            <section>
               <h2>Goals</h2>
               <ul>
                  <li>More familiarity with XProc 3.0, with more syntax</li>
                  <li>Get hands a little dirty – and practice washing up</li>
                  <li>First look at XProc pipeline organization</li>
               </ul>
            </section>
            <section>
               <h2>Prerequisites</h2>
               <p>You have done <a href="../tutorial/source/acquire/acquire_101_src.html"
                     class="LessonUnit">Setup 101</a>, <a href="../tutorial/source/acquire/acquire_101_src.html"
                     class="LessonUnit">Setup 102</a> and <a href="../tutorial/source/walkthrough/walkthrough_101_src.html"
                     class="LessonUnit">Unpack 101</a>.</p>
            </section>
            <section>
               <h2>Resources</h2>
               <p>Take a quick look <i>now</i> (and a longer look later):</p>
               <ul>
                  <li>This tutorial's handmade <a href="../tutorial/xproc-links.md">XProc links page</a>
                  </li>
                  <li>Also, the official <a href="../">XProc.org dashboard page</a>
                  </li>
                  <li>If interested, check out XProc index materials produced in this repository: <a href="../projects/xproc-doc/readme.md">XProc docs</a>
                  </li>
                  <li>In any case, the same pipelines you ran in setup: <a href="../tutorial/source/acquire/acquire_101_src.html"
                        class="LessonUnit">Setup 101</a>.</li>
               </ul>
            </section>
            <section>
               <h2>Learning more about XProc</h2>
               <p>A partial list of ways to learn more about XProc:</p>
               <ul>
                  <li>Search engines: use keywords <q>XProc3</q> or <q>XProc 3.0</q> to help distinguish from 1.0
               technologies</li>
                  <li>Resources: <a href="../tutorial/xproc-links.md">links</a> here and elsewhere</li>
                  <li>Hands on exercises</li>
                  <li>Work the notes - save out and annotate these pages</li>
               </ul>
            </section>
            <section>
               <h2>Details details!</h2>
               <p>XProc pipelines described in <a href="../tutorial/source/walkthrough/walkthrough_101_src.html"
                     class="LessonUnit">the previous lesson unit</a>
            contain a few noteworthy features.</p>
               <p> To edit these files, use any XML-capable plain text editor (that is, with care, any editor at all that
            saves text files as UTF-8).</p>
               <section>
                  <h3>TEST-XSPEC</h3>
                  <ul>
                     <li>Where XSpec documents are bound to the input port <code>source</code>, they have
                     <code>content-type='application/xml'</code> given. This is because with the unconventional file
                  suffix <code>xspec</code> (useful for other reasons), the XProc engine needs extra information to know
                  they should be read as XML, not some other data format. Try removing the <code>content-type</code> to
                  see what happens when the engine does not know an XML file is XML.</li>
                     <li>The step <code>p:for-each</code> is not just a step: it also contains steps. It is a <em>compound
                     step</em>. You would be correct to infer this step enables us to perform operations on several
                  inputs in parallel: just what this pipeline needs.</li>
                     <li>Within the <code>p:for-each</code>, the step <code>ox:execute-xspec</code> is named in the
                     <code>ox</code> namespace, which resolves to the string
                     <code>http://csrc.nist.gov/ns/oscal-xproc3</code>, a value assigned for this project. This step is
                  defined in the <a href="../xspec/xspec-execute.xpl">imported pipeline</a>. XProc is indefinitely
                  extensible: the namespace feature allows us to create new steps without fear of name clashes with old
                  steps – or steps that are still uninvented and unnamed. We can develop and name steps in our own
                  namespace, while also acquiring and using steps in other namespaces.</li>
                     <li>The <code>p:identity</code> step is used twice in this pipeline for one purpose only: to indicate
                  messages the XProc engine should deliver. In the normal configuration, you should see these messages
                  in the console when the pipeline runs. This is a common use for <code>p:identity</code>.</li>
                     <li>The repository observes a couple of conventions with regard to steps and messages. For example: any
                     <code>p:load</code> or <code>p:save</code> step should have a message; and messages should always
                  be prefixed with a bracketed indicator of the pipeline that issues them, for example the
                     <code>[TEST-XSPEC]</code> messages that are emitted here, once for each input and again once when
                  the pipeline finishes.</li>
                     <li>Yes, those conventions are enforced in the repository by <a href="../testing/xproc3-house-rules.sch">a Schematron</a> that can be applied to any
                  pipeline, both in development and when it is committed to the repository under <a href="../tutorial/source/walkthrough/walkthrough_301_src.html"
                           class="LessonUnit">CI/CD (continuous integration / continous
                     development)</a>. Assuming we take care to run our tests and validations, this does most of the
                  difficult work maintaining consistency, namely detecting the inconsistency. The result, assuming we do
                  things correctly, is a more-than-human level of consistency in error checking and correction. This is
                  discussed again <a href="../tutorial/source/courseware/courseware_101_src.html"
                           class="LessonUnit">in a subsequent
                     Lesson Unit</a>.</li>
                     <li>Reassuring messages aside, no XSpec reports are actually captured by this XProc! The
                     <code>ox:execute-xspec</code> steps produces results – and produces its own runtime messages – but
                  those results have no <a href="../">connection</a> given in
                  the main pipepline, and it has no output port (<code>p:output</code>). Accordingly the pipeline
                     <em>sinks</em> by default – the documents delivered at the end are discarded. (For smoke test
                  purposes, we care only to see that it runs and completes without error.) The inputs are all
                  controlled, so we know what those reports say. (Or we can find out by altering the pipeline to capture
                  the findings, not discard them.)</li>
                  </ul>
               </section>
               <section>
                  <h3>PRODUCE-PROJECTS-ELEMENTLIST</h3>
                  <p>The pipeline <a href="../tutorial/PRODUCE-PROJECTS-ELEMENTLIST.xpl">PRODUCE-PROJECTS-ELEMENTLIST.xpl</a> has
                  <q>real-world complexity</q>. Reviewing its steps can give a sense of how XProc combines simple
               capabilities into complex operations. Notwithstanding the title of this section, it is not important to
               understand every detail – knowing they are there is enough.</p>
                  <ul>
                     <li>The prologue here contains a single <code>p:input</code> configuration. This one gives the input in
                  line, as an XML document. Within this XML, all the project folders to be covered by the index are
                  listed. Their order also matters since one of the two indexes built works incrementally, prior
                  elements affecting what happens with later elements.</li>
                  </ul>
                  <p>This pipeline exploits one other important feature of XProc: so far at least, it is all XML. Even short
               of of dynamically generating and executing XProc (a feature the language  <a href="../">provides for</a>), the common ground of XML
               format means that with XProc working XML tools, XML and its related ecosystem (most significantly but not
               only XPath) is always available as an instrumental force magnifier or power tool, something we can put to
               work again and again. In XProc, errors are reported in XML. So we can capture, aggregate and index error
               messages in XProc. In XProc, a directory file listing is provided in XML. So we have a ready way to
               present views of file systems – the XML shows all the names and structures – as well as analyze them and
               access the files in them. Etc.</p>
               </section>
            </section>
            <section>
               <h2>Messing around</h2>
               <p>Taking some time to make and test small adjustments to working code is a great way to develop a sense of how
            it behaves.</p>
               <p>An easy way to do this without perturbing the working code in the repository is to copy a pipeline and
            modify the copy. Modifying any of the pipelines presented so far, see what happens when:</p>
               <ul>
                  <li>An <code>@href</code> points to a location on the system where there is no file</li>
                  <li>A file is there, but it is not what is expected (for example: XML is expected but the file is not well
               formed)</li>
                  <li>A <code>p:namespace-delete</code> step is removed from the end of a pipeline – how does the result
               change?</li>
                  <li>Other steps are excluded</li>
                  <li>New elements are renamed (etc.)</li>
               </ul>
               <p>When changes introduce errors, runtime failures and tracebacks will <i>sometimes</i> appear. The indicated
            problem or the source of the reported problem must be repaired.</p>
               <p>And sometimes a process will run successfully, despite an <q>error</q>. Whether it is in error then depends
            on how well it conforms to its requirements. Does it deliver the results we want and expect?</p>
               <p>As an exercise, make some changes in copies of the test pipelines. Make at least one change that produces
            outputs (such as echoing a document to the console) that are visibly different from the results of the
            original pipeline.</p>
               <section>
                  <h3>Disabling your code</h3>
                  <p>For newcomers to XML coding – you can <q>comment out</q> code in any XML by wrapping it in comment
               syntax:</p>
                  <pre>&lt;tagged&gt;Text&lt;/tagged&gt;</pre>
                  <p>becomes:</p>
                  <pre>&lt;!--  &lt;tagged&gt;Text&lt;/tagged&gt; --&gt;</pre>
                  <p>A code editor that supports XML might let you do this with a keystroke, for example <code>ctrl-,</code>
               (Control key plus comma), after selecting the text you wish to include in the comment.</p>
                  <p>Take care when doing this that the XML is still intact with all the tags balanced. This is a very useful
               technique for rapidly and interactively testing your pipelines, by deactivating and reactivating blocks
               of code.</p>
               </section>
            </section>
            <section>
               <h2>Take note</h2>
               <section>
                  <h3>Where are these downloads coming from?</h3>
                  <p>Pipelines can use a few different strategies for resource acquisition, depending on the case, and on
               where and in what form the resource is available. (Sometimes a file on Github is easiest to download
               "raw", sometimes an archive is downloaded and opened, and so on.) For now, it is not necessary to
               understand details in every case, only to observe the variation and range. (With more ideas welcome.
               Could XProc be used to build a <q>secure downloader</q> that knows how, for example, to compare
               hash-based signatures?)</p>
                  <p>Wherever you see <code>href</code> attributes, take note.</p>
                  <p>Since <code>href</code> is how XProc <q>sees</q> the world, either to read data in or to write data out,
               this attribute is a reliable indicator of an assumed feature, often a dependency of some kind. For
               example, a download will not succeed if the resource indicated by the <code>href</code> for the download
               returns an error, or nothing. In XProc, <code>href</code> attribute settings become important <i>points
                  of control</i> for interaction between an XProc pipeline, and its runtime environment.</p>
                  <p>Useful detail: where XProc has <code>p:store href="some-uri.file"</code>, the <code>href</code> is read
               by the processor as the intended location for storage of pipeline data, that is, for a <i>write</i>
               operation. In other cases <code>href</code> is always an argument for a <i>read</i> operation.</p>
               </section>
               <section>
                  <h3>Syntax tips</h3>
                  <p>In XPath syntax, <code>$foo</code> (a name with a <code>$</code> prefixed) indicates a <b>variable
                  reference</b> named (in this case) <q>foo</q>. XProc also uses a <i>value expansion syntax</i>
                  (either<i>text value syntax</i> or <i>attribute value syntax</i>) using curly braces - so syntax such
               as <code>href="{$some-xml-uri}"</code> is not uncommon. Depending on use, this would mean <q>read [or
                  write] to the URI given by <code>$some-xml-uri</code>
                     </q>.</p>
                  <p>An XProc developer always knows where <code>href</code> is used in a pipeline, and how to test for and
               update its use. As always with syntax, the easiest way to learn it is to try making changes and observing
               outcomes.</p>
               </section>
            </section>
         </section>
         <section class="unit learner"
                  id="walkthrough_219"
                  data-track="learner">
            <h1>219: XProc, XML and XDM  (the XML Data Model)</h1>
            <section>
               <h2>Goals</h2>
               <ul>
                  <li>Consider XProc in its operational context including available <b>standards</b> and applicable
                  <b>requirements</b>, both generalized and local</li>
                  <li>Consider the context of XProc by learning or relearning some deep XML history</li>
                  <li>Inform your capability to assess the utility and appropriateness of XProc in particular and XML in
               general, for a given problem or domain</li>
               </ul>
            </section>
            <section>
               <h2>Resources</h2>
               <p>The same pipelines you ran in setup: <a href="../tutorial/source/acquire/acquire_101_src.html"
                     class="LessonUnit">Setup
            101</a>.</p>
               <p>Also, <a href="../">XProc.org dashboard page</a>.</p>
               <p>Also, XProc index materials produced in this repository: <a href="../projects/xproc-doc/readme.md">XProc docs</a>.</p>
            </section>
            <section>
               <h2>XProc as XML</h2>
               <p>XProc is defined as an XML vocabulary. A schema for the XProc language, considered as core steps (compound
            and atomic) plus optional community-defined steps, is referenced from the <a href="../">XProc Specification</a>. <a href="../">This RNG schema</a> is very useful.</p>
               <p>It may often be considered gratuitous to validate XProc files against a schema, when the application
            (whether it be Morgana, XML Calabash or other) must in any case take responsibility for conformance issues,
            as it sees fit. The reference schema becomes useful if we find or suspect bugs in processor features, but
            until then it need not have any direct role in any runtime operation.</p>
               <p>Nevertheless, since XProc is XML, its schema still serves as a reference and an object for querying –
            queries whose results tell us about XProc. <a href="../tutorial/GRAB-XPROC-RESOURCES.xpl">A pipeline</a> for
            acquiring both the RNG schema. and its RNC (compact syntax) variant. are provided for interest and possible
            later use.</p>
               <section>
                  <h3>Survey of XProc elements</h3>
                  <p>All elements defined by XProc (at time of writing) are listed in this analytical breakout.</p>
                  <p>
                     <i>However</i>, this list is provisional and intended only to offer a wider view. For the most up to date
               information, refer to specifications. In particular, XProc 3.1 makes this list a moving target. This may
               be out of date by the time you are able to read it.</p>
                  <p> NB also – see Erik Siegel's <a href="../">XProcRef</a> indexing project for
               more detailed summaries.</p>
                  <table>
                     <tbody>
                        <tr>
                           <th>Function</th>
                           <th>XProc elements / p: namespace</th>
                        </tr>
                        <tr>
                           <td>Documentation</td>
                           <td>
                              <code>p:documentation</code>
                           </td>
                        </tr>
                        <tr>
                           <td>Top-level</td>
                           <td>
                              <code>p:declare-step</code>, <code>p:library</code>
                           </td>
                        </tr>
                        <tr>
                           <td>Imports</td>
                           <td>
                              <code>p:import</code>, <code>p:import-functions</code>
                           </td>
                        </tr>
                        <tr>
                           <td>Prologue</td>
                           <td>
                              <code>p:input</code>, <code>p:output</code>, <code>p:option</code>
                           </td>
                        </tr>
                        <tr>
                           <td>Compound steps</td>
                           <td>
                              <code>p:group</code>, <code>p:for-each</code>, <code>p:viewport</code>, <code>p:if</code>,
                        <code>p:choose</code> (with <code>p:when</code> and <code>p:otherwise</code>),
                        <code>p:try</code> (with <code>p:catch</code> and <code>p:finally)</code>, <code>p:run</code>
                     (with <code>p:run-input</code>, <code>p:run-option</code>)</td>
                        </tr>
                        <tr>
                           <td>Atomic steps - core - XML</td>
                           <td>
                              <code>p:add-attribute</code>, <code>p:add-xml-base</code>, <code>p:delete</code>,
                        <code>p:filter</code>, <code>p:identity</code>, <code>p:insert</code>,
                        <code>p:label-elements</code>, <code>p:make-absolute-uris</code>,
                        <code>p:namespace-delete</code>, <code>p:namespace-rename</code>, <code>p:pack</code>,
                        <code>p:rename</code>, <code>p:replace</code>, <code>p:set-attributes</code>,
                        <code>p:uuid</code>, <code>p:unwrap</code>, <code>p:wrap-sequence</code>, <code>p:wrap</code>,
                        <code>p:xinclude</code>, <code>p:xquery</code>, <code>p:xslt</code>
                           </td>
                        </tr>
                        <tr>
                           <td>Atomic steps - core - zipping</td>
                           <td>
                              <code>p:archive</code>, <code>p:archive-manifest</code>, <code>p:unarchive</code>,
                        <code>p:uncompress</code>
                           </td>
                        </tr>
                        <tr>
                           <td>Atomic steps - core - JSON</td>
                           <td>
                              <code>p:json-join</code>, <code>p:json-merge</code>, <code>p:set-properties</code>
                           </td>
                        </tr>
                        <tr>
                           <td>Atomic steps - core - plain text</td>
                           <td>
                              <code>p:string-replace</code>, <code>p:text-count</code>, <code>p:text-head</code>,
                        <code>p:text-join</code>, <code>p:text-replace</code>, <code>p:text-sort</code>,
                        <code>p:text-tail</code>
                           </td>
                        </tr>
                        <tr>
                           <td>Atomic steps - core - utility</td>
                           <td>
                              <code>p:cast-content-type</code>, <code>p:compare</code>, <code>p:compress</code>,
                        <code>p:count</code>, <code>p:error</code>, <code>p:hash</code>, <code>p:http-request</code>,
                        <code>p:load</code>, <code>p:sink</code>, <code>p:split-sequence</code>, <code>p:store</code>,
                        <code>p:www-form-urldecode</code>, <code>p:www-form-urlencode</code>
                           </td>
                        </tr>
                        <tr>
                           <td>Atomic steps - optional - file system</td>
                           <td>
                              <code>p:directory-list</code>, <code>p:file-copy</code>, <code>p:file-delete</code>,
                        <code>p:file-info</code>, <code>p:file-mkdir</code>, <code>p:file-move</code>,
                        <code>p:file-create-tempfile</code>, <code>p:file-touch</code>
                           </td>
                        </tr>
                        <tr>
                           <td>Atomic steps - optional - validation</td>
                           <td>
                              <code>p:validate-with-nvdl</code>, <code>p:validate-with-relax-ng</code>,
                        <code>p:validate-with-schematron</code>, <code>p:validate-with-xml-schema</code>,
                        <code>p:validate-with-json-schema</code>
                           </td>
                        </tr>
                        <tr>
                           <td>Other optional steps</td>
                           <td>
                              <code>p:os-info</code>, <code>p:os-exec</code>, <code>p:css-formatter</code>,
                        <code>p:xsl-formatter</code>, <code>p:markdown-to-html</code>
                           </td>
                        </tr>
                        <tr>
                           <td>Variable declaration</td>
                           <td>
                              <code>p:variable</code>
                           </td>
                        </tr>
                        <tr>
                           <td>Connectors</td>
                           <td>
                              <code>p:with-input</code>, <code>p:with-option</code>, <code>p:pipe</code>,
                        <code>p:pipeinfo</code>, <code>p:document</code>, <code>p:inline</code>,
                     <code>p:empty</code>
                           </td>
                        </tr>
                     </tbody>
                  </table>
               </section>
            </section>
            <section>
               <h2>XML and the XML Data Model (XDM): context and rationale</h2>
               <p>XML (Extensible Markup Language) is a text-based data format. XDM (the XML Data Model) is an abstract object
            model defining types and interfaces. XDM is designed to provide tools that work with XML data with a common
            model, permitting the description and specification of arbitrary operations of instances. XDM provides the
            conceptual foundation of XPath, XSLT and XQuery.</p>
               <p>These technologies have been developed under the auspices of the World Wide Web Consortium (W3C) and other
            standards bodies since before the publication of XML in 1998. This longevity is not an accident: for
            stability over a long term (years and decades rather than months), we have required solutions that are:</p>
               <ul>
                  <li>Standard, non-proprietary and freely available without restriction</li>
                  <li>Consistently and repeatedly shown to be capable at scale (size/complexity)</li>
                  <li>Supported by commodity tools, easing problems of proprietary product dependencies</li>
               </ul>
               <p>Importantly, we need tools that are freely available to use without restriction, an important qualification
            for this distribution, which has a prior commitment <i>not to endorse particular technological solutions to
               any problem</i>, however posed or circumscribed. Accordingly, solutions here are not offered as
            recommendations, but rather as stipulations of (minimum) viable functionality in tools or capabilities, and
            not only using tools as <q>black boxes</q>, but under control and conformant to external specifications –
            i.e., standards.</p>
               <p>Users should keep in mind the model whereby we imagine the viability of a tools market and ecosystem that
            enables both large and small software developers – including independent developers, academic researchers,
            and students – to participate meaningfully, finding an appropriate value or service proposition to support
            immediate and long-term goals. Translated, this means the tools must be capable enough for industrial use at
            scale, while they must also <q>scale down</q> to demonstration or classroom use.</p>
               <p>In web standards including HTML and Javascript (ECMAScript) we arguably have the beginnings of such an
            ecosystem, while it is also contested and turbulent. Within the publishing sector more broadly and
            intersecting with the web, the XML family of standards arguably provides the best demonstration of complete
            or near-complete capabilities at least with respect to the harder problems of document processing.</p>
               <ul>
                  <li>XSLT up to <a href="../">XSLT 3.0</a> (in <a href="../">Saxon</a>)</li>
                  <li>
                     <a href="../">XQuery</a> (in Saxon)</li>
                  <li>
                     <a href="../">Schematron</a> (in <a href="../">SchXSLT</a>, an open-source implementation in XSLT of <a href="../">Schematron</a> including the <a href="../">ISO/IEC 19757-3</a>
               specification</li>
                  <li>
                     <a href="../">XSpec</a>, a community-maintained XSLT-based framework for
               test-driven development, supporting testing XSLT, XQuery and Schematron</li>
               </ul>
               <p>Since they are known to be highly conformant to their respective specifications as well as well tested,
            these tools provide a useful functional baseline for evaluating other tooling that addresses the same
            functional requirements.</p>
               <p>They are also, relatively speaking, <i>mature</i> technologies, at least in comparison to similar
            offerings.</p>
               <p>And when XProc works, we also have the functional underpinnings we need for comparing – for example –
            different XSLT implementations.</p>
               <p>Initiated in 1996, XML continues to be generative in 2024.</p>
            </section>
            <section>
               <h2>Snapshot history: an XML time line</h2>
               <p>[TODO] Evidently we could use help completing and fact checking all this...</p>
               <table>
                  <thead>
                     <tr>
                        <th>Year</th>
                        <th>Publication</th>
                        <th>Capabilities</th>
                        <th>Processing frameworks</th>
                        <th>Platforms</th>
                     </tr>
                  </thead>
                  <tbody>
                     <tr>
                        <td>1987</td>
                        <td>SGML (ISO-IEC 8879-1)</td>
                        <td>parsing logic; schema validation; configurable syntax; (implicit) tree of elements and
                     attributes</td>
                        <td>Proprietary stacks</td>
                        <td>Mainframes, workstations</td>
                     </tr>
                     <tr>
                        <td>1996</td>
                        <td>Unicode 2.0</td>
                        <td>standard character sets</td>
                        <td>… support for Unicode is slow to come</td>
                        <td>PCs</td>
                     </tr>
                     <tr>
                        <td>1998</td>
                        <td>XML 1.0</td>
                        <td>standard syntax</td>
                        <td>Batch processing, shell scripts, <code>make</code>
                        </td>
                        <td>Mainframes, workstations, PCs (x86 generation), *nix, shell, sed/awk, Perl</td>
                     </tr>
                     <tr>
                        <td>1999</td>
                        <td>XPath 1.0, XSLT 1.0</td>
                        <td>basic tree querying and transformations (<q>down hill</q>); functional support for namespaces</td>
                        <td>Web browsers? (some, sort of); standalone XSLT processors</td>
                        <td>
                        </td>
                     </tr>
                     <tr>
                        <td>2000</td>
                        <td>
                        </td>
                        <td>XML-configured software builds</td>
                        <td>Apache Ant</td>
                        <td>Java</td>
                     </tr>
                     <tr>
                        <td>
                        </td>
                        <td>
                        </td>
                        <td>
                        </td>
                        <td>
                        </td>
                        <td>Perl, Python</td>
                     </tr>
                     <tr>
                        <td>
                        </td>
                        <td>XQuery 1.0</td>
                        <td>
                        </td>
                        <td>
                        </td>
                        <td>
                        </td>
                     </tr>
                     <tr>
                        <td>
                        </td>
                        <td>XPath 2.0</td>
                        <td>
                        </td>
                        <td>Server frameworks (Apache Cocoon)</td>
                        <td>
                        </td>
                     </tr>
                     <tr>
                        <td>2001</td>
                        <td>XML Schema Definition language (XDM)</td>
                        <td>Standardizes atomic data types (foundations of XSD); namespace-based validation (RNG also offers
                     this, 2001-2002)</td>
                        <td>
                        </td>
                        <td>
                        </td>
                     </tr>
                     <tr>
                        <td>2001-2003</td>
                        <td>(Conference papers)</td>
                        <td>Early pipelining demonstrations</td>
                        <td>
                        </td>
                        <td>
                        </td>
                     </tr>
                     <tr>
                        <td>2003</td>
                        <td>
                        </td>
                        <td>Pipelining as a build process</td>
                        <td>Apache Ant</td>
                        <td>Java</td>
                     </tr>
                     <tr>
                        <td>2003-2004</td>
                        <td>W3C Document Object Model (DOM)</td>
                        <td>API for HTML and XML documents</td>
                        <td>
                        </td>
                        <td>
                        </td>
                     </tr>
                     <tr>
                        <td>2005</td>
                        <td>
                           <q>The XML data model</q> (W3C)</td>
                        <td>
                           <a href="../">An essay</a>
                        </td>
                        <td>
                        </td>
                        <td>
                        </td>
                     </tr>
                     <tr>
                        <td>2006</td>
                        <td>XProc 1.0 Requirements</td>
                        <td>
                        </td>
                        <td>
                        </td>
                        <td>Proof-of-concept demonstrations</td>
                     </tr>
                     <tr>
                        <td>2007</td>
                        <td>XSLT 2.0</td>
                        <td>Transformations (<q>up hill</q>) including grouping, string processing, pipelining</td>
                        <td>
                        </td>
                        <td>
                        </td>
                     </tr>
                     <tr>
                        <td>
                        </td>
                        <td>XDM (XPath/XQuery data model)</td>
                        <td>Unifying a data model for XPath, XSLT and XQuery</td>
                        <td>
                        </td>
                        <td>Client- and server-side XML processing stacks</td>
                     </tr>
                     <tr>
                        <td>
                        </td>
                        <td>
                        </td>
                        <td>
                        </td>
                        <td>XQuery+XSLT in eXist-db or BaseX (XQuery engines)</td>
                        <td>
                        </td>
                     </tr>
                     <tr>
                        <td>2010</td>
                        <td>XProc 1.0</td>
                        <td>Standard vocabulary for XDM-based pipelining</td>
                        <td>
                        </td>
                        <td>
                        </td>
                     </tr>
                     <tr>
                        <td>2014</td>
                        <td>XPath 3.0</td>
                        <td>JSON interchange support</td>
                        <td>
                        </td>
                        <td>
                        </td>
                     </tr>
                     <tr>
                        <td>2017</td>
                        <td>XSLT 3.0/3.1</td>
                        <td>JSON harmonization; higher-order functions; map and array objects</td>
                        <td>
                        </td>
                        <td>
                        </td>
                     </tr>
                     <tr>
                        <td>2022</td>
                        <td>XProc 3.0</td>
                        <td>Improves on XProc 1.0 with easier syntax, plus adding support for JSON and other content
                     types</td>
                        <td>
                        </td>
                        <td>
                        </td>
                     </tr>
                     <tr>
                        <td>2022</td>
                        <td>Unicode 15.0</td>
                        <td>
                        </td>
                        <td>
                        </td>
                        <td>
                        </td>
                     </tr>
                  </tbody>
               </table>
               <p>The technologies have been in constant use over this period.</p>
               <p>Historically, the requirements of processing frameworks have often been met by software developers' build
            utilities (for example, GNU <code>make</code> or Apache Ant). This is not an accident: in certain respects,
            a publishing framework can be considered as a <q>documentary build</q>, to be run at intervals corresponding to the publishing cycle with its proof runs.</p>
            </section>
            <section>
               <h2>XPath</h2>
               <p>Like other XDM-based technologies, XProc embeds and incorporates XPath, an expression language for XML.
            XPath 3.0 is a functional language in its own right, although not designed for end-to-end processing of
            encoded source data into encoded results, but only for certain critical operations that ordinarily need to
            be performed within such end-to-end processing. Importantly, XPath is defined not in terms of any data
            notation (such as XML syntax or any other) but rather against an <i>abstract data object</i>, namely an <a href="../">XDM</a> instance (XML data model), a putative information
            object that may be provided to the system by parsing an XML (syntax) instance, or by other means. As the
            query language for <a href="../">XDM</a> and the basis for XQuery, <a href="../">XPath</a> is the <q>other half</q> of the data model, which any
            architect of a system using this technology must know. Learning XPath equips you mentally for dealing with
            the XDM in XQuery, XSLT, XProc or anywhere you find it.</p>
               <p>For those not already familiar with XPath, on line resources can be helpful. Keep in mind that <a href="../">XPath 3.1</a> outstrips earlier versions of the language in many
            important respects, supporting map and function objects with higher-order functions among other
            features.</p>
               <section>
                  <h3>Documents and data</h3>
                  <p>One of the more important features of XPath and the XDM is that they are designed not only to meet needs
               for the representation and transmission of structured data. A specialized class of data formats has
               evolved that represent information in ways that are not <q>unstructured</q>, but that contrast with more
               common or usual structures of data formats, whether they be tabular data, serialization formats for
               object models, or some other regular (formalized and codified) arrangement for purposes of
               machine-readability. One might say <q>common</q> or <q>usual</q> with reservation, since of course
               documents are not uncommon where they are common. The predominance of so-called <q>structured data</q> in
               digital systems may tell us more about the limits of those systems, considered within their historical
               and evolutionary context, than it does about information in general.</p>
                  <p>At the same time it does not seem like a mistake to identify <q>information</q> with <q>structure</q>, or
               at least to observe there is a mutual and perhaps definitional interdependence there.</p>
                  <p>We see a great deal of structured data these days if only because it is so easy to make structured data
               with machines, and we now have the machines. What remains difficult is to translate something that has
                  <i>not</i> been created by (or only by) a machine – perhaps, for example, it results from a more
                  <q>organic</q> process of development (maybe a Shakespeare play?) – into a form that a machine can
                  <q>recognize</q>, or rather into a form we can recognize in and with the machine, without mishandling
               it and distorting it. Since machines do not recognize anything (nothing is <q>mishandling</q> to them),
               what this often reduces to in practice is deciding how to agree on a <b>representation</b> for
               information that any creator and any consumer can recognize and work with, without seeing the information
               first. In itself this is a formidable challenge.</p>
                  <p>So documents are called <q>unstructured</q> but they might better be called <q>relatively irregular</q>,
               meaning not that they have no structure, but that each one is structured in itself, and moreover, likely
               to be incompatible or not fully compatible with encodings designed to capture other structures.</p>
                  <p>And to the extent this is the case, any encoding capable of describing documents must have the capability
               of supporting each document's own distinctive structure and organization, whether that be due to its
               family (what is called a <b>document type</b>) or an expression of its own intrinsic logic. The format
               must be not only structured, but <i>structurable</i>, and its structures must to some extent be capable
               of self-description – combining data with metadata.</p>
                  <p>And this is to give no consideration to the fact that these structures can be described at <i>multiple
                  levels</i> of generality or specificity with regard to either their supposed semantics, or their
               configuration in operation.</p>
                  <p>Documentary data formats, especially declarative markup formats, are designed to work in this in-between
               space.</p>
                  <p>And so we get XPath - a query syntax which permits working with an organized structure of a particular
               kind (an <i>XDM document tree</i>), which in turn is designed for handling the combination of <i>highly
                  regular</i> and <i>quite irregular</i> data structures that characterize information sets we (loosely)
               call <b>documentary</b>.</p>
               </section>
               <section>
                  <h3>XPath illustrative examples</h3>
                  <p>This is not the place to learn XPath, but a selection of XPath expressions can offer a hint of its
               capabilities.</p>
                  <table>
                     <thead>
                        <tr>
                           <th>XPath</th>
                           <th>Returns</th>
                           <th>XPath long (explicit) notation</th>
                        </tr>
                     </thead>
                     <tbody>
                        <tr>
                           <td>
                              <code>/html</code>
                           </td>
                           <td>An XML document root (top-level) element named <code>html</code> (subject to namespace
                        resolution)</td>
                           <td>
                              <code>/child::html</code>
                           </td>
                        </tr>
                        <tr>
                           <td>
                              <code>//p</code>
                           </td>
                           <td>All the elements named <code>p</code> in the document</td>
                           <td>
                              <code>/descendant-or-self::element()/ child::p</code>
                           </td>
                        </tr>
                        <tr>
                           <td>
                              <code>//seg[@type='null']</code>
                           </td>
                           <td>All the elements named <code>seg</code> with an attribute <code>type</code> with value
                           <code>null</code>
                           </td>
                           <td>
                              <code>/descendant-or-self::element()/ child::seg[attribute::type='null']</code>
                           </td>
                        </tr>
                        <tr>
                           <td>
                              <code>/*</code>
                           </td>
                           <td>Any document (rather, any element at the top of a document) - <code>*</code> is a wildcard
                        character</td>
                           <td>
                              <code>/child::element()</code>
                           </td>
                        </tr>
                        <tr>
                           <td>
                              <code>/section[exists(.//table)]</code>
                           </td>
                           <td>An element inside the top-level element, named <code>section</code>, that contains a
                           <code>table</code> element anywhere inside it</td>
                           <td>
                              <code>/child::section[exists(self::node()/ descendant-or-self::element()/ child::table)]</code>
                           </td>
                        </tr>
                        <tr>
                           <td>
                              <code>/descendant::p[10]</code>
                           </td>
                           <td>The tenth <code>p</code> element in the document</td>
                           <td>
                              <code>/descendant::p[position() eq 10]</code>
                           </td>
                        </tr>
                        <tr>
                           <td>
                              <code>//p[10]</code>
                           </td>
                           <td>All <code>p</code> elements, that are the tenth <code>p</code> inside their respective
                        parents</td>
                           <td>
                              <code>/descendant-or-self::element()/ child::p[position() eq 10]</code>
                           </td>
                        </tr>
                        <tr>
                           <td>
                              <code>//section[count(.//p) gt 10]</code>
                           </td>
                           <td>All <code>section</code> elements that contain more than 10 <code>p</code> elements, at any
                        depth</td>
                           <td>
                              <code>/child::section[count(self::node()/ descendant-or-self::element()/ child::p) gt
                        10]</code>
                           </td>
                        </tr>
                     </tbody>
                  </table>
                  <p>Where do you find XPath? Any <code>select</code> or <code>match</code> expression in XSLT or XProc shows
               an example. XPath also constitutes the core of XQuery.</p>
               </section>
            </section>
            <section>
               <h2>Exercise: Discussion board</h2>
               <p>Create or contribute to a Github discussion board offering perspective or (especially!) relevant information
            or experience on any of the larger questions.</p>
            </section>
         </section>
         <section class="unit observer"
                  id="walkthrough_301"
                  data-track="observer">
            <h1>301: Automated XProc</h1>
            <section>
               <h2>Goals</h2>
               <ul>
                  <li>See how XProc supports software testing, including testing itself, supportive of test-driven development
               (TDD)</li>
                  <li>Exposure to the configuration of the Github repository supporting dynamic testing on Pull Requests and
               releases, subject to extension</li>
               </ul>
            </section>
            <section>
               <h2>Prerequisites</h2>
               <p>You have made it this far.</p>
            </section>
            <section>
               <h2>Resources</h2>
               <p>Prepare to run pipelines in the <a href="../testing/readme.md">testing directory</a> as described
            below.</p>
               <ul>
                  <li>
                     <a href="../testing/REPO-FILESET-CHECK.xpl">Repository file set check XProc</a>
                  </li>
                  <li>
                     <a href="../testing/VALIDATION-FILESET-READYCHECK.xpl">Validation file set ready check
               XProc</a>
                  </li>
                  <li>
                     <a href="../testing/RUN_XPROC3-HOUSE-RULES_BATCH.xpl">House Rules check on defined XProc3
                  FILESET</a>
                  </li>
                  <li>
                     <a href="../testing/REPO-XPROC3-HOUSE-RULES.xpl">House Rules check on all XProc3 files, found
                  dynamically</a>
                  </li>
                  <li>
                     <a href="../testing/RUN_XSPEC_BATCH.xpl">Run all XSpecs listed in the XSpec FILESET</a>
                  </li>
               </ul>
               <p>Additionally, the <a href="../.github/workflows/test.yml">testing configuration file</a> configured to
            run under Github Actions will potentially be of interest.</p>
            </section>
            <section>
               <h2>XProc for quality testing</h2>
               <p>As currently configured, the repository provides two kinds of testing, both supported under XProc 3.0:</p>
               <ul>
                  <li>Schematron validations providing quality checks are applied to XProcs - the so-called <em>House
                  Rules</em> for XProc files in this repository.</li>
                  <li>
                     <em>XSpec tests</em> are run to provide functional testing, unit testing and regression testing for
               XSLT, Schematron or XQuery, as needed</li>
               </ul>
               <p>Both kinds of tests can be configured and executed using XProc. Pipelines here provide for such executions
            in useful ways, both in local copies of the repository under development, and under automation, such as by
            Github Actions in this repository.</p>
               <p>Specifically, tests that are run anytime a Pull Request is updated against the home repository serve to
            guard against accepting non-functional code into the repository code base.</p>
               <p>The tests themselves are so far fairly rudimentary – while paying for themselves in the consistency and
            quality they help enforce.</p>
               <section>
                  <h3>Pipelines useful for the developer:</h3>
                  <ul>
                     <li>
                        <a href="../testing/VALIDATION-FILESET-READYCHECK.xpl">VALIDATION-FILESET-READYCHECK.xpl</a>
                  runs a pre-check to validate that files referenced in FILESET Xprocs are in place</li>
                     <li>
                        <a href="../testing/REPO-FILESET-CHECK.xpl">REPO-FILESET-CHECK.xpl</a> for double checking the
                  listed FILESET pipelines against the repository itself - run this preventatively to ensure files are
                  not left off either list inadvertantly</li>
                     <li>
                        <a href="../testing/RUN_XPROC3-HOUSE-RULES_BATCH.xpl">RUN_XPROC3-HOUSE-RULES_BATCH.xpl</a>
                  applies House Rules Schematron to all XProcs listed in the House Rules FILESET - just like the
                  HARDFAIL House Rules pipeline except ending gracefully with error reports</li>
                     <li>
                        <a href="../testing/REPO-XPROC3-HOUSE-RULES.xpl">REPO-XPROC3-HOUSE-RULES.xpl</a> applies House
                  Rules Schematron to all XProc documents in the repository</li>
                     <li>
                        <a href="../testing/RUN_XSPEC_BATCH.xpl">RUN_XSPEC_BATCH.xpl</a> runs all XSpecs listed in the
                  XSpec FILESET, in a single batch, saving HTML and JUnit test results</li>
                  </ul>
               </section>
               <section>
                  <h3>Pipelines run under CI/CD</h3>
                  <ul>
                     <li>
                        <a href="../testing/HARDFAIL-XPROC3-HOUSE-RULES.xpl">HARDFAIL-XPROC3-HOUSE-RULES.xpl</a> runs a
                  pipeline enforcing the House Rules Schematron to every XProc listed in the imported FILESET pipeline,
                  bombing (erroring out) if an error is found - useful when we want to ensure an ERROR condition comes
                  back on an error reported by a <i>successful</i> Schematron run</li>
                     <li>
                        <a href="../testing/RUN_XSPEC-JUNIT_BATCH.xpl">RUN_XSPEC-JUNIT_BATCH.xpl</a> runs all XSpecs
                  listed in the XSpec FILESET, saving only JUnit results (no HTML reports)</li>
                  </ul>
               </section>
               <section>
                  <h3>File set listings as step declarations</h3>
                  <p>These pipelines are used only as components in other pipelines, which import them. They are used to
               provide central control of file listings for batching purposes (process or validation) - i.e., other
               pipelines can access these steps to get to the named resources. They can also be validated externally,
               for early detection of broken links.</p>
                  <ul>
                     <li>
                        <a href="../testing/FILESET_XPROC3_HOUSE-RULES.xpl">FILESET_XPROC3_HOUSE-RULES.xpl</a> provides
                  a list of resources (documents) to be made accessible to importing pipelines</li>
                     <li>
                        <a href="../testing/FILESET_XSPEC.xpl">FILESET_XSPEC.xpl</a> provides a list of XSpec files to
                  be run under CI/CD</li>
                  </ul>
               </section>
               <section>
                  <h3>About the XProc House Rules</h3>
                  <p>Read more about this in <a href="../testing/house-rules.md">project documentation</a>, and check
               out the <a href="../testing/xproc3-house-rules.sch">Schematron source code</a>. This set of rules
               helps to ensure smooth operations across projects.</p>
                  <p>A good example of a rule enforced over the XProc with this rules set: we never use XProc
                  <code>p:store</code> without emitting a message to the console (or runtime) announcing it. This aids
               in transparency by alerting operators when files are produced and written. As a courtesy, the same rule
               is applied to <code>p:load</code> so that an operator knows when data sets are loaded dynamically (as
               opposed to bound to an input port).</p>
                  <p>More routinely, the House Rules checks include link integrity checks, making it possible to detect and
               correct broken links as soon as they appear.</p>
               </section>
               <section>
                  <h3>About XSpec testing in this repository</h3>
                  <p>This project supports XSpec since experience shows that <q>serious</q> development of robust and general
               applications must entail unit testing, and we consider XSpec to be the appropriate choice for unit
               testing XSLT and XQuery. We wholly endorse the goals of test-driven development and consider automated
               testing including functional testing and unit testing to be essential to the successful development and
               maintenance of pipelines with real-world complexity.</p>
                  <p>At the same time since the focus of development is on XProc, testing of subordinate XSLT and Schematron
               logic has often been neglected in favor of priorities.</p>
                  <p>The <a href="../testing/FILESET_XSPEC.xpl">XSpec FILESET</a> will show XSpecs run under CI/CD, but
               not all XSpecs in the repository will be listed there. Some projects may have XSpec of their own, to be
               used with or without XProc.</p>
               </section>
            </section>
            <section>
               <h2>XProc running under continuous integration and development (CI/CD)</h2>
               <p>Any XProc pipelines designed, like the smoke tests or validations just described, to provide for quality
            checking over carefully maintained code bases, are natural candidates for running dynamically and on demand,
            for example when file change commits are made to git repositories under CI/CD (continuous integration /
            continuous deployment).</p>
               <p>Consider this a ready-made framework for all kinds of testing including functional unit testing, regression
            testing and conformance testing as defined by traceable inputs and outputs, especially well suited for
            testing XSLT-, XQuery/XPath- or XProc-based solutions as well – to say nothing of testing and validation of
            applications (including validation applications!) provided as parts of such solutions.</p>
               <p>The <a href="../.github/workflows/test.yml">Github Actions file configuring testing</a> in this repo
            show how this can be done.</p>
               <p>At this time, there are no plans to use this feature for its most obvious purpose, namely dynamic
            publishing-of edited and curated source contents into readable results (such as HTML, SVG, PDF or other).
            Putting the tutorial production XProc pipelines into the Github Actions command sequence would have this
            effect.</p>
            </section>
         </section>
         <section class="unit learner"
                  id="walkthrough_401"
                  data-track="learner">
            <h1>401: The XSLT Factor</h1>
            <section>
               <h2>Goals</h2>
               <p>What is this XSLT? Read this page for important background and context:</p>
               <ul>
                  <li>If you don't know XSLT and do not care to, consider skimming to help you understand what is XSLT and
               what it does.</li>
                  <li>If you know XSLT or plan to learn it, read to understand something more about how it fits with
               XProc.</li>
                  <li>XQuery is also mentioned. Much of what is said about XSLT here applies to XQuery as well.</li>
               </ul>
               <p>XSLT offers XProc a core capability. Even if not always indispensable, what it brings is important and
            frequently necessary, helping XProc to address problems with real-world complexity that evolve – or are only
            revealed – over time. It would be unfair to introduce developers or proprietors of data processing systems
            to XProc without offering some sense of XSLT and its uses and strengths.</p>
            </section>
            <section>
               <h2>Prerequisites</h2>
               <p>You have run and inspected pipelines mentioned earlier, such as <a href="../tutorial/PRODUCE-PROJECTS-ELEMENTLIST.xpl">PRODUCE-PROJECTS-ELEMENTLIST</a>, which contain
               <code>p:xslt</code> steps. In any case the idea of a <q>transformation</q> of one data structure into
            another is not new.</p>
               <p>Possibly, you have also inspected XSLT files (standalone transformations or <em>stylesheets</em>), to be
            found more or less anywhere in this repository, especially directories named <code>src</code>, with the file
            suffix <code>xsl</code> by convention. (XSLT being a part of XSL.)</p>
            </section>
            <section>
               <h2>Resources</h2>
               <p>XSLT links! Absorbing these documents is not necessary; but you need to know they exist. These provide the
            basis and history of the XML Data Model (XDM), the foundation of XProc.</p>
               <section>
                  <h3>XSLT 1.0 and XPath 1.0</h3>
                  <p>This <q>Original Gangster</q> (OG) version is still available in browsers, and still capable, albeit not
               as general or powerful as it was to become.</p>
                  <ul>
                     <li>
                        <a href="../">XML Path Language (XPath) Version 1.0</a>
                  W3C Recommendation 16 November 1999</li>
                     <li>
                        <a href="../">XSL Transformations (XSLT) Version 1.0</a> W3C
                  Recommendation 16 November 1999</li>
                  </ul>
               </section>
               <section>
                  <h3>XSLT 2.0 and XQuery 1.0</h3>
                  <p>With capabilities for grouping, better string processing (regular expressions), a more extensive type
               system aligned with XQuery, <em>temporary trees</em> (to reprocess results) and other needed features,
               XSLT 2.0 was widely deployed in document production back-ends, and used successfully within XProc
               1.0.</p>
                  <p>The only reason not to use it today is that XSLT 3.0/3.1 and XQuery 3.0 are available. The 2.0
               technologies are still viable for developers using tools supporting that generation, while providing a
               basis for forward migration.</p>
                  <ul>
                     <li>
                        <a href="../">XSL Transformations (XSLT) Version 2.0 (Second Edition)</a>
                  W3C Recommendation 30 March 2021 (Amended by W3C)</li>
                     <li>
                        <a href="../">XQuery 1.0: An XML Query Language (Second Edition)</a> W3C
                  Recommendation 14 December 2010</li>
                     <li>World Wide Web Consortium.&nbsp;<em>XQuery 1.0 and XPath 2.0 Data Model (XDM) (Second Edition)</em>. W3C
                  Recommendation, 14 December 2010. See&nbsp;<a href="../"
                           style="color: rgb(0, 0, 204); background: transparent;">http://www.w3.org/TR/xpath-datamodel/</a>.</li>
                     <li>World Wide Web Consortium.&nbsp;<em>XQuery 1.0 and XPath 2.0 Formal Semantics (Second Edition)</em>. W3C
                  Recommendation, 14 December 2010. See&nbsp;<a href="../"
                           style="color: rgb(0, 0, 204); background: transparent;">http://www.w3.org/TR/xquery-semantics/</a>.</li>
                     <li>World Wide Web Consortium.&nbsp;<em>XQuery 1.0 and XPath 2.0 Functions and Operators (Second
                  Edition)</em>&nbsp;W3C Recommendation, 14 December 2010. See&nbsp;<a href="../"
                           style="color: rgb(0, 0, 204); background: transparent;">http://www.w3.org/TR/xpath-functions/</a>.</li>
                     <li>World Wide Web Consortium.&nbsp;<em>XSLT 2.0 and XQuery 1.0 Serialization (Second Edition)</em>. W3C
                  Recommendation, 14 December 2010. See&nbsp;<a href="../"
                           style="color: rgb(0, 0, 204); background: transparent;">http://www.w3.org/TR/xslt-xquery-serialization/</a>.</li>
                  </ul>
               </section>
               <section>
                  <h3>XSLT 3.0, XQuery 3.0, XPath 3.1</h3>
                  <p>The current generation of the language – although work progresses on XPath 4.0, more capable than
               ever.</p>
                  <ul>
                     <li>
                        <a href="../">XSL Transformations (XSLT) Version 3.0</a> W3C
                  Recommendation 8 June 2017</li>
                     <li>
                        <a href="../">Normative references</a> for XSLT 3.0 -
                  data model, functions and operators, etc., including <b>XPath 3.1</b>
                     </li>
                     <li>
                        <a href="../">XQuery 3.0: An XML Query Language</a> W3C Recommendation
                  08 April 2014</li>
                  </ul>
               </section>
            </section>
            <section>
               <h2>XSLT: XSL (XML Stylesheet Language) Transformations</h2>
               <p>XSLT has a long and amazing history to go with its checkered reputation. Its role in XProc is similarly
            ambiguous: in one sense it is an optional power feature: a nice-to-have. In another sense it can be regarded
            as foundational. One of the best reasons to have XProc is in how easy it makes it to deploy and run
            XSLT.</p>
               <p>Chances are good that if you are not current on the latest XSLT version, you have little idea of what we are
            talking about, as despite appearances, it may have changed quite a bit since you last saw it. You may think
            you know it but you might have to reconsider.</p>
               <p>Users who last used XSLT 1.0 and even 2.0, in particular, can consider their knowledge out of date until
            they have taken a look at XSLT 3.0.</p>
               <p>Moreover, within the context of XProc, experienced users of XSLT may find their XSLT becomes simpler, since
            XProc has taken over many of the <q>chores</q>.</p>
               <p>Over time, we have seen repeated demonstrations of the principle of pipelining, iterative amelioration (as
            it might be described) or <q>licking into shape</q> as applied to document processing. Of course it proves
            easier to do a complicated task when it is broken into a series of simpler tasks. Pipelining text files in
            Unix was being done long before pipelining structured objects. On Java alone, ways of deploying XML
            transformations and modifications into sequences of steps include at least <a href="../">Apache Ant</a>, Apache Tomcat/<a href="../">Cocoon</a> (a web processing
            framework), XQuery (using engines such as <a href="../">BaseX</a> or <a href="../">eXist-db</a> engines) and XSLT itself (<a href="../">Saxon</a>), to say
            nothing of batch scripts, shell scripts and <q>transformation scenarios</q> or the like, as offered by XML
            tools and toolkits.</p>
               <p>All this can appear disturbingly haphazard. In contrast, XProc offers a single unified approach using a
            standard declarative vocabulary specifically for dealing with process orchestration and I/O (inputs and
            outputs, i.e. interfaces). Thus it helps quite a bit by taking over from XSLT, to whatever extent necessary
            and useful, all those aspects of processing that require any sort of interaction with the wider system. This
            way XSLT plays to its strengths, while XProc standardizes and simplifies how it works. Consequently, XProc
            enables XSLT when needed, on the one hand, while on the other XProc may enable us largely to do without it,
            as it <i>additionally</i> offers both its own useful feature set with regard to routine chores like
            designating sets of inputs and outputs, or sequencing operations. The <a href="../">Rule of Least Power</a> applies here: it saves our
            allies effort (including present and future selves) if we can arrange and manage to do fewer things less.
            XProc lets us do less.</p>
               <p>With XSLT together, this effect is magnified. XSLT lets us write less XProc, and XProc lets us write less
            XSLT. Together they are easier than either would be without the other to lighten the lift.</p>
               <p>XProc lets us use XSLT when we must, but also keeps routine and simple things both simple and consistent.
            And it adapts itself well to new requirements as they become more complicated. Ultimately, it spares the
            XSLT developer the problem of having to design, build and test something like XProc.</p>
               <section>
                  <h3>Reflecting on XSLT</h3>
                  <p>Programmers can think of XSLT as a domain-specific language (DSL) or fourth-generation language (4GL)
               designed for the purpose of manipulating data structures suitable for documents and messages as well as
               for structured data sets. As such, XSLT is highly generalized and abstract and can be applied to a very
               broad range of problems. Its main distinguishing feature among similar languages (which tend to be
               functional languages such as Scala and Scheme) is that it is optimized for use specifically with
               XML-based data formats, offering well-defined handling of information sets expressed in XML, while the
               language itself uses XML syntax, affording nice composability, reflection and code generation
               capabilities. XSLT's processing model is both broadly applicable, and workable in a range of environments
               from widely distributed client software, to encapsulated (<q>containerized</q>), secure software
               configurations and deployments.</p>
                  <p>If your XSLT is strong enough, you don't need XProc, or not much. But as a functional language, XSLT is
               best used in a functionally pure, <q>stateless</q> way that does not interact with the system: no <q>side
                  effects</q>. This is related to its definitions of conformant processing (X inputs produce Y outputs)
               and the determinism, based in mathematical formalisms, that underlies its idea of conformance. However
               one cost of mathematical purity is that operations that do interact with stateful externalities –
               operations such as reading and writing files – are not in XSLT's <q>comfort zone</q>. XSLT works by
               defining what a new structure <b>A'</b> (<q>A prime</q>) should look like for any given structure
                  <b>A</b>, using such terms as a conformant XSLT engine can then effectuate. But to turn an actual A
               into an actual A' we must first acquire A – or an effective surrogate thereof – and then make our A'
               available, in some form. XSLT leaves it up to its processors and <q>calling applications</q> to handle
               this aspect of the problem – which they typically do by offering interfaces for an XSLT transformation's
               nominal <em>source</em> and (primary) <em>result</em>, but which must also go beyond these. Does your
               processor read and parse XML files off the file system? Can it be connected to upstream data producers in
               different ways? Can it use HTTP <code>GET</code> and <code>PUT</code>? The answer may be Yes to any or
               all of these. Throughout its history, XSLT in later versions was also extended in this direction, with
               features such as the <code>collection()</code> function, <code>xsl:result-document</code>,
                  <code>doc-available()</code> and other features we may not need if we are using XProc.</p>
                  <p>Much of this can be set aside when using XSLT with XProc, making the XSLT simpler and easier.</p>
               </section>
               <section>
                  <h3>Running XSLT without XProc</h3>
                  <p>XSLT can also be run without XProc, often to exactly the same ends. But as you start addressing more
               complex requirements, you might find yourself reinventing XProc wheels in XSLT....</p>
               </section>
            </section>
            <section>
               <h2>Using XSLT in XProc: avoiding annoyances</h2>
               <p>If you are an experienced XSLT user, congratulations! The power XProc puts into your hands is everything you
            might think and hope.</p>
               <p>There are a couple of small but potentially annoying considerations when embedding XSLT literals in your
            XProc code. They do not apply when your XSLT is called from out of line, acquired by binding to an input
            port or even <code>p:load</code>. If you acquire and even manipulate your XSLT without including literal
            XSLT code in your XProc, that eliminates the syntax-level clashes at the roots of both these problems.</p>
               <section>
                  <h3>Namespaces in and for your XSLT</h3>
                  <p>
                     <a href="../tutorial/source/oscal-convert/oscal-convert_350_src.html"
                        class="LessonUnit">A subsequent Lesson Unit on
                  namespaces in XProc</a> may help newcomers or anyone mystified by XML namespaces. They are worth
               mentioning here because everything tricky in XProc regarding namespaces is doubly tricky with XSLT in the
               picture.</p>
                  <p>In brief: keep in mind XSLT has its own features for both configuring namespace-based matching on
               elements by name (such as <code>xpath-default-namespace</code>), and for managing namespaces in
               serialization (<code>exclude-namespace-prefixes</code>). In the XProc context, however, your XSLT will
               typically not be writing results directly, instead only producing the same kind of (XDM) tree as is
               emitted and consumed by other steps.</p>
               </section>
               <section>
                  <h3>Text and attribute value syntax in embedded XSLT</h3>
                  <p>If you like XSLT and are prone to plant it into your XProc (it is an excellent golden hammer), this
               applies to you.</p>
                  <p>If not yet conversant with XSLT, you can read more about this topic in an <a href="../tutorial/source/oscal-convert/oscal-convert_102_src.html"
                        class="LessonUnit">upcoming Lesson Unit</a> on data
               conversion. Or you can avoid the problem by always using a <code>p:document/@href</code> to refer to XSLT
               kept out of line.</p>
                  <p>XSLT practitioners know that within XSLT, in attributes and (in XSLT 3.0) within text (as directed), the
               curly brace signs <code>{</code> and <code>}</code> have special semantics as <a href="../">attribute</a> or <a href="../">text value templates</a>. In the latter
               case, the operation can be controlled with an <code>xsl:expand-text</code> setting. When effective as
               template delimiters, these characters can be escaped and hidden from processing by doubling them:
                  <code>{{</code> for <code>{</code> etc.</p>
                  <p>XProc offers a similar feature for expanding expressions dynamically, indicated with a
                  <code>p:expand-text</code> setting much like XSLT's.</p>
                  <p>Because they both operate, an XSLT author must take care to provide for the correct escaping (sometimes
               more than one level) or settings on either language's <code>expand-text</code> option. Searching the
               repository for the string value <code>{{</code> (two open curlies together) will turn up instances of
               this – or skip ahead and try <a href="../tutorial/worksheets/NAMESPACE_worksheet.xpl">a worksheet XProc with
                  some XSLT embedded</a>.</p>
               </section>
            </section>
            <section>
               <h2>Learning XSLT the safer way</h2>
               <p>If setting out to learn XSLT, pause to read the following <i>short but important</i> list of things to which
            you should give early attention, in order:</p>
               <ol>
                  <li>Namespaces in XML and XSLT: names, name prefixes, unprefixed names and the
                  <code>xpath-default-namespace</code> setting (not available until XSLT 2.0).</li>
                  <li>XPath, especially absolute and relative location paths such as <code>/child::oscal:catalog</code> or
                  <code>path/to/node[qualified(.)]</code>: start easy and work up.</li>
                  <li>Templates and modes in XSLT: template matching, <code>xsl:apply-templates</code>, built-in templates,
               and using modes to configure default behaviors when no template matches.</li>
               </ol>
               <p>Understanding each of these will provide also provide useful insights into XProc, both for its commonalities
            with XSLT and for its differences.</p>
            </section>
            <section>
               <h2>XProc without XSLT?</h2>
               <p>As noted, XProc does not require XSLT absolutely, even if XSLT is indispensable for some XProc libraries,
            including those in this repository.</p>
               <p>How could we do without it?</p>
               <ul>
                  <li>Use XQuery anytime queries get complicated</li>
                  <li>Modify documents with XProc where possible, for example using steps that support matches on patterns for
               XSLT-like functionality. Such steps include <code>p:insert</code>, <code>p:label-elements</code>,
                  <code>p:add-attribute</code> and others.</li>
                  <li>Similarly, rely on iterators and <code>p:viewport</code>
                  </li>
                  <li>High-level design and refactoring: use a smarter (declarative, data-centric) format to simplify
               transformation requirements?</li>
               </ul>
               <p>Chances are, there is a limit. One thing XSLT does better than almost any comparable technology is support
            generalized or granular mappings between vocabularies. Typically, the place we begin with XSLT is to create
            HTML for viewing from an XML source. But since it is also very fine for other vocabulary mappings in the
            middle and back, it becomes indispensable almost as soon as it is available for use.</p>
               <p>An XSLT that is used repeatedly – or an arrangement of them – can be encapsulated as an XProc step.</p>
            </section>
            <section>
               <h2>XProc, XDM (the XML data model) and the standards stack</h2>
               <p>Another critical consideration is whether and to what extent XProc and XSLT introduce unwanted dependencies,
            which make them strategically not a good choice (or not a good choice for everyone) at least in comparison
            to alternatives. These are standards in every way including nominally, emerging as the work of organizations
            such as W3C and ISO, while not escaping a reputation as <q>boutique</q> or <q>niche</q> technologies. Yet
            alternative approaches to software development – whether offered by large software vendors and service
            providers, or by forests of Javascript libraries, or a bespoke stack using a developers' favorite flavor of
            Markdown or microformats – have not all fared very well either. Often spurned or ignored, XSLT has a
            reputation today for projects migrating away from it as much as towards it. Yet look closely, and when
            problems arise, XSLT is never the issue in isolation. (A project not able to use XSLT because of a lack of
            understanding or skills is something different.) Often the question is, were you even using the right tool?
            XSLT's reputation suffers when people decide not to use it or to migrate away. But no one talks about all
            the systems that take advantage of it quietly.</p>
               <p>The <em>Golden Hammer</em> is an <a href="../">anti-pattern</a> – related to the <b>Silver Bullet</b> – but this does not make hammers superfluous. It
            helps when your application is within the sweet spot of XSLT and XProc's document processing at scale (and
            there is a sweet spot), but even this is not an absolute rule. Sometimes the question is, are you actually
            fitting the capabilities of the processing model to the problem at hand. Too often, that fit happens by
            accident. Too often, other considerations prevail and compromises are made – then the resulting system is
            blamed.</p>
               <p>So where has XML-based processing been not only tenable but rewarding over the long term? Interestingly, its
            success is to be found often in projects that have survived across more than one system or platform over
            time, that have grown from one system into another, and that have morphed and adapted and grown new limbs.
            In many cases, look at them today and you do not see the same system as you would have only five years ago,
            or twenty.</p>
               <p>Systems achieve sustainability when they are not only stable, but adaptive. This is a fine balance, but one
            that can be found by an evolutionary process of development and experiment. XProc 3.0 and its supporting
            technologies show the results of such an evolution. The demonstration should be in its ease of use combined
            with capability and maintainability.</p>
            </section>
         </section>
      </section>
      <section class="lesson"
               id="oscal-convert"
               name="oscal-convert">
         <section class="unit observer"
                  id="oscal-convert_101"
                  data-track="observer">
            <h1>101: Converting OSCAL – XML to JSON and JSON to XML</h1>
            <section>
               <h2>Goals</h2>
               <p>Learn how OSCAL data can be converted between JSON and XML formats, using XProc.</p>
               <p>Learn something about potential problems and limitations when doing this, and about how they can be detected
            and avoided, prevented or mitigated.</p>
               <p>Become familiar with the idea of generic conversions between syntaxes such as XML and JSON, versus
            conversions capable of handling a single class or type of documents, such as OSCAL, which offers a fully
            defined mapping supporting lossless, error-free translation across syntaxes.</p>
            </section>
            <section>
               <h2>Prerequisites</h2>
               <p>Having succeeded in prior exercises, including tools installation and setup, you are ready to run
            pipelines.</p>
            </section>
            <section>
               <h2>Resources</h2>
               <p>This unit relies on the <a href="../projects/oscal-convert/readme.md">oscal-convert project</a> in
            this repository, with its files. Like all projects in the repo, it aims to be reasonably self-contained and
            self-explanatory. The pipelines there (described below) provide rudimentary support for data conversions,
            demonstrating simplicity and scalability.</p>
            </section>
            <section>
               <h2>Pipeline rundown</h2>
               <p>The XProc pipelines do not do much, as they are wired up to work with minimalistic demonstration data sets.
            This keeps them plain, simple and obvious in exposition, while at the same time it should be apparent how
            they can be readily reconfigured or customized to work on any suitable inputs that may be provided.</p>
               <p>The idea here is simple: run the pipeline and observe its behaviors, including not only the messages it
            emits to the console window, but also the file outputs it generates. Each pipeline should be provided with
            inline comments describing its workings. Also see the <a href="../tutorial/source/oscal-convert/oscal-convert_102_src.html"
                     class="LessonUnit">next lesson unit</a> for more details.</p>
               <section>
                  <h3>
                     <a href="../projects/oscal-convert/GRAB-RESOURCES.xpl">GRAB-RESOURCES</a>
                  </h3>
                  <p>Like other pipelines with this name, run this to acquire resources. In this case, XSLTs used by other
               XProc steps are downloaded from the OSCAL release page.</p>
               </section>
               <section>
                  <h3>
                     <a href="../projects/oscal-convert/BATCH_JSON-TO-XML.xpl">BATCH-JSON-TO-XML</a>
                  </h3>
                  <p>This pipeline uses an XProc input port to include a set of JSON documents, which it translates into XML
               using <a href="../">generic semantics defined
                  in XPath</a>. For each JSON file input, an equivalent XML file with the same filename base is
               produced, in the same place.</p>
                  <p>As posted, the pipeline reads some minimalistic fictional data, which can be found in the system at the
               designated paths.</p>
                  <p>It then uses a pipeline step defined in an imported pipeline, which casts the data and stores the result
               for each JSON file input. In the XProc source, the imported step can be easily identified by its
               namespace prefix, different from the prefix given to XProc elements, as designated by the pipeline's
               developer or sponsor.</p>
                  <p>Follow the <code>p:import</code> link (via its <code>href</code> file reference) to find the step that is
               imported. An imported step is invoked by using its <code>type</code> name, given at the top of the XML
               (as is described in more depth <a href="../tutorial/source/oscal-convert/oscal-convert_201_src.html"
                        class="LessonUnit">in a subsequent
                  lesson unit</a>).</p>
               </section>
               <section>
                  <h3>
                     <a href="../projects/oscal-convert/BATCH_XML-TO-JSON.xpl">BATCH-XML-TO-JSON</a>
                  </h3>
                  <p>This pipeline performs the reverse of the JSON-to-XML batch pipeline. It loads XML files and converts
               them into JSON.</p>
                  <p>Note however how in this case, no guarantees can be made that any XML inputs will result in valid JSON.
               Unless already in a form known in advance – <a href="../">the same vocabulary</a> defined
               by XPath –  XML inputs will typically result in errors, as no comprehensive, rules-bound cast can be
               defined across its variations. To alleviate this problem, exception handling logic in the form of an
               XProc <code>p:try/p:catch</code> can be found in the imported pipeline step (which performs the
               casting).</p>
                  <p>Additionally, this variant has a fail-safe (look for <code>p:choose</code>) that prevents the production
               of JSON from files not named <code>*.xml</code> – strictly speaking, this is only a naming convention,
               but respecting it prevents unseen and unwanted name collisions. It does <i>not</i> defend against
               overwriting any other files that happen to be in place with the target name.</p>
               </section>
               <section>
                  <h3>
                     <a href="../projects/oscal-convert/CONVERT-OSCAL-XML-DATA.xpl">CONVERT-OSCAL-XML-DATA</a>
                  </h3>
                  <p>The requirement that any XML to be converted must already be JSON-ready by virtue of conforming to a
               JSON-descriptive vocabulary, is obviously an onerous one. To achieve a clean, complete and appropriate
               recasting and relabeling of data, depending on its intended semantics, the way those semantics are to be
               expressed in JSON must be fully defined.</p>
                  <p>OSCAL solves this problem by defining its XML and JSON formats in parity, such that a complete
               bidirectional conversion can be guaranteed over data sets already schema-valid. The bidirectional
               conversions themselves can be performed implicitly or overtly by tools that read and write OSCAL, or they
               can be deployed as XSLT transformations, providing for the conversion to be performed by any XSLT 3.0
               engine.</p>
                  <p>For XSLT 3.0, XProc has Saxon. The XSLTs in question can be acquired from an <a href="../">OSCAL release</a>, as shown in the <a href="../projects/oscal-convert/GRAB-RESOURCES.xpl">GRAB-RESOURCES</a> pipeline.</p>
                  <p>
                     <a href="../projects/oscal-convert/CONVERT-OSCAL-XML-DATA.xpl">CONVERT-OSCAL-XML-DATA</a> applies
               one of these XSLTS to a set of given OSCAL XML files, valid to the catalog model, to produce JSON. It
               works on any XML file that has <code>catalog</code> as its root element, in the OSCAL namespace. It does
                  <i>not</i> provide for validation of the input against the schema: Instead, the garbage-in/garbage-out
               (GIGO) principle is respected. This means that some pipelines will run successfully while producing
               defective outputs, which must be discovered in the result (via formal validation and other checks). An
               XProc pipeline with a validation step preceding the conversion would show such errors earlier.</p>
                  <p>The reverse pipeline is left as an exercise. Bring valid OSCAL JSON back into XML. Let us know if you
               have prototyped this and wish for someone to check your work! The task is not trivial because there are
               several ways to use the conversion stylesheet, any of which should work.</p>
               </section>
               <section>
                  <h3>
                     <a href="../projects/oscal-convert/CONVERT-OSCAL-XML-FOLDER.xpl">CONVERT-OSCAL-XML-FOLDER</a>
                  </h3>
                  <p>This pipeline performs the same conversion as <a href="../projects/oscal-convert/CONVERT-OSCAL-XML-DATA.xpl">CONVERT-OSCAL-XML-DATA</a> with an
               important distinction: instead of designating its sources, it processes all XML files in a designated
               directory.</p>
               </section>
            </section>
            <section>
               <h2>Working concept: return trip</h2>
               <p>Note in this context that comparing the inputs and results of a round-trip conversion is an excellent way of
            determining, to some base level, the correctness and validity of your data set. While converting it twice
            cannot guarantee that anything in your data is <q>true</q>, if having converted XML to JSON and back again
            to XML, the result looks the same as the original, you can be sure that its <i>representation</i> is
            consistent.</p>
               <p>Here's an idea: a single pipeline that would accept either XML or JSON inputs, and produce either, or both,
            as outputs. Would that be useful? Should such an <q>OSCAL Army Knife</q> XProc also perform validation?</p>
            </section>
            <section>
               <h2>What is this XSLT?</h2>
               <p>Readers of the prior Lesson Unit on the <a href="../tutorial/source/walkthrough/walkthrough_401_src.html"
                     class="LessonUnit"> XSLT Factor</a>will already have seen XSLT.</p>
               <p>If your criticism of XProc so far is that it makes it look easy when it isn't, you have a point.</p>
               <p>Conversion from XML to JSON isn't free, assuming it works at all. The runtime might be effectively free, but
            developing it isn't.</p>
               <p>Here, the heavy lifting is done by the XSLT component - the Saxon engine invoked by the <code>p:xslt</code>
            step, applying logic defined in an XSLT stylesheet (aka <b>transformation</b>) stored elsewhere. It happens
            that a converter for OSCAL data is available in XSLT, so rather than having to confront this considerable
            problem ourselves, we drop in the solution we have at hand.</p>
               <p>In later units, more examples are shown of how using the XProc steps described, rudimentary data
            manipulations can be done using XProc by itself, without entailing the use of either XSLT or XQuery.</p>
               <p>At the same time, while pipelines are based on the idea of passing data through a series of processes, there
            are many cases where logic is sufficiently complex that it becomes essential to maintain – and test – that
            logic externally from the XProc. At what point it becomes more efficient to encapsulate logic separately
            (whether by XSLT, XQuery or other means), depends very much on the case.</p>
               <p>The <code>p:xslt</code> pipeline step in particular is so important for real-world uses of XProc that it is
            introduced early, to show such a black-box application. There is also an <a href="../">XQuery</a> step – for many purposes, functionally
            equivalent. Among all these – XSLT, XQuery and XProc itself – there is useful redundancy and often more than
            one good way.</p>
               <p>XProc also makes a fine environment for testing XSLT developed or acquired to handle specific tasks – and it
            can support automated testing and test-driven development using <a href="../">XSpec</a>.</p>
               <p>Indeed XSLT and XQuery being, like XProc itself, declarative languages, it makes sense to factor out where
            we can while maintaining easy access and transparency for analysis and auditing purposes.</p>
            </section>
            <section>
               <h2>What could possibly go wrong?</h2>
               <p>Three things can happen when we run a pipeline:</p>
               <ul>
                  <li>The pipeline can fail to run, typically terminating with an error message (or, unusually, failing to
               terminate)</li>
                  <li>The pipeline can run successfully, but result in incorrect outputs given the inputs</li>
                  <li>The pipeline can run successfully and correctly</li>
               </ul>
               <p>Among the range of possible errors, those that show up in your console with error messages are the easy
            ones. This will often be errors of <q>carelessness</q> (providing the wrong kind of input, etc.), easy to
            repair once found. Sometimes it is an input resource, not the pipeline, that must be corrected, or a
            different pipeline developed for the different input. If XML is expected but not provided, a conforming
            processor must emit an error. Correct it or plan on processing plain text.</p>
               <p>The second category is much harder. The most important concern when engineering a pipeline is to see to it
            that no data quality problems are introduced inadvertantly. Anomalous inputs might process <q>correctly</q>
            (for the input provided) but result in lost data or disordered results. Often this is obvious in testing,
            but not always. The key is defining and working within a scope of application (range of inputs) within which
               <q>correctness</q> can be specified, unambiguously and demonstrably, both with respect to the source
            data, and the processing requirement. Given such a specification, testing is possible. Without testing,
            things can be difficult or impossible to know, and evenif known, they can be difficult to demonstrate.</p>
               <p>While in comparison to syntax or configuration problems, data quality issues can be subtle, there is also
            good news: the very same tools we use to process inputs into outputs, can also be used to test and validate
            data to both applicable standards and local rules.</p>
               <p>Generally speaking, OSCAL maintains <em>validation parity</em> between its XML and JSON formats with respect
            to their schemas. That is to say, the XSD (XML schema) covers essentially the same set of rules for OSCAL
            XML data as the JSON Schema does for OSCAL JSON data, accounting for differences between the two notations,
            the data models and how information is mapped into them. A consequence of this is that valid OSCAL data,
            either XML or JSON, can reliably be converted to valid data in the other notation, while invalid data may
            come through with significant gaps, or not be converted at all.</p>
               <p>For this reason (as it applies to OSCAL) and related reasons on open systems (applying across the board, and
            not only to data conversions), the working principle in XML is often to define and formalize a model as
            early as possible – or identify and adopt a model already built – as a way to institute and enforce schema
            validation as a <i>prerequisite</i> and <i>primary requirement</i> for working with any data set. We do this
            by acquiring or writing and deploying schemas. To this end, XProc supports several kinds of schema
            validation including  <a href="../">XML DTD (Document Type Definition)</a>, <a href="../">XSD (W3C
               Schema Definition language)</a>, <a href="../">RelaxNG (ISO
               ISO/IEC 19757-2)</a>, <a href="../">Schematron</a>
            and <a href="../">JSON
               Schema</a>, making it straightforward to enforce this dependency at any point in a pipeline, whether
            applied to inputs or to pipeline results including interim results and pipeline outputs. Resource validation
            is described further in subsequent coverage including the next <a href="../tutorial/source/oscal-convert/oscal-convert_102_src.html"
                     class="LessonUnit">
                     <q>Maker</q> lesson unit</a>.</p>
               <section>
                  <h3>The playing field is the Internet</h3>
                  <p>File resources in XProc are designated and distinguished by URIs. Keep in mind that XProc in theory, and
               your XProc engine in practice, may read its inputs using whatever <a href="../">URI schemes</a> it supports,
               while the schemes <code>file</code> and <code>http</code> (or <code>https</code>) are required for
               conformance, and work as they do on the Worldwide Web.</p>
                  <p>Of course, permissions must be in place to read files from system locations, or save files to them. When
               authentication is configured or resources are openly available, using <code>http</code> to reach
               resources or sources can be a very convenient option.</p>
                  <p>While this is important and powerful, it comes with complications. Internet access is not always a given,
               making such runtime dependencies fragile. XML systems that rely on URIs frequently also support one or
               another kind of URI indirection, such as <a href="../">OASIS XML Catalogs</a>, to
               enable resource management, redirection and local caching of standard resources. For the XProc developer,
               this can be a silent source of bugs, hard to find and hard to duplicate and analyze. The <a href="../tutorial/source/oscal-convert/oscal-convert_102_src.html"
                        class="LessonUnit">next lesson unit</a> describes some functions
               that can be used to provide the transparency needed.</p>
               </section>
            </section>
            <section>
               <h2>More catalogs needed!</h2>
               <p>As we go to press we have only one example OSCAL catalog to use for this exercise.</p>
               <p>Other valid OSCAL catalogs are produced from other projects in this repo, specifically <a href="../projects/CPRT-import/">CPRT import</a> and <a href="../projects/FM6-22-import/">FM6-22-IMPORT</a>. Run the pipelines in those projects to produce more catalogs (in XML) useful as
            inputs here.</p>
            </section>
         </section>
         <section class="unit maker"
                  id="oscal-convert_102"
                  data-track="maker">
            <h1>102: Hands on data conversions</h1>
            <section>
               <h2>Goals</h2>
               <p>Learn how OSCAL data can be converted between JSON and XML formats, using XProc.</p>
               <p>Learn about potential problems and limitations when doing this, and about how to detect, avoid, prevent or
            mitigate them.</p>
               <p>Learn something about XProc features designed for handling JSON data (XDM <b>map</b> objects that can be
            cast to XML).</p>
            </section>
            <section>
               <h2>Prerequisites</h2>
               <p>Run the pipelines described in <a href="../tutorial/source/oscal-convert/oscal-convert_101_src.html"
                     class="LessonUnit">the 101 Lesson
               Unit</a> in this topic.</p>
            </section>
            <section>
               <h2>Resources</h2>
               <p>Same as the <a href="../tutorial/source/oscal-convert/oscal-convert_101_src.html"
                     class="LessonUnit">101 lesson</a>.</p>
            </section>
            <section>
               <h2>Some breaking and making</h2>
               <p>Every project you examine provides an opportunity to alter pipelines and see how they fail when not encoded
            correctly – when <q>broken</q>, any way we can think of breaking them. Then build good habits by repairing
            the damage. Experiment and observation bring learning.</p>
               <p>After reading this page and <a href="../projects/oscal-convert/readme.md">the project readme</a>, run
            the pipelines while performing some more disassembly / reassembly. Here are a few ideas:</p>
               <ul>
                  <li>Switch out the value of an <code>@href</code> on a <code>p:document</code> or <code>p:load</code> step.
               See what happens when the file it points to is not actually there.</li>
                  <li>There is a difference between <code>p:input</code>, used to configure a pipeline in its prologue, and
                  <code>p:load</code>, a step that loads data. Ponder what these differences are. Try changing a
               pipeline that uses one into a pipeline that uses the other.</li>
                  <li>Similarly, there is a difference between a <code>p:output</code> configuration for a pipeline, and a
                  <code>p:store</code> step executed by that pipeline. Consider this difference and how we might define
               a rule for when to prefer one or the other. How is the pipeline used – is it called directly, or intended
               for use as a step in other pipelines? How is it to be controlled at runtime?</li>
                  <li>Try inserting <code>p:store</code> steps into a pipeline to capture intermediate results, that is, the
               output of any step before they are processed by the next step. Such steps can aid in debugging, among
               other uses.</li>
                  <li>
                     <code>@message</code> attributes on steps provide messages for the runtime traceback. They are optional
               but this repo follows a rule that any <code>p:load</code> or <code>p:store</code> should be provided with
               a message. Why?</li>
                  <li>A <code>p:identity</code> step passes its input unchanged to the next step. It can also be provided with
               a <code>@message</code>. The two commonest uses of <code>p:identity</code> are probably to provide for a
                  <q>no-op</q> option, for example within a conditional or try/catch – and to provide runtime messages
               to the console.</li>
               </ul>
               <p>After breaking anything, restore it to working order. Create modified copies of any pipelines for further
            analysis and discussion.</p>
               <ul>
                  <li>Concept: copy and change one of the pipelines provided to acquire a software library or resource of your
               choice.</li>
               </ul>
            </section>
            <section>
               <h2>Value templates in attributes and text: { XPath-expr }</h2>
               <p>A feature was <a href="../tutorial/source/walkthrough/walkthrough_401_src.html"
                     class="LessonUnit">mentioned earlier</a>
            that warrants caution when embedding code in code: when producing string values dynamically (i.e., at
            runtime), XProc supports a <em>template syntax</em> borrowed from XSLT and XQuery, as <a href="../">attribute value templates</a>, <a href="../">text value templates</a>, or <a href="../">enclosed expressions</a>. An XPath expression
            within the brackets is to be evaluated dynamically by the processor. This is one of the most useful
            convenience features in the language.</p>
               <p>
                  <a href="../">This syntax</a> is concise, but expressive. Upon
            seeing:</p>
               <pre>&lt;p:identity message="Processing { $filename } at { current-date() }"/&gt;</pre>
               <p>the XProc developer understands:</p>
               <ul>
                  <li>The date, in some form should be written into the message. (Try it and see.) The XPath function <a href="../">format-date</a> can also be used if
               we want a different format: for example, <code>current-date() =&gt; format-date('[D] [MNn]
               [Y]')</code>.</li>
                  <li>The variable reference <code>$filename</code> is defined somewhere, and here will expand to a string
               value due to the operation of the (attribute value) template.</li>
               </ul>
               <p>If you need to see actual curly brackets, escape by doubling: <code>{{</code> for the single open and
               <code>}}</code> for the single close.</p>
               <p>As mentioned earlier, one complication sometimes arises: because XSLT and XQuery support similar syntax,
            clashes can occur, since their functioning will depend on correctly interpreting the syntax within literal
            code. One solutions include using double escaping (see examples in <a href="../">the spec</a>). This can be tried with <a href="../tutorial/worksheets/NAMESPACE_worksheet.xpl">a worksheet XProc</a>.</p>
               <p>A more comprehensive solution is to use an XProc setting: an <code>expand-text</code> attribute, or
               <code>p:expand-text</code> when appearing on an element not already bound to the <code>p</code>-prefixed
            namespace, i.e. XProc). Setting this to <code>false</code> turns the templating feature off: the brackets
            become regular brackets again. <a href="../">The spec
               also describes</a> an attribute <code>p:inline-expand-text</code> that can be used in places where the
            regular <code>expand-text</code> would interfere with a functional requirement (namely the representation of
            literal XML provided in your XProc using <code>p:inline</code>). Either of these settings can be used inside
            elements already set, resulting in <q>toggling</q> behavior (it can be turned on and off), as any
               <code>expand-text</code>, by applying to descendants, overrides settings on its ancestors.</p>
               <p>For the most part it is enough to know that the <code>expand-text</code> setting is <q>on</q>
               (<code>true</code>) by default, but it can be turned off (<code>false</code>) – and (for handling edge
            cases) back on, lower down in the hierarchy. And watch the curly brackets.</p>
            </section>
            <section>
               <h2>Designating inputs</h2>
               <p>One feature of the pipelines we have looked at so far is that their inputs are hard-wired. While this is
            sometimes helpful (such a pipeline can be nicely self-contained), it should also be possible to apply a
            pipeline to an XML document (or other input) without having to designate the document inside the pipeline
            itself. The user or calling application should be able to say <q>run this pipeline, but this time with this
               input</q>.</p>
               <p>This is important not only for itself but because it is the key to composability: reusing pipelines as steps
            in other pipelines.</p>
               <p>The input ports for a pipeline, specified using <a href="../">
                     <code>p:input</code>
                  </a> within the prologue, provide for this.</p>
               <p>For example, the <a href="../projects/oscal-convert/CONVERT-OSCAL-XML-DATA.xpl">CONVERT-OSCAL-XML-DATA</a> pipeline defines an input port:</p>
               <pre>&lt;p:input port="source" sequence="true"&gt;
    &lt;p:document href="data/catalog-model/xml/cat_catalog.xml"/&gt;
&lt;/p:input&gt;</pre>
               <p>By default, this pipeline will pick up and process the data set (here, an XML document) it finds at path
               <code>data/catalog-model/xml/cat_catalog.xml</code>, relative to the pipeline instance (XProc file). But
            any call to this pipeline, whether directly or as a step in another pipeline, can override this to provide a
            different input.</p>
               <p>An XProc processor defines a command syntax for binding inputs to ports, such as is described in<a href="../">Morgana documentation</a>. Morgana's looks
            like this (when used with the script deployed with this repository):</p>
               <pre>$ ../xp3.sh <i>PIPELINE.xpl</i> -input:<i>portname=path/to/a-document.xml</i> -input:<i>portname=path/to/another-document.xml</i>
               </pre>
               <p>(<a href="../">XML Calabash 3</a> has
            analogous but not identical syntax.)</p>
               <p>Here, two different <code>-input</code> arguments are given for the same port. You can have as many as
            needed if the port, like this one, has <code>sequence="true"</code>, meaning any number of documents (zero,
            one or more) can be bound to the port, and the pipeline will accommodate. When more than one port is defined
            for a pipeline, one (only) can be designated as <code>primary="true"</code>, allowing it to be provided
            implicitly when a port connection is required (by a step) but not designated. Notice that the name of the
            port must also appear in the command argument, as in <code>-input:portname</code>, since while pipelines can
            have ports supporting sequences, they can also have different ports, named differently, for documents
            playing different roles in the pipeline.</p>
               <p>In place of <code>portname</code> here, a common name for a port (conventional when it is the pipeline's
            only or primary input) is <code>source</code>. But you can also expect to see ports (especially secondary
            ports) with names like <code>schema</code>, <code>stylesheet</code> and <code>insertion</code>: port names
            that offer hints as to what the step does.</p>
               <p>A port designated with <code>sequence="true"</code> can be empty (no documents at all) and a process will
            run. But by default, <code>sequence="false"</code> is assumed, for a single document, or an error condition
            when none is provided.</p>
               <p>Among other things, this means that a pipeline that has <code>&lt;p:input name="x"/&gt;</code>  cannot be run
            unless a (single) document for the <code>x</code> port (as it is named here) is provided with the
            invocation.</p>
               <p>For when an empty document binding is wanted (i.e., no document is fine), XProc offers <a href="../">
                     <code>p:empty</code>
                  </a> to make this explicit. Use with
               <code>sequence="true"</code>.</p>
               <section>
                  <h3>Lightening the <code>p:load</code>
                  </h3>
                  <p>As an alternative to binding inputs to using <code>p:input/p:document</code> (on a pipeline definition)
               or <code>p:with-input</code> (on a step invocation), XProc offers another way to acquire data from
               outside the pipeline: by using a <code>p:load</code> step. This is somewhat different in operation: as it
               is a step in itself, errors produced by <code>p:load</code> cannot be detected until the pipeline is run,
               whereas failures with <code>p:input</code> should be detected when the pipeline itself is parsed and
               compiled (i.e. during <em>static analysis</em>), and processors may be able to apply different kinds of
               exception handling, fallbacks or support for redirects. (As always you can try, test and determine for
               yourself.) Apart from this distinction the two approaches have similar effects – whether to use one or
               the other depends often on how you expect the pipeline to be used, distributed, and maintained, since
               either can work in operation.</p>
                  <p>Although one distinction is that <code>p:document</code> appears on input ports, which can be overridden
               (or rather, set dynamically), this does not mean that <code>p:document</code> cannot be essentially
                  <q>private</q> to a pipeline or pipeline step. For example, if you wish to acquire, without
                  <code>p:load</code>, more than a single document known in advance (i.e. the file names can be
               hard-coded), provide your step (<code>p:identity</code> in this case) with inputs like so:</p>
                  <pre>&lt;p:identity&gt;
  &lt;p:with-input&gt;
    &lt;p:document href="..."/&gt;
    &lt;p:document href="..."/&gt;
    ...
  &lt;/p:with-input&gt;
&lt;p:identity&gt;</pre>
                  <p>This binds the documents to the input of the step (as <code>p:identity</code> supports a sequence, more
               than one is fine), without exposing an input port in the main pipeline.</p>
                  <p>Combining the approaches permits another useful capability: first, acquire a list of file names, for
               example (here using <code>p:input/p:inline)</code>:</p>
                  <pre>&lt;p:input port="source"&gt;
   &lt;p:inline&gt;
      &lt;FILELIST&gt;
         &lt;FILE&gt;A&lt;/FILE&gt;
         &lt;FILE&gt;B&lt;/FILE&gt;
      &lt;/FILELIST&gt;
   &lt;/p:inline&gt;
&lt;/p:input&gt;</pre>
                  <p>Then in our subpipeline we use the compound step <code>p:for-each</code> to process each FILE element in
               the list:</p>
                  <pre>&lt;p:for-each&gt;
   &lt;p:with-input select="//FILE"/&gt;
   &lt;p:load href="{ string(.) }"/&gt;
&lt;/p:for-each&gt;   </pre>
                  <p>This has the effect of traversing the document given in line (the file list) and for each of its
                  <code>FILE</code> elements, loading the document named as the <code>FILE</code> element's string
               value, that is <q>A</q>, <q>B</q> (in this example) and so on. This is just as if A and B had been bound
               directly to the port. In either case, what we get is a sequence of XDM <em>document</em> objects, one for
               each of the resources parsed.</p>
                  <p>One tradeoff is that the override mechanism will be different. We override the first approach by binding
               the pipeline's <code>source</code> port directly to whatever documents we want in place of A and B. We
               override the second approach by providing a different FILELIST document. Alternatively such a FILELIST
               can be referenced instead of included … <code>p:document href="the-filelist.xml</code>, providing us a
               resource that we can maintain separately.</p>
                  <p>This makes the second approach especially appealing if the file list can be derived from some kind of
               metadata resource or, indeed, <code>p:directory-list</code>….</p>
               </section>
            </section>
            <section>
               <h2>Warning: do you know where your source files are?</h2>
               <p>As noted in the <a href="../tutorial/source/oscal-convert/oscal-convert_101_src.html"
                     class="LessonUnit">101 Lesson Unit</a>, one of the
            advantages of using URIs, over and above the Internet itself, is that systems can support URI redirection
            when appropriate. This will ordinarily be in order to provide local (cached) copies of standard resources,
            thereby mitigating the need for copying files over the Internet. While this is a powerful and useful feature
            – arguably essential for systems at scale – it can present problems for transparency and debugging if the
            resource obtained by reference to a URI is not the same as the developer (or <q>contract</q>) expects.</p>
               <p>A similar problem results from variations in URI syntax, both due to syntax itself and due to the fact that
            URIs can be relative file paths, so <code>file.xml</code> and <code>../file.xml</code> could be the same
            file, or not, depending on the context of evaluation.</p>
               <p>To help avoid or manage problems resulting from this (i.e., from features as bugs), XPath and XProc offer
            some useful functions:</p>
               <ul>
                  <li>XPath <a href="../">base-uri()</a> will return the
               (nominal) base URI of a node in an XML document, regarded as an XDM property of nodes. Other XDM objects
               including maps do not have base URIs, but XML documents do. This is an important and special property in
               XProc, as it is the basis of its own resource management semantics. Expect to see more of
                  <code>base-uri()</code> along with similar (and aligned) XProc functionality such as
                  <code>document-property(.,'base-uri')</code>.</li>
                  <li>XPath <a href="../">resolve-uri()</a> can be used
               to expand a relative URI into an absolute URI</li>
                  <li>XProc <a href="../">p:urify</a> will normalize URIs and rewrite
               file system paths as URIs – very useful.</li>
                  <li>In XProc 3.1, a new function <a href="../">p:lookup-uri</a> can query the processor's URI resolver regarding a URI, without actually retrieving
               its resource. This makes available to the developer what address is actually to be used when a URI is
               followed – detecting any redirection – and permits defensive code to be written when appropriate.</li>
               </ul>
            </section>
            <section>
               <h2>Probing error space – data conversions</h2>
               <p>The topic here picks up from the earlier consideration of <a href="../tutorial/source/oscal-convert/oscal-convert_101_src.html"
                     class="LessonUnit">What could possibly go wrong</a>.</p>
               <p>Broadly speaking, problems encountered running these conversions (or indeed, transformations in general)
            fall into two categories, the distinction being simple, namely whether a bad outcome is due to an error in
            the processor and its logic, or in the data inputs provided. The term <q>error</q> here hides a great deal.
            So does <q>bad outcome</q>. One type of bad outcome takes the form of failures at runtime – the term
               <q>failure</q> again leaving questions open, while at the same time it seems fair to assume that not
            being able to conclude successfully is a bad thing. But other bad outcomes are not detectable at runtime. If
            inputs are bad (inconsistent with stated contracts such as data validation), processes can run
               <i>correctly</i> and deliver incorrect results: correctly representing inputs, in their incorrectness.
            Again, the term <i>correct</i> here is underspecified and underdefined, except in the case.</p>
               <p>For these and other reasons we sometimes prefer to call them <q>exceptions</q>, while at the same time we
            know many errors are not actually errors in the process but in the inputs. We need reliable ways to tell
            this difference. A library of reliable source examples -- a test suite – is one asset that helps a great
            deal. Even short of unit tests, however, a great deal can be discovered when working with <q>bad inputs</q>
            interactively. This knowledge is especially valuable once we are dealing with examples that are only
               <q>normally bad</q>.</p>
               <p>Some ideas on how to do this appear below.</p>
               <section>
                  <h3>Converting broken XML or JSON</h3>
                  <p>Create a syntactically-invalid (not <b>well-formed</b>) XML or JSON document - or rather (more
               correctly), a text document that might have been XML or JSON but for some incidental problem.</p>
                  <p>
                     <i>Try using this as input</i> to any XProc. Note how the processor delivers error messages bringing
               attention to problems it discovers.</p>
               </section>
               <section>
                  <h3>Converting not-OSCAL</h3>
                  <p>XML practitioners understand how XML can be well-formed and therefore legible for processing, without
               being a valid instance of a specific markup vocabulary. You can have XML, for example, without having
               OSCAL. This was discussed in <a href="../tutorial/source/oscal-convert/oscal-convert_101_src.html"
                        class="LessonUnit">the previous lesson
                  unit</a>.</p>
                  <p>But a hands-on appreciation, through experience, of how this actually looks, is better than a merely
               intellectual understanding of why it must be.</p>
                  <p>When providing XML that is not OSCAL to a process that expects OSCAL inputs, you should properly see
               either errors (exceptions), or bad results (outputs missing or wrongly expressed) or both. A tutorial is
               the perfect opportunity to experiment and see.</p>
                  <p>For example, try using the OSCAL XML-to-JSON pipeline on an XProc document (which is XML, but not
               OSCAL).</p>
                  <p>The interesting thing here is how permissive XProc is, unless we code it to be jealous. Detection of bad
               results is an important capability, which is why we also need to be able to <em>validate</em> data
               against external constraint sets such as schemas, also covered in more detail later.</p>
               </section>
               <section>
                  <h3>Converting broken OSCAL</h3>
                  <p>The same thing applies to attempting to process inputs when OSCAL is expected, yet the data sources fail
               to meet requirements in some important respect, sometimes even a subtle requirement, depending on the
               case. The more fundamental problem here is the definition of <q>correct</q> versus <q>broken</q>.</p>
                  <p>We begin generally with the stipulation that by <q>OSCAL</q> what we mean is, any XML (or JSON or YAML)
               instance conformant to an OSCAL schema, and thereby defined in such a manner as to enable their
               convertibility. The reasoning is thus somewhat circular. If we can convert it successfully, we have a
               basis to claim it is OSCAL, by virtue of its <i>evident</i> conformance to OSCAL models in operation. If
               we know it to be OSCAL by virtue of schema validation, we have assurances also regarding its
               convertibility.</p>
                  <p>In contrast, data that is not schema-valid (as can be reasoned) cannot be <i>confidently</i> and
                  <i>completely</i> qualified or described at all, so only very simple (<q>global</q>, generic or
                  <q>wildcard</q>) mappings from arbitrary inputs can be specified. Almost by definition, these will
               usually be poor for actual cases. But a mapping can be specified for inputs that are known, such as OSCAL
               inputs. An OSCAL converter respects the validation rules not by enforcing them directly, but rather by
               depending on the consistency they describe and constrain.</p>
                  <p>Fortunately, by means of Schematron and transformations, XProc is an excellent tool not only for altering
               data sets, but also for imposing such validation rules, by detecting variances, either in inputs or its
               results. XPath, the query language, becomes key. With XPath to identify features (both good and bad), and
               XProc for modifications, these capabilities – detection and amelioration – can be used together, and
               separately. When a pipeline cannot guarantee correct outputs, it can at least provide feedback.</p>
                  <p>Depending on the application and data sources, XML that is <q>broken</q> in various subtle ways is more
               or less inevitable. See what it looks like by making this happen on purpose.</p>
               </section>
            </section>
            <section>
               <h2>XProc diagnostic how-to</h2>
               <p>These methods are noted above, but they are so important they should not be skipped.</p>
               <section>
                  <h3>Emitting runtime messages</h3>
                  <p>Most XProc steps support a <code>message</code> attribute for designating a message to be emitted to the
               console or log. As shown, these also support Attribute Value Syntax for dynamic evaluation of XPath.</p>
                  <p>For example, again using <code>p:identity</code>:</p>
                  <pre>&lt;p:identity message="Processing { p:document-property(.,'base-uri') } with content-type { p:document-property(.,'content-type') }"/&gt;</pre>
                  <p>This step does not change the document, but reports its current Base URI and content-type at that point
               in the pipeline.</p>
                  <p>This can be useful information since both those properties can (and should) change based on your
               pipeline's operations.</p>
               </section>
               <section>
                  <h3>Saving out interim results</h3>
                  <p>Learn to use the <code>p:store</code> step, if only because it is so useful for saving interim pipeline
               results to a place where they can be inspected.</p>
                  <p>
                     <a href="../projects/FM6-22-import/PRODUCE_FM6-22-chapter4.xpl">Produce-FM6-22-chapter4</a> is a
               demonstration pipeline in this repo with a switch at the top level, in the form of an option named
                  <code>writing-all</code>. When set to <code>true()</code>, it has the effect of activating a set of
                  <code>p:store</code> steps within the pipeline using the XProc <a href="../">use-when feature</a> feature, to write intermediate
               results. The resulting set of files is written into a <code>temp</code> directory to keep them separate
               from final results: they show the changes being made over the input data set, at useful points for
               tracing the pipeline's progress.</p>
               </section>
            </section>
            <section>
               <h2>Validate early and often</h2>
               <p>One way to manage the problem of ensuring quality is to validate the inputs before processing, either as a
            dependent (prerequisite) process, or built into a pipeline. This enables a useful separation between
            problems resulting from bad inputs, and problems within the pipeline. Whatever you want to do with invalid
            inputs, including skipping or ignoring them, producing warnings or runtime exceptions, or even making
            corrections when possible and practical – all this can be defined in a pipeline much like anything else.</p>
               <p>Keep in mind that since XProc brings support for multiple schema languages plus XPath, <q>validation</q>
            could mean almost anything. This must be determined for the case.</p>
               <p>In the <a href="../projects/oscal-publish/publish-oscal-catalog.xpl">publishing demonstration project
               folder</a> is an XProc that valides XML against an OSCAL schema, before running steps to  convert it to
            HTML, for display in a browser. The same could be done for an XProc that converts OSCAL data into JSON --
            since OSCAL has both XSD for XML, and JSON Schema for JSON, this could be done before the conversion, after,
            or both.</p>
               <p>Two projects in this repository (at time of writing) deal extensively with validation: <a href="../projects/oscal-validate/">oscal-validate</a> and <a href="../projects/schema-field-tests/">schema-field-tests</a>.</p>
            </section>
         </section>
         <section class="unit learner"
                  id="oscal-convert_201"
                  data-track="learner">
            <h1>201: Anatomy of an XProc pipeline</h1>
            <section>
               <h2>Goals</h2>
               <p>Get more in-depth information about XProc internals, including especially the parts of an XProc pipeline
            step, as a step.</p>
               <p>This includes its imports, its prologue, subpipeline and steps.</p>
            </section>
            <section>
               <h2>Prerequisites</h2>
               <p>You have succeeded in prior exercises, including tools installation and setup. You have seen enough XProc to
            be impressed, at least, by the concept.</p>
               <p>While concepts here are basic, what is presented is not always obvious.</p>
            </section>
            <section>
               <h2>Resources</h2>
               <p>Pipelines throughout the repository serve as examples for the description that follows.</p>
               <p>Earlier coverage of  <a href="../tutorial/source/walkthrough/walkthrough_219_src.html"
                     class="LessonUnit">the XML Data Model
               (XDM) as it relates to XProc </a> can also be kept in mind.</p>
            </section>
            <section>
               <h2>XProc as XML (redux)</h2>
               <p>Recall that an XProc pipeline itself takes the form of an XML <a href="../">document entity</a>. Put less mysteriously, this
            means that XProc is designed to be written and maintained in XML syntax, and most XProc systems will be
            reliant on XProc kept in XML files (in a file server or database). We recognize XProc by its XML
               <em>vocabulary</em> expressed using <em>tags</em> that follow the XML tagging rules:</p>
               <ul>
                  <li>They follow XML's rules with respect to naming, whitespace, delimiters and reserved characters</li>
                  <li>They are correctly balanced, with an end tag for every start tag – for a <code>&lt;start&gt;</code>
               there must be a <code>&lt;/start&gt;</code> (an end to the start).</li>
                  <li>They are cleanly nested with no overlap: end tags always close the most recently opened element, so no
               element ever extends beyond the boundaries of its <q>ancestors</q> or containing elements</li>
               </ul>
               <p>These rules are fairly simple, and they are well supported by tools designed to read and write XML – to
            respect, follow and enforce the rules on our behalf.</p>
               <p>Thus they are also quickly and easily internalized, often in only a few minutes of working with XML.</p>
               <p>Over and above being XML, XProc has some rules of its own ...</p>
               <section>
                  <h3>XProc at the top</h3>
                  <p>At the very top of an XProc file, expect to see something not unlike this:</p>
                  <pre>&lt;p:declare-step version="3.0"
   xmlns:p="http://www.w3.org/ns/xproc"
   xmlns:ox="http://csrc.nist.gov/ns/oscal-xproc3"
   type="ox:TEST-XPROC3"
   name="TEST-XPROC3"&gt;
...
&lt;/p:declare-step&gt;</pre>
                  <p>XProc pipelines are XML documents using the XProc vocabulary. At the top (paradoxically, we call this the
                  <q>root</q>), an XProc instance is identified by tagging it either of <code>p:declare-step</code> or
                  <code>p:library</code>. XProc in this repository includes at least one <code>p:library</code>, and it
               might be nice to have more. More on this below.</p>
                  <p>As noted next, the element at the top ordinarily provides namespace prefix bindings (namespace
               declaration attributes) along with a <code>name</code> and a <code>type</code> for the step.</p>
               </section>
               <section>
                  <h3>Namespaces</h3>
                  <pre>   xmlns:p="http://www.w3.org/ns/xproc"
   xmlns:ox="http://csrc.nist.gov/ns/oscal-xproc3"</pre>
                  <p>Namespaces are discussed <a href="../tutorial/source/oscal-convert/oscal-convert_350_src.html"
                        class="LessonUnit">soon enough</a>. Here it
               suffices to reiterate that XProc needs to be able to use its own namespace,
                  <code>http://www.w3.org/ns/xproc</code>) at the very minimum, as well as others, both for the data
               sets in scope (inputs and process results) and for processes: namespaces also serve to support
               extensibility up and down the stack (in XProc, XSLT and XPath/XQuery). Accordingly you will probably also
               want a namespace (or more than one) whose prefix and URI bindings you control.</p>
                  <p>Since namespaces declared at the top of the document will apply throughout the document, this is a good
               (and conventional) place to put namespace declaration attributes, keeping them out of the way and make
               them easier to find.</p>
               </section>
               <section>
                  <h3>@name and @type</h3>
                  <p>On <code>p:declare-step</code>, whether at the top or in a step definition within a pipeline, either or
               both a <code>@name</code> and a <code>@type</code> are permitted.</p>
                  <pre>   type="ox:TEST-XPROC3"
   name="TEST-XPROC3"</pre>
                  <p>The name makes it possible to reference the pipeline itself by name, as a step. This makes it possible to
               create bindings inside the pipeline to the pipeline's own ports, when implicit (i.e., defaulted) bindings
               are not sufficient.</p>
                  <p>Understandably, the name of an XProc must be different from the names given to all the steps in the XProc
               (which must also be distinct).</p>
                  <p>This repository follows a rule that a step name should correspond to its file base name (i.e., without a
               filename suffix), so <code>identity_</code> for <code>identity_.xproc</code>, etc. But that is a rule for
               us, not for XProc in general.</p>
                  <p>A step may also have an assigned <code>@type</code>. Unlike the name, which can be in any namespace or
               none, the <code>@type</code> must be assigned to a namespace.</p>
                  <p>The assigned typed is just as important as the name, as a step is called by its <em>type name</em>.
               Decoupling the name from the type name provides useful flexibility, as both can be important but they do
               not need to be the same.</p>
               </section>
            </section>
            <section>
               <h2>Prologue and body</h2>
               <p>A pipeline will typically be a collection or sequence of steps, of arbitrary complexity. By this we mean
            that any step might be simple or complex in its operations; and the sequence may be short or long, or simple
            and singular or branching, multiple (with respect to inputs or outputs) and conditional. In addition to this
            orchestration, simple or complex, a pipeline must have one other thing, namely an interface or
               <em>signature</em>. This is what gives the pipeline (if one dare say) <q>semantics</q> in the form of a
            specification, whether explicit or implicit, of what constitute valid inputs, expected outputs, and
            recognized runtime options. This interface is defined in the pipeline's <em>prologue</em>. The sequence or
            arrangement of steps, however long or short, constitutes the pipeline's <em>body</em> and serve as the
            pipeline's <a href="../">subpipeline</a> .</p>
               <p>Think of the subpipeline as the working components of the pipeline, while the prologue defines its interface
            or <q>control surface</q>, as on a black box. That is, what kinds of inputs need to be provided to it (if
            any), what kinds of results it exposes (apart from outputs it has taken on itself to produce), and what
            kinds of options or configurations are available.</p>
               <p>But even before the prologue we may see import declarations. Add to these any local step definitions for
            embedded pipelines (not common but not unheard of), and at a high level we see these element groups:</p>
               <ul>
                  <li>Imports (optional) - configuring where the processor can find logic to be called from external
               pipelines: <code>p:import</code>, <code>p:import-functions</code>
                  </li>
                  <li>Prologue - configuring inputs, outputs and options for the pipeline, if any - the prologue can be empty:
               , <code>p:input</code>, <code>p:output</code>, <code>p:option</code>
                  </li>
                  <li>Local step definitions, if any: <code>p:declare-step</code>
                  </li>
                  <li>Subpipeline - step invocations, connected implicitly or explicitly, with supporting variable
               declarations and documentation</li>
               </ul>
               <p>In total, the list of elements coming before the subpipeline is short, which helps: six in total between
               <code>p:import</code> and <code>p:declare-step</code>. Everything coming after is part of the
            subpipeline.</p>
               <p>Imports will be discussed later, or can be reasoned about readily from examples.</p>
               <p>The <em>prologue</em> is used to define <em>ports</em> and <em>options</em> for the pipeline.  Defining
            ports and options is how you give the users of the step with the affordances or control points they need to
            use it. It is common and conventional to have a single input port named <code>source</code> as <em>primary
               input</em>. But some pipelines require no input bindings since they acquire data in other ways. If your
            pipeline is intended to be self-contained, its prologue can be empty. More commonly, ports and options are
            defined, at least to provide default settings.</p>
               <p>Keep in mind that just because a pipeline has no exposed ports for inputs or outputs, does not mean it does
            nothing. Among other things, pipelines can read and write (or be asked to write) arbitrary resources to a
            file system. Its exposed interfaces provide for functional composition: since they have inputs and outputs,
            pipelines can be used as steps in pipelines. But those do not in any way preclude its operations. Unlike the
            functional languages it embeds (XSLT and XQuery), XProc does <i>not</i> seek to be side-effect free.</p>
               <p>Following the prologue, a step may also have local step definitions (<code>p:declare-step</code>). One might
            think of these as an XProc equivalent of a <q>macro</q>: these locally-defined pipelines can be used
            internally for logic that is used repeatedly, or that warrants separating from the main pipeline for some
            other reason.</p>
               <p>After imports, prologue and (optional) step declarations, the step sequence that follows comprises the <a href="../">subpipeline</a>.</p>
               <p>One other complication: among the steps in the subpipeline, <code>p:variable</code> (a variable declaration)
            and <code>p:documentation</code> (for pipeline documentation) are also permitted – these are not properly
            steps, but can be useful to have with them.</p>
               <p>In summary with more detail: any XProc pipeline, viewed as a step declaration, can have the following:</p>
               <ul>
                  <li>Pipeline name and type assignment (if needed), given as attributes at the top</li>
                  <li>
                     <b>Imports</b>: step declarations, step libraries and functions to make available</li>
                  <li>The pipeline <b>prologue</b>: any of the elements named <code>p:input</code>, <code>p:output</code> and
                  <code>p:option</code>, defining this pipeline's ports and options</li>
                  <li>Optionally (and not common): step declarations for local steps, appearing at
               <code>p:declare-step</code>. Each of these will have its own name, type, prologue and steps</li>
                  <li>For this pipeline, one or more steps, called the <a href="../">subpipeline</a>
                     <ul>
                        <li>Standard atomic and compound steps in XProc namespace (probably prefixed <code>p:</code>)</li>
                        <li>Imported steps - in their own namespaces (in this repository, prefixed <code>ox:</code>)</li>
                        <li>Variable declarations - <code>p:variable</code>
                        </li>
                     </ul>
                  </li>
                  <li>Finally, as noted above, <code>p:documentation</code> can appear anywhere in a pipeline, but it will be
               ignored except when appearing inside <code>p:inline</code>. What to do with these is a topic to be
               covered later.</li>
               </ul>
               <p>A useful exercise can be to open a few pipelines and distinguish its internal boundaries.</p>
            </section>
            <section>
               <h2>XProc steps</h2>
               <p>The <i>step</i> is the core conceptual unit of XProc. An XProc processing pipeline is composed of steps. But
            a pipeline is also considered as a step in itself. As such it can be used in other pipelines, and so on.</p>
               <p>In other words, steps in XProc are <i>compositional</i>. They are building block assemblies made out of
            smaller building block assemblies. A step is a way to process data. A pipeline is a way of orchestrating and
            arranging such processes.</p>
               <p>The distinction between pipelines and steps is relative and provisional, but important and useful. The
            pipeline is the logical and actual definition of how your data is to be processed. Every pipeline is
            composed of an arrangement, often a series, of operations. These operations – the steps – include
               <q>primitives</q>, being designed for generality and reusability for the most common operations. But they
            can also include new steps we have written, as pipelines, and such custom-designed steps can be used in
            combination with the primitives or core compound steps of the language.</p>
               <p>At a higher level, defining new steps with new step declarations, and using them in combination with other
            steps, is how we manage complexity and change in processing requirements. This strategy maximizes
            adaptability, while also supporting an incremental maturity model, in which all defined processes can be
            improved with reuse, building and testing over time. Careful use and deployment of new steps is how we save
            work, by focusing optimization and making it possible to scale up to address data processing requirement
            sets that are both large and complex.</p>
               <p>Thus a kind of <q>recursive orderability</q>, wherein encapsulation itself is scaled up and out arbitrarily
            but only as much as needed, can be a working principle with XProc. As a pipeline, a step definition may
            include only a single step (a pipeline can be an interface or wrapper to another pipeline, or to a simple
            operation), or a long sequence or complex choreography of steps. In either case it can become a relatively
            self-contained black box process available to other processes.</p>
               <p>Accommodating this design, an XProc <i>file</i> considered as an XML instance (as noted) is either of two
            things: a <i>step declaration</i>, or a collection of such declarations, a <i>library</i>. At the top level
            (as noted), recognize an XProc step declaration by the element, <code>p:declare-step</code> (in the XProc
            namespace) and a library by the element <code>p:library</code>.</p>
               <pre>&lt;p:declare-step xmlns:p="http://www.w3.org/ns/xproc" version="3.0" 
    name="a-first-step"&gt;
...
&lt;/p:declare-step&gt;</pre>
               <p>Additionally, step declarations can include their own pipeline (step) declarations, making a hybrid
            architecture: the pipeline comprises a step, with its own library not imported but in line. This can also be
            useful.</p>
               <p>An example of a step library in this repository is <a href="../xspec/xspec-execute.xpl">xspec-execute.xpl</a>, which collects several steps supporting XSpec, one each for supporting the XSpec
            testing framework for XSLT, XQuery and Schematron respectively.</p>
               <p>The advantage of defining a step at the top level, rather than putting all steps into libraries, is that
            such a step can be invoked without prior knowledge of its type name, which is used by XProc to distinguish
            it from other steps. The pipeline simply needs to be presented to the processor, which does the rest. Your
            library of steps then looks very similar to your directory full of XProc files – and can be treated, where
            appropriately, as self-contained scripts encapsulating everything they need for a given runtime.</p>
            </section>
            <section>
               <h2>Atomic and compound steps</h2>
               <p>Given an understanding of the organization of an XProc pipeline, the focus shifts to the steps themselves,
            which follow a common pattern. Briefly put, an atomic step is any step you use by simply invoking it with
            inputs and options: its logic is self-contained, and the operation it carries out is (at least conceptually)
            single and unified. A compound step, in contrast, combines one or more other steps in its own
               <em>subpipelines</em> and manages these together through a single interface, while providing – unlike an
            atomic step -- some other functionality depending on the step.</p>
               <p>XProc keeps things workable by providing only a few compound steps supporting the identified range of needs.
            This does not prove to be a practical limitation, since all steps including atomic steps can have multiple
            inputs and outputs, distinguished by type and role. (For example, a validation step might output both a copy
            of the input, potentially annotated, along with a validation report.) Atomic steps are not necessarily
            simple, and may include compound steps in their own subpipelines. All steps you define will be atomic steps.
            Accordingly, compound steps are not necessarily more complex than atomic steps: they are useful because they
            handle common contingencies such as splicing (with <code>p:viewport</code>), splitting (with
               <code>p:for-each</code>, perform an operation in parallel over a set of inputs, not a single document),
            conditionals (<code>p:if</code>, <code>p:choose</code>) and exception handling (<code>p:try</code> with
               <code>p:catch</code> and <code>p:finally</code>).</p>
               <p>Here are all the compound steps:</p>
               <ul>
                  <li>
                     <a href="../"
                        style="color: rgb(3, 69, 117); text-decoration: none; border-bottom: 1px solid rgb(112, 112, 112); padding: 0px 1px; margin: 0px -1px; font-family: sans-serif; font-size: medium; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: -120px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal;">
                        <code class="tag-element language-construct"
                              style="font-family: Consolas, Monaco, &#34;Andale Mono&#34;, monospace; font-size: 16px; break-inside: avoid; hyphens: none; text-transform: none; text-align: left; white-space: pre; color: black; text-shadow: white 0px 1px; direction: ltr; word-spacing: normal; word-break: normal; tab-size: 4; padding: 0.1em; border-radius: 0.3em;">p:group</code>
                     </a>
               - group a subpipeline (step sequence) into a single logical step</li>
                  <li>
                     <a href="../"
                        style="color: rgb(3, 69, 117); text-decoration: none; border-bottom: 1px solid rgb(112, 112, 112); padding: 0px 1px; margin: 0px -1px; font-family: sans-serif; font-size: medium; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: -120px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal;">
                        <code class="tag-element language-construct"
                              style="font-family: Consolas, Monaco, &#34;Andale Mono&#34;, monospace; font-size: 16px; break-inside: avoid; hyphens: none; text-transform: none; text-align: left; white-space: pre; color: black; text-shadow: white 0px 1px; direction: ltr; word-spacing: normal; word-break: normal; tab-size: 4; padding: 0.1em; border-radius: 0.3em;">p:if</code>
                     </a>
               - execute a subpipeline conditionally</li>
                  <li>
                     <a href="../"
                        style="color: rgb(3, 69, 117); text-decoration: none; border-bottom: 1px solid rgb(112, 112, 112); padding: 0px 1px; margin: 0px -1px; font-family: sans-serif; font-size: medium; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: -120px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal;">
                        <code class="tag-element language-construct"
                              style="font-family: Consolas, Monaco, &#34;Andale Mono&#34;, monospace; font-size: 16px; break-inside: avoid; hyphens: none; text-transform: none; text-align: left; white-space: pre; color: black; text-shadow: white 0px 1px; direction: ltr; word-spacing: normal; word-break: normal; tab-size: 4; padding: 0.1em; border-radius: 0.3em;">p:choose</code>
                     </a>
               - execute a subpipeline conditionally (<code>switch/case</code> operator)</li>
                  <li>
                     <a href="../"
                        style="color: rgb(3, 69, 117); text-decoration: none; border-bottom: 1px solid rgb(112, 112, 112); padding: 0px 1px; margin: 0px -1px; font-family: sans-serif; font-size: medium; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: -120px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal;">
                        <code class="tag-element language-construct"
                              style="font-family: Consolas, Monaco, &#34;Andale Mono&#34;, monospace; font-size: 16px; break-inside: avoid; hyphens: none; text-transform: none; text-align: left; white-space: pre; color: black; text-shadow: white 0px 1px; direction: ltr; word-spacing: normal; word-break: normal; tab-size: 4; padding: 0.1em; border-radius: 0.3em;">p:for-each</code>
                     </a>
               - produce subpipeline results for each member of a sequence of inputs (XDM nodes or values treated as
               documents)</li>
                  <li>
                     <a href="../"
                        style="color: rgb(3, 69, 117); text-decoration: none; border-bottom: 1px solid rgb(112, 112, 112); padding: 0px 1px; margin: 0px -1px; font-family: sans-serif; font-size: medium; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: -120px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal;">
                        <code class="tag-element language-construct"
                              style="font-family: Consolas, Monaco, &#34;Andale Mono&#34;, monospace; font-size: 16px; break-inside: avoid; hyphens: none; text-transform: none; text-align: left; white-space: pre; color: black; text-shadow: white 0px 1px; direction: ltr; word-spacing: normal; word-break: normal; tab-size: 4; padding: 0.1em; border-radius: 0.3em;">p:viewport</code>
                     </a>
               - reproduce outputs, except splicing subpipeline results in place of matched nodes (elements) in the
               input</li>
                  <li>
                     <a href="../"
                        style="color: rgb(3, 69, 117); text-decoration: none; border-bottom: 1px solid rgb(112, 112, 112); padding: 0px 1px; margin: 0px -1px; font-family: sans-serif; font-size: medium; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: -120px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal;">
                        <code class="tag-element language-construct"
                              style="font-family: Consolas, Monaco, &#34;Andale Mono&#34;, monospace; font-size: 16px; break-inside: avoid; hyphens: none; text-transform: none; text-align: left; white-space: pre; color: black; text-shadow: white 0px 1px; direction: ltr; word-spacing: normal; word-break: normal; tab-size: 4; padding: 0.1em; border-radius: 0.3em;">p:try</code>
                     </a>
               - execute a subpipeline and deliver its results, or if it fails, run a fallback subpipeline given in a
                  <code>p:catch</code>
                  </li>
               </ul>
               <p>All other steps are atomic steps. A survey of built-in atomic steps <a href="../tutorial/source/walkthrough/walkthrough_219_src.html">has been offered earlier</a>; additionally they are
            well-indexed <a href="../projects/xproc-doc/out/repository-step-list.html">in this repository</a>
            (generated with <a href="../projects/xproc-doc/XPROC-STEP-INDEX-HTML.xpl">XProc</a>) and <a href="../">on line</a> (by Erik Siegel). But of course the atomic steps
            include steps you write and deploy yourself, or acquire from external libraries.</p>
               <p>Additionally to these elements, XProc subpipelines may contain variable declarations
               (<code>p:variable</code>) and documentation (<code>p:documentation</code>), as noted.</p>
            </section>
            <section>
               <h2>Namespaces and extension steps</h2>
               <p>XML Namespaces is <a href="../tutorial/source/oscal-convert/oscal-convert_350_src.html">the topic that will not go away</a>.</p>
               <p>We recognize steps because we either recognize them by name - for standard steps in the <code>p:</code>
            (XProc) namespace such as <code>p:filter</code> and <code>p:add-attribute</code> - or sometimes by process
            of elimination – because they cannot be anything else. This is because extension steps, whether written or
            acquired, can be named anything. Fortunately, extension steps in XProc take the form of elements in an
               <em>extension namespace</em>. Generally speaking, that is, any element not prefixed with <code>p:</code>
            is treated as out of scope for XProc and to be ignored, while subject to evaluation as an extension.</p>
               <p>But this is an important category, since such extensions may include XProc steps whose functioning is core
            to the pipeline as a whole.</p>
               <details>
                  <summary>Question: Where are extension steps used in the XProcs run so far?</summary>
                  <p>One answer:
               The <a href="../smoketest/TEST-XSPEC.xpl">XSpec smoke test</a> calls an extension step named
                  <code>ox:execute-xspec</code>, defined in an imported pipeline. In this document, the prefix
                  <code>ox</code> is bound to a utility namespace,
               <code>http://csrc.nist.gov/ns/oscal-xproc3</code>.</p>
                  <p>In an XProc pipeline (library or step
               declaration) one may also see additional namespaces, including:</p>
                  <ul>
                     <li>The namespaces needed for XSLT, XSD, or other supported technology</li>
                     <li>One or more namespaces deployed by the XProc author to support either steps or internal operations
                  (for example, XSLT functions)</li>
                     <li>The namespace <code>http://www.w3.org/ns/xproc-step</code>, usually associated with the name prefix
                     <code>c:</code>. This namespace is designated by XProc in order to help standardize the interfaces
                  (inputs and outputs) supported by standard steps.</li>
                     <li>The namespace <code>http://www.w3.org/ns/xproc-error</code>, for XProc's error reporting</li>
                  </ul>These declarations (often but not always at the top of the document) are critically important for XPath
            and hence for the correct operation of your pipelines. See <a href="../">the specification</a> for more information.</details>
            </section>
            <section>
               <h2>Schema for XProc 3.0</h2>
               <p>See coverage of <a href="../tutorial/source/walkthrough/walkthrough_219_src.html"
                     class="LessonUnit">XProc, XML and XDM (the
               XML Data Model)</a> in the prior lesson unit for a link to the schema for XProc.</p>
            </section>
         </section>
         <section class="unit maker"
                  id="oscal-convert_350"
                  data-track="maker">
            <h1>350: Namespaces in XML and XProc</h1>
            <section>
               <h2>Goals</h2>
               <p>Learn enough about namespaces in XML to be able to make sense of XProc's handling of embedded vocabularies,
            whether extending its capabilities or for the purpose of embedding (literal) data sets.</p>
               <p>Become comfortable with prefixed names such as <code>p:sleep</code>, <code>xsl:template</code> or
               <code>ox:oscal-xml-as-json</code> as a technology-wide convention for helping to disambiguate the
            semantics of different vocabularies, when mixed and in general.</p>
            </section>
            <section>
               <h2>Prerequisites</h2>
               <p>None. If you don't need to know about XML's namespace mechanism, you can gladly skip this coverage.</p>
            </section>
            <section>
               <h2>Resources</h2>
               <p>The <a href="../">Namespaces in XML Recommendation</a> was published by W3C
            in 2009, and remains the authoritative source.</p>
               <p>
                  <a href="../tutorial/worksheets/NAMESPACE_worksheet.xpl">A namespaces worksheet XProc</a>provides a place to
            experiment with namespaces in XProc.</p>
            </section>
            <section>
               <h2>XML and namespaces</h2>
               <p>XProc like many XML technologies uses the XML namespace mechanism to disambiguate between mixed
            vocabularies.</p>
               <p>Using namespace-qualified names in the form of <em>colonized names</em> (such as <code>p:try</code>)  makes
            it possible to distinguish between <code>p:try</code> and <code>xsl:try</code> reliably (to use a real
            example).</p>
               <p>Moreover, a formal mechanism defined in XML syntax (<em>namespace declaration</em> by way of attributes)
            enables associating the prefixes in a given scope with a controlled set of authority references (in the form
            of namespace URIs). Thus any namespace, considered as set of prefixed names under by a given declaration)
            can easily with its putative owner, proprietor or maintainer, in a way that lets them decide the fine
            distinctions: so <code>http://www.w3.org/1999/XSL/Transform</code> for XSLT,
               <code>http://www.w3.org/ns/xproc</code> for XProc and so forth: same maintainance umbrella, different
            applications.</p>
            </section>
            <section>
               <h2>Namespace fixup and namespace cleanup steps</h2>
               <p>Because they are heavily used in XSD and XPath for data typing, in XPath, XSLT and XQuery for extension
            functionality, XProc itself, and all its XML-based inputs and outputs, namespaces are inescapable – and they
            can be messy.</p>
               <p>To help, XProc specifies a process called <em>namespace fixup</em> that constitutes essentially a set of
            rewrite rules for a tree of elements and attributes with qualified names, in order to normalize and
            streamline the deployment of namespaces, with benefits in file size and legibility. You should think of
            namespace fixup as something the processor might do on your behalf at any time, and a good thing. Minimize
            its impacts by avoiding new namespaces where possible, and controlling namespaces in use.</p>
               <p>While namespace fixup will help, it cannot optimize for your case. The step <code>p:delete-namespace</code>
            is a common remedy for <em>inflamed namespaces</em>, a condition that happens to XML that has been handled
            recklessly by too many handlers with too many namespaces. Also, <code>p:namespace-rename</code>.</p>
            </section>
            <section>
               <h2>Namespace tips and tricks</h2>
               <p>A few small points to make namespace handling easier.</p>
               <section>
                  <h3>Coining new namespaces</h3>
                  <p>Do not be misled by the fact that namespaces are defined by means of URIs. The URIs point to pages that
               often do not exist. Making new namespaces to provide a rational scope for a new set of elements or
               functions is normal, and nothing needs to appear at that URI.</p>
                  <p>Developers may have non-technical reasons for using or avoiding certain URIs when doing so, but creating
               new XML namespaces is something that simply has to be done, if we are to use them as intended and
               consistently.</p>
               </section>
               <section>
                  <h3>On-the-fly namespace declarations</h3>
                  <p>When doing so, or when deploying any namespace declarations (the HTML namespace
                  <code>http://www.w3.org/1999/xhtml</code> is a frequent offender), the associated <code>xmlns:</code>
               attribute does <i>not</i> have to be placed only at the document root. Pipelines in the repository
               sometimes show namespaces declared internally, especially when their scope of application is limited.</p>
               </section>
               <section>
                  <h3>Overloading prefixes</h3>
                  <p>One problem presented by namespaces is the flip side of one of their most important features, that
               prefixes can be bound to names dynamically.</p>
                  <p>Consider this XML start tag for an XProc pipeline:</p>
                  <pre>&lt;p:declare-step xmlns:p="http://www.w3.org/ns/xproc" version="3.0"
   xmlns:ox="http://csrc.nist.gov/ns/oscal-xproc3"
   xmlns="http://csrc.nist.gov/ns/oscal-xproc3"
   xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
   xmlns:xs="http://www.w3.org/2001/XMLSchema"
   type="ox:a-step" name="a-step"&gt;</pre>
                  <p>Within the scope of this element, both unprefixed names such as <code>div</code> or <code>p</code> will
               identify the same elements as <code>ox:div</code> and <code>ox:p</code>. None of these elements are the
               familiar HTML <code>div</code> and <code>p</code> since the namespace is wrong – for HTML we would expect
               the HTML namespace, or none.</p>
                  <p>This pipeline might wish to scrub its namespaces with a <a href="../">p:namespace-delete</a> XProc step.
               Declarations are provided for the XSLT and XSD namespaces, which will accordingly appear (and be
               propagated) into literal elements given in the pipeline and hence, pipeline results.</p>
               </section>
               <section>
                  <h3>Matching with namespace wildcard</h3>
                  <p>In XPath since version 2.0, <code>*:p</code> will identify any <code>p</code> element in any namespace –
               not always pretty, but useful.</p>
               </section>
            </section>
         </section>
         <section class="unit learner"
                  id="oscal-convert_400"
                  data-track="learner">
            <h1>400: Documents in XProc</h1>
            <section>
               <h2>Goals</h2>
               <p>Learn how XProc's conception of a <em>document</em> can accommodate just about any kind of digital
            information.</p>
               <p>This is especially important with respect to XProc's handling of formats beyond XML, including HTML, JSON,
            plain text, text-based syntaxes and binaries.</p>
            </section>
            <section>
               <h2>Resources</h2>
               <p>A <a href="../tutorial/worksheets/CONTENT-TYPE_worksheet.xpl">content-types worksheet XProc</a> is offered for
            trying out <em>content-type</em> options on XProc inputs and outputs.</p>
               <p>XProc content types are closely tied to its definition of a <em>document</em>, an object for which a string
            property <code>content-type</code> indicates the expected or nominal content type. See links given in the
            remarks below. This is a topic you can also learn by through trial and error.</p>
               <p>For those who want the authoritative word on this topic, <a href="../">the XProc Specification</a> is open and available.
            This coverage also dovetails with what has already been presented on <a href="../tutorial/source/walkthrough/walkthrough_219_src.html"
                     class="LessonUnit">the XML Data Model (XDM) and XProc
            </a>.</p>
            </section>
            <section>
               <h2>What is an XProc document</h2>
               <p>XProc has a concept of a document that permits just about any digital object or encoded entity to be
            considered and handled as such. XProc manages this by dividing its world into precincts (more on this
            below), which may be relatively well-known and well-specified, or the opposite. Well-known things can be
            handled in well known ways. Things not so well known might still be handled by an extension – making them
            tractable without a standardized approach). Because the boundaries are not fixed, more kinds of things can
            be rendered as documents over time, assuming we can find ways to represent them using the data model at
            hand. The model, as noted, is provided by XDM.</p>
               <p>For an XProc engine to read XML, JSON, HTML and plain text is expected; for it to read (for example)
            metadata from inside image formats is a feature your XProc engine <i>may</i> offer. What XProc <q>sees</q>
            in either case is a <b>document</b>. There is a certain circularity here, if a document is <q>what XProc
               reads</q>, and what XProc reads are documents, but XProc is able to avoid this by saying, in effect:</p>
               <ul>
                  <li>Any XDM value can be a document in XProc, even values that are not <em>document nodes</em> in XDM. So an
               XProc document might contain an XML element tree or it might be a simple string such as <q>Hello
                  World</q>
                  </li>
                  <li>But anything else can also be a document in XProc, if a processor knows how to handle it in some way.
               For example, a step might know how to open an image format and extract metadata from it.</li>
               </ul>
               <p>Accordingly, XML or HTML files make documents <q>naturally</q> – reflecting the fact that XDM was designed
            for the purpose of representing them. But the tree-shaped element structure and data typing system of XDM
            can also be generalized.</p>
               <p>So any JSON file can also be a document, with the stipulation that like other inputs, it will be rendered
            (internally) by the XProc engine as an XDM object corresponding to this JSON. XDM - not a language or a
            syntax, but a data model – becomes the shared assumption across formats. Passed between steps in a pipeline,
            an XDM map is a document. At the end of the pipeline or at any interim point, an XDM map object can readily
            be saved out as JSON. An XDM element tree can be saved out as XML.</p>
               <p>Core to thinking about XProc is this notion that there are <b>documents</b>, and then there are
               <b>serializations</b> of documents - the way documents are encoded as character streams in editing
            applications or persistent storage – and these are not the same, albeit one may consdered as a
            representation of the other. An XML document stored in a system and an XML document as processed by XProc
            are related and isomorphic, but not necessary identical. An <a href="../">XProc document as defined formally</a> comprises an
            XDM (in this context called the <b>representation</b>) with certain properties for XProc's convenience such
            as a base URI (actual or nominal) and a content type (for disambiguation).</p>
               <p>Anything that can be rendered into XDM can be a document for XProc. XML and JSON come for free; other
            formats take more effort, or specialized steps (such as <a href="../">a step used to uncompress archive file formats</a>,
            thus converting one <q>document</q> into many). An XProc step supporting <a href="../">Invisible XML</a>, provided with a
            grammar, can be deployed to write specialized steps that are able to handle and render the format described
            by such a grammar.</p>
            </section>
            <section>
               <h2>A universe of content types</h2>
               <p>XProc needs a contract or working agreement with the world at large regarding how to refer to the various
            kinds of things XProc consumes and produces. The XProc concept and deployment of <code>content-type</code>
            properties is one of the main ways it does this. Read about more about XProc's content types<a href="../"> in the specification</a>.</p>
               <p>The short and easier story is that XProc's content types are aligned closely with media types or <q>file
               types</q> as they are defined broadly across Internet standards and protocols. Web developers will have a
            head start if they know how tools already distinguish between file and application formats in web
            technologies using identifiers such as <code>text/html</code> or <code>application/svg+xml</code>.</p>
               <p>The longer story is that by relying on content types, XProc can effectively divide the world, like Gaul,
            into three parts, thereby making it possible to divide and conquer – or at least to recognize and negotiate
            with.</p>
               <section>
                  <h3>XML and XML-like content</h3>
                  <p>Always our first choice for working in XProc, this category includes these content types:</p>
                  <ul>
                     <li>
                        <code>text/html</code> and <code>application/xhtml+xml</code> are the <a href="../">HTML media types</a>
                     </li>
                     <li>
                        <code>application/xml</code>, <code>text/xml</code>, and all types matching
                        <code>
                           <i>something</i>/<i>something</i>+xml</code> except <code>application/xhtml+xml</code>
                  constitute the <a href="../">XML media types</a>
                     </li>
                  </ul>
                  <p>Since XProc has XPath – and of course, by design – XML is entirely transparent and natural to work with.
               XSLT and XQuery give us powerful and optimable methods for transformations and queries at scale, as soon
               as the source is recognizable as some kind of XML.</p>
                  <p>Since it is either XML (whether as XHTML or as HTML that happens to be well-formed XML syntax) or has
               well-defined mappings into XML (for example, <code>&lt;br&gt;</code> is parsed as an empty element), HTML
               can count as a specialized form of XML for XProc purposes. This is a joy: it means HTML is no different
               except in how it is read and written, and sometimes not even then. In particular, within XProc, XPath and
               other XML tools (such as XSLT and Schematron) both work with HTML in the same way as they do with
               XML.</p>
               </section>
               <section>
                  <h3>JSON and other text-based formats</h3>
                  <ul>
                     <li>
                        <code>text/<i>something</i>
                        </code> including <code>text/plain</code>, except for
                     <code>text/xml</code> and <code>text/html</code> are the <a href="../">text media types</a>
                     </li>
                     <li>
                        <code>application/json</code> and any <code>application/<i>something</i>+json</code> are the <a href="../">JSON media types</a>
                     </li>
                  </ul>
                  <p>Both these types are accommodated straightforwardly as XDM objects: in the case of plain text, we have
               what amounts to a string, while JSON becomes an appropriate XDM type, most often a map, which is
               isomorphic to a JSON object with properties.</p>
                  <p>Additionally, the <a href="../">XProc step
                     <code>p:ixml</code>
                     </a> provides for parsing any text-based format, provided with a grammar, into
               an XML structure.</p>
                  <p>As noted above, however, an XProc document does not actually require an element structure: it can be any
               XDM object when provided with internal properties for handling it. As such, any XDM can be passed through
               an XProc pipeline, operated on by steps and made available on their ports.</p>
                  <p>Thus while a common way of handling any information in XProc is to present an XML representation for it,
               that is not always necessary. Readers who have no prior experience with XPath 3.1 (or later) may never
               have seen the XPath <em>map</em> and <em>array</em> objects or the functions and operations associated
               with them. Some syntax might look like (and it is no mistake if this is familiar to JSON users):</p>
                  <table>
                     <tbody>
                        <tr>
                           <th>XPath expression</th>
                           <th>Explanation</th>
                        </tr>
                        <tr>
                           <td>
                              <code>map { "a": "A", "b": "B" }</code>
                           </td>
                           <td>A map with two entries, with keys "a" and "b" </td>
                        </tr>
                        <tr>
                           <td>
                              <code>map { "a": "A", "b": "B" }?a</code>
                           </td>
                           <td>The value "A" returned from a map, using <code>?</code>, the lookup operator</td>
                        </tr>
                        <tr>
                           <td>
                              <code>let $m := map { "a": "A", "b": "B" } return $m('b')</code>
                           </td>
                           <td>The value "B" returned, using function call syntax</td>
                        </tr>
                     </tbody>
                  </table>
                  <p>In XPath, a map's keys (keywords for its entries) can be any atomic value, and an entry's value can be
               anything, including nodes (roots of trees) or arbitrary sequences. Because this fully comprehends the
               JSON object types, any JSON can be read into an XDM map - while the reverse is not always the case. For
               example, using the <q>JSON</q> serialization rules, serializing an XDM map with a node tree on a property
               will write that tree out, using XML syntax, into the property.</p>
                  <p>See also: <a href="../">XPath 3.1 on maps,
                  arrays and JSON</a>.</p>
               </section>
               <section>
                  <h3>Binaries and what-have-you</h3>
                  <p>Because they cannot have a uniform parsing strategy, support for any content types not named above
               necessarily depends on implementors providing XProc steps to handle them. Certain standard steps such as
                  <code>p:unarchive</code> and its relatives are useful to define for certain at-large binary file
               types, with effective standardization for certain content types (compressed files, i.e. zips and the
               like). But handling for arbitrary or unknown files and file types is a different matter.</p>
                  <p>Apart from XML, HTML, and text formats that can be parsed into an XDM object – whether a string, a map
               (reading JSON syntax) or an XML structure defined by iXML – any other digital data will fall into this
                  <q>anything else</q> bucket. Inevitably, mileage varies. Depending on your processor some binary
               formats such as raster image formats might be effectively opaque, with only some metadata readable; other
               formats such as compression formats are simply wrappers for other resources. The semantics will vary
               according to the processor and file type(s) implicated.</p>
                  <p>This does not mean that such content types cannot be effectively standardized and supported as such. One
               important member of this category (as just noted) is an archived (zipped) file set, as defined for the
               three steps <code>p:archive</code>, <code>p:archive-manifest</code>, and <code>p:unarchive</code>. Due to
               felicitous choices by early engineers, common word processor and spreadsheet formats including
                  <code>docx</code>, <code>xlsx</code> (Microsoft's <a href="../">Office Open XML</a>, ECMA-376 /
               ISO/IEC 29500:2008), <code>odt</code> and <code>ods</code> (<a href="../">Open Document Format</a>) can be unzipped to reveal
               sets of files including legible XML files – making these formats at least relatively accessible for
               XProc. Along similar lines, the step <code>p:uncompress</code> can handle the GZIP compression
               format.</p>
                  <p>
                     <code>content-type="*/*"</code> is a match for <q>any content type</q>, very useful for handling
               arbitrary inputs, to the extent the processor supports them. Often even if a processor cannot read a
               file's contents meaningfully into a pipeline for manipulation, it can nevertheless work on the file
               resource, for example to copy, move or delete it.</p>
               </section>
            </section>
         </section>
         <section class="unit maker"
                  id="oscal-convert_401"
                  data-track="maker">
            <h1>401: XProc, XML, JSON and content types</h1>
            <section>
               <h2>Goals</h2>
               <p>Understand a little more about JSON and other data formats in an XML processing environment.</p>
            </section>
            <section>
               <h2>Resources</h2>
               <p>A <a href="../tutorial/worksheets/CONTENT-TYPE_worksheet.xpl">content-types worksheet XProc</a> is offered for
            trying out content-type options on XProc inputs and outputs.</p>
               <p>The pipeline <a href="../tutorial/worksheets/READ-JSON-TESTING.xpl">READ-JSON-TESTING.xpl</a> provides an
            experimental surface for working functionality specifically related to JSON and XDM map objects.</p>
            </section>
            <section>
               <h2>Exercise some options</h2>
               <p>The worksheets just cited provide an opportunity to try out <code>content-type</code> configuration options.
            Note how you can specify a content type that will serve as a constraint on inputs and outputs, analogous in
            some ways to the type signature on a function. And the step <code>p:content-type</code> serves to cast one
            content type to another, according to <a href="../">rules
               given in the Specification</a>. Note that for this step to work, both inputs and outputs must conform to
            certain requirements: not everything can be cast!</p>
               <ul>
                  <li>Try specifying a <code>content-type</code> on input or outputs to constrain the inputs or outputs,
               producing errors for mismatches</li>
                  <li>Try using a <a href="../">p:cast-content-type</a> step
               to convert between content types.</li>
                  <li>Use the function <code>p:document-properties(.,'content-type')</code> to bring back the content type of
               a document on a port or in a pipeline. In XPath, <code>.</code> refers to an designated as the (dynamic)
               processing context: so <code>p:document-properties($doc,'content-type')</code> works for any $doc
               considered by XProc to be or serve as a <em>document</em>.</li>
                  <li>Interestingly, this means we can expect to find <code>content-type="application/json"</code> whenever an
               XProc document is nothing more than an object or map – as can happen, by design.</li>
               </ul>
               <p>
                  <a href="../tutorial/worksheets/READ-JSON-TESTING.xpl">READ-JSON-TESTING.xpl</a> is a sandbox for playing with
            JSON objects as XDM maps. The <a href="../tutorial/worksheets/CONTENT-TYPE_worksheet.xpl">content-types
               worksheet</a> is set up for trying content-type options on inputs and outputs.</p>
            </section>
         </section>
      </section>
      <section class="lesson"
               id="courseware"
               name="courseware">
         <section class="unit observer"
                  id="courseware_101"
                  data-track="observer">
            <h1>Courseware 101: Producing this tutorial</h1>
            <section>
               <h2>Goals</h2>
               <p>Understand better how this tutorial is produced.</p>
               <p>See an example of a small but lightweight and scalable publishing system can be implemented in XProc and
            XSLT.</p>
            </section>
            <section>
               <h2>Prerequisites</h2>
               <p>None.</p>
               <p>Readers who wish not only to inspect and refresh tutorial contents, but also to edit or extend, or alter the
            tutorial pipelines, are invited to look at <a href="../tutorial/source/courseware/courseware_219_src.html"
                     class="LessonUnit">the next
               exercise (lesson unit)</a>.</p>
            </section>
            <section>
               <h2>Resources</h2>
               <ul>
                  <li>Everything in the course you have seen so far</li>
                  <li>Everything else you have seen relating to XProc</li>
                  <li>Your own problems</li>
                  <li>Your own examples</li>
               </ul>
            </section>
            <section>
               <h2>Tutorial production pipelines</h2>
               <p>The source files for these tutorials are all stored in the repository in a source directory, and maintained
            in HTML format. The expectation is that all files will be represented as valid and complete HTML 5 in XML
            syntax, i.e. XHTML to be treated as an HTML document type. However, not all files in this subdirectory are
            used as source materials; this is only where they are stored and maintained.</p>
               <p>At the top of the tutorial structure, a file <a href="../tutorial/lesson-plan.xml">lesson-plan.xml</a> is used as
            a publication driver or configuration file, when read as a sequence of links back to folders. Each folder
            named as a <code>Lesson/@key</code> is read, with all HTML files surnamed <code>_src.html</code> from that
            folder considered to be tutorial source materials, in nominally alphabetic order. The tutorial as a whole is
            built by reading each of these files, converting it to Markdown (with adjustments), and producing indexes to
            the entire set, all done by writing files into the <a href="../tutorial/source/courseware/sequence/">sequence directory</a>. Such a
            Markdown file set can be read and edited in Github in the usual way, or refreshed by running a production
            pipeline again.</p>
               <p>The authoring model and its design rationales as well as its tooling support, including automated proof
            checking under CI/CD, is described in a <a href="../tutorial/source/courseware/courseware_219_src.html"
                     class="LessonUnit">subsequent
               lesson unit</a>. Authoring in HTML rather than Markdown provides all kinds of leverage for ad-hoc
               <q>tutorial semantics</q> especially with the help of a structured (tagless) editor.</p>
               <p>Each of the pipelines at the top performs a specific task with respect to tutorial production.</p>
               <p>As always, pipelines should be provided with explanatory comments.</p>
               <p>When reading and writing files to file systems, the usual security considerations apply.</p>
               <section>
                  <h3>
                     <a href="../tutorial/PRODUCE-TUTORIAL-PREVIEW.xpl">PRODUCE-TUTORIAL-PREVIEW</a>
                  </h3>
                  <p>Generates a preview (reading) version of the tutorial, for proofing.</p>
               </section>
               <section>
                  <h3>
                     <a href="../tutorial/PRODUCE-TUTORIAL-MARKDOWN.xpl">PRODUCE-TUTORIAL-MARKDOWN</a>
                  </h3>
                  <p>Generates Markdown from the HTML source data, populating an output folder with these results.</p>
               </section>
               <section>
                  <h3>
                     <a href="../tutorial/PRODUCE-TUTORIAL-TOC.xpl">PRODUCE-TUTORIAL-TOC</a>
                  </h3>
                  <p>Generates a Table of Contents in Markdown, saving it as <a href="../tutorial/sequence/lesson-sequence.md">lesson-sequence.md</a> in a location where it will function on publication (by the repository).</p>
               </section>
               <section>
                  <h3>
                     <a href="../tutorial/PRODUCE-PROJECTS-ELEMENTLIST.xpl">PRODUCE-PROJECTS-ELEMENTLIST</a>
                  </h3>
                  <p>Generates an index of XProc elements used in projects in the repository, stored as<a href="../tutorial/sequence/element-directory.md">element-directory.md</a>.</p>
               </section>
            </section>
         </section>
         <section class="unit maker"
                  id="courseware_219"
                  data-track="maker">
            <h1>Courseware 219: Learn by Teaching</h1>
            <section>
               <h2>Goals</h2>
               <p>Help yourself, your team and allies.</p>
               <p>Produce a useful spin-off from a task or problem you need to master anyway.</p>
               <p>Learn not only by doing but by writing it down for yourself and others.</p>
            </section>
            <section>
               <h2>Prerequisites</h2>
               <p>Those with no prior experience in XML-based publishing or declarative markup might be venturing into deep
            waters. At the same time there is nothing here that will faze readers who have experience editing HTML or
            XML.</p>
               <p>Writing HTML by hand can be arduous; accordingly, for producing tutorial pages we have used a structured XML
            authoring environment <a href="../">(oXygen XML Author)</a>. While
            this excellent software is not necessary, it provides with many features including styling in display; full
            control over styling; testing Schematron rules in the background along with UI support for content
            corrections according to those rules; etc. A tool such as this or its functional equivalent is highly
            recommended – again demonstrating the value of a standards-based encoding.</p>
               <p>However, any text editor or programmers' coding environment also works (to whatever extent generic HTML is
            supported), and Schematrons applied to HTML files can be run in XProc (as described).</p>
            </section>
            <section>
               <h2>Resources</h2>
               <ul>
                  <li>Everything in the course you have seen so far</li>
                  <li>Everything else you have seen relating to XProc</li>
                  <li>Your own problems</li>
                  <li>Your own examples</li>
               </ul>
            </section>
            <section>
               <h2>Improve or enhance a lesson or lesson unit</h2>
               <p>Alert readers will have observed that a Markdown-based deployment invites editing. But the authoring or data
            acquisition model of this tutorial is not Markdown-based - Markdown is being used here in its usual way,
            instead serving as one of several publication formats for this data set. Source data itself is being written
            and maintained in in an XML-based HTML5 tag set defined for the project. By writing, querying and indexing
            in XHTML we can use XProc from the start. The HTML dialect means things <q>just work</q> using HTML tools
            such as web browsers, while we have transformations (provided in pipelines) to render into any other
            publication format we may happen to need. Markdown is one of a range of choices.</p>
               <p>Extensibility and flexibility is one of the strengths of this approach: to publish a new or rearranged
            tutorial sequence can be done with a few lines and commands. A drag and drop interface supporting XProc
            makes this even easier, while again XProc is already supported under CI/CD, meaning both editorial and code
            quality checks can be done with every commit by simply listing the appropriate pipeline with others.</p>
               <p>The workflow supporting this publication model is simple. A set of lessons (lesson units) is gathered
            together in a single folder. That folder is listed in <a href="../tutorial/lesson-plan.xml">a directory file</a>.
            To publish the tutorial, an XProc pipeline is executed that polls these directories and produces Markdown
            files corresponding to the inputs, only in a new sequence with links rewritten.</p>
               <p>Adding a page is as simple as copying and altering a page in place, and seeing to it the new page is valid
            to both HTML5 and project requirements, and running the production pipeline with the new or edited
            sources.</p>
               <p>Other pipelines can be run to update the directory to lesson units and other higher-level production such as
            a single-page HTML reading preview version. See <a href="../tutorial/source/courseware/courseware_101_src.html"
                     class="LessonUnit">the
               earlier treatment</a> for more details.</p>
               <p>Improving a page is as simple as editing the HTML source in the folder and running pipelines to refresh its
            results.</p>
               <section>
                  <h3>Apply Schematron to your edits</h3>
                  <p>
                     <a href="../tutorial/src/lesson-html.sch">A Schematron for lesson unit files</a> also ensures that links are in
               place and project conventions are observed.</p>
                  <p>You will be grateful to do this interactively in an editor that supports Schematron in the
               background.</p>
               </section>
            </section>
            <section>
               <h2>Create a new set of lessons</h2>
               <p>Create a new folder in the <code>tutorial/source</code> subdirectory, and list it in<a href="../tutorial/lesson-plan.xml">the tutorial lesson plan XML</a>. Provide your new lesson plan with a new
            title and version.</p>
               <p>(You may find other work-in-progress already there: feel free to help.)</p>
               <p>When production pipelines are run, all HTML files present in the newly-listed folders will be included to
            the tutorial as lesson units. Within the folder they will be listed alphabetically. Be sure that your new
            HTML is also Schematron-valid to <a href="../tutorial/src/lesson-html.sch">the Schematron here</a>.</p>
            </section>
            <section>
               <h2>Produce a new project and document it with a tutorial</h2>
               <p>You could make a new lesson topic with lessons on your own new project.</p>
               <p>This repository is provided with a <a href="../project-template/readme.md">project template</a> to
            make it easier to get started with new pipelines for new applications. Start a new project by copying this
            folder into the <code>projects</code> folder, renaming it, and proceeding to edit its file contents.</p>
            </section>
         </section>
      </section>
   </body>
</html>