diff --git a/changelog.d/20240102_154417_jsick_1.md b/changelog.d/20240102_154417_jsick_1.md new file mode 100644 index 00000000..2a756bae --- /dev/null +++ b/changelog.d/20240102_154417_jsick_1.md @@ -0,0 +1,3 @@ +### Other changes + +- Update the migration docs around the migration tool and convert the previous manual migration docs into a reference of the file-by-file changes. diff --git a/docs/technotes/migrate.rst b/docs/technotes/migrate.rst index 7e9100c8..b2a9c23e 100644 --- a/docs/technotes/migrate.rst +++ b/docs/technotes/migrate.rst @@ -1,32 +1,31 @@ -##################################### -Migrate legacy Sphinx technical notes -##################################### +:og:description: Rubin Observatory has a new technical note format for Markdown and reStructuredText documents. This page provides a guide to migrating legacy reStructuredText/Sphinx technotes to the current format. -Introduced in December 2023, Rubin Observatory has a new technical note format for Markdown and reStructuredText documents, based on the Sphinx_ and `Technote `__ Python packages. -If you started writing your technical note before December 2023, it is likely in the legacy Sphinx format. -You can tell if your technical note is in the legacy format if it has a :file:`metadata.yaml` file. -If you are continuing to write such a technical note, you should migrate it to the new format to take advantage of new features. -This page provides a guide to migrating your technical note to the new format. +###################################################### +Migrate legacy reStructuredText/Sphinx technical notes +###################################################### -To get started on your migration, be sure to have your technote repository checked-out locally. -Work on a ticket branch, `per the Developer Guide `__, so you can check your work with a pull request before merging to the ``main`` branch. +Introduced in December 2023, Rubin Observatory has a new technical note format for Markdown and reStructuredText documents. +This update provides many `new features `__ included a responsive and branded page design, Markdown support, improved build configuration, automated bibliography management, and citation-ready metadata. -.. _technote-migration-tool: +This page shows how to migrate your existing technotes with a command-line tool that largely automates the process. +This page also provides file-by-file notes in case you need to revise the migration. -Running the migration tool -========================== +.. tip:: + + Your technote is in the legacy format if it has a :file:`metadata.yaml` file at the top level of its repository. -Documenteer provides a migration tool, :command:`documenteer technote migrate`, that can help automate this process. + LaTeX-format technotes are unaffected by this migration. -If you cannot use this tool, you can follow the `detailed steps <#technote-migration-detailed>`__ below. -For example, if you have a customized build process, you may want to manually migrate your technote. -Alternatively you can run the migration tool, and then use Git to unstage any changes you don't want to commit. +Step 1. Install pipx +==================== -Preparation ------------ +The migration tool needs **Python 3.11 or later**. +You can verify this by running ``python --version`` from your shell. -To run the migration command in an isolated Python environment, without affecting other projects, you can use the ``pipx`` tool. -First, install ``pipx``: +This official migration procedure uses the pipx_ tool to install and run the migration tool in an isolated Python environment. +You can verify that pipx_ is installed by running ``pipx --version`` from your shell and checking for a version number. + +If you don't have the :command:`pipx` command-line tool already, you can install ``pipx`` several ways: .. tab-set:: @@ -48,50 +47,125 @@ First, install ``pipx``: brew install pipx -.. important:: +Step 2. Clone your technote and create a branch +=============================================== - Documenteer, and therefore pipx, need Python 3.11 or later. +Since the migration is performed locally, you need to clone the repository: -Run the migration ------------------ +.. code-block:: bash -From the root of the cloned technote repository, you can run the migration tool: + git clone https://github.com/{org}/{repo} -.. prompt:: bash +Then create a branch for the migration (see the `Developer Guide `_ for more information on branching): - pipx run documenteer technote migrate +.. code-block:: bash -.. tip:: + git switch -c {...} - You can also add author information during the migration. - Open `authordb.yaml`_ on GitHub and find the keys that identify the technote's authors. - For example, ``sickj`` is the key for Jonathan Sick. - Then run the migration command with a ``-a`` option for each author: +Working from a branch allows you to create a pull request to verify the migration before merging it into the main branch. - .. prompt:: bash +Step 3. Look up author IDs +========================== + +The migration tool needs the IDs of the technote's authors to fully configure the technote's metadata. + +Open `authordb.yaml`_ on GitHub and find the YAML keys that identify the technote's authors. +For example, ``sickj`` is the ID for Jonathan Sick. + +.. _technote-migration-tool: + +Step 4. Run the migration tool +============================== + +From the root of the cloned technote repository, run the migration tool, listing any identified authors that are relevant to your technote: - pipx run documenteer technote migrate -a sickj -a economouf +.. code-block:: bash + :caption: Set author IDs with the -a option + + pipx run documenteer technote migrate -a sickj -a economouf When the migration runs, it shows a summary of the files changed and deleted. -Use ``git diff`` to see the changes in detail. -You may want to tweak those changes before committing them. +Use ``git diff`` to review the changes in case you need to make tweaks before committing. + +If you want to learn more about the changes made by the migration tool, you can review the :ref:`detailed changes, below `. + +.. _technote-migration-markdown: + +Step 5 (optional). Migrate to Markdown +====================================== + +The new technote format supports Markdown as well as reStructuredText. +If you wish to use Markdown, you can use MyST Parser's migration tool: + +.. prompt:: bash + + pipx install "rst-to-myst[sphinx]" + rst2myst convert index.rst + +The Markdown flavor used by the new technote format is MyST Markdown, which is a superset of CommonMark Markdown with support for Sphinx roles and directives. +See the `MyST Parser documentation `__ for more information. + +Step 6 (optional). Build the technote locally +============================================= + +You can build the technote locally to verify that the migrated technote compiles successfully. + +GitHub Actions will test the technote build when you open a pull request (next step), so you can skip this step. +However, you can get faster feedback by building locally. + +.. prompt:: bash + + make init + make html -If you want to learn more about the changes made by the migration tool, you can review the :ref:`detailed steps below `. +The technote's website will be in the ``_build/html`` directory. +Open the ``index.html`` file in your web browser to view the technote: + +.. prompt:: bash + + open _build/html/index.html + +You can repeat the :command:`make html` command to rebuild the technote after making changes. + +You might also want to run the linter to check links and find common issues: + +.. prompt:: bash + + make lint + +If you have any questions or issues about the build, you should still proceed with committing and creating a pull request (see next step). +This way you can link to the repository when you reach out for help in `#dm-docs-support`_ on Slack. + +Step 7. Commit the migration, pull request, and merge +===================================================== + +At this point, you should have a working technote in the new format. +If you haven't already, commit your work, push your branch to the GitHub repository, and open a pull request. +GitHub Actions will build the technote and publish a preview version that is linked from the ``/v`` path of your technote's website. + +If the build works, you can merge the pull request. + +If there are build errors, you can reach out to `#dm-docs-support`_ on Slack for help. +Include the repository URL and ideally a link to the pull request or GitHub Actions workflow run that failed. .. _technote-migration-detailed: -Detailed steps -============== +Migration details +================= -Step 1. Create a technote.toml file ------------------------------------ +The migration is automated by the :command:`documenteer technote migrate` command, as described above. +This section describes the steps performed by the migration tool in detail in case you need to make adjustments or understand the changes made to your technote. + +technote.toml file (added) +-------------------------- The :file:`technote.toml` file replaces the original :file:`metadata.yaml` file. This new file provides both metadata and Sphinx configuration for your document. -Here is a simple :file:`technote.toml` file to get you started: +Here is a simple :file:`technote.toml` file: .. code-block:: toml + :caption: technote.toml [technote] id = "EXAMPLE-000" @@ -101,6 +175,9 @@ Here is a simple :file:`technote.toml` file to get you started: github_default_branch = "main" date_created = 2015-11-18 date_updated = 2023-11-01 + organization.name = "Vera C. Rubin Observatory" + organization.ror = "https://ror.org/048g3cy84" + license.id = "CC-BY-4.0" [[technote.authors]] name = {given = "Drew", family = "Developer"} @@ -110,13 +187,6 @@ Here is a simple :file:`technote.toml` file to get you started: name = "Rubin Observatory Project Office" internal_id = "RubinObs" -When you're done, add this file to your repository and commit it: - -.. prompt:: bash - - git add technote.toml - git commit -m "Add technote.toml file" - .. note:: The schema for this file is described in the `Technote package documentation `__, and elsewhere in the :doc:`Documenteer documentation for Rubin technotes `. @@ -131,7 +201,7 @@ When you're done, add this file to your repository and commit it: - ``date_updated`` is an optional field that specifies when the technote was last updated. If you omit this field, the current date is used. - Each author is specified with a ``[[technote.authors]]`` table (in TOML, the double brackets represent a table in an **array of tables**). Use the :command:`mamke add-author` command to add an author to this file using data from `authordb.yaml`_. It's important to use the ``internal_id`` field to identify authors with their corresponding key in `authordb.yaml`_. This enables Documenteer to update author information with the :command:`make sync-authors` command. -Step 2. Update conf.py +conf.py file (updated) ---------------------- The :file:`conf.py` file directly configures the Sphinx build process. @@ -139,25 +209,22 @@ New technotes use a different configuration set provided by Documenteer that use For most technotes, the :file:`conf.py` file should be a single line: .. code-block:: python + :caption: conf.py from documenteer.conf.technote import * # noqa: F401, F403 -Commit any changes: - -.. prompt:: bash - - git add conf.py - git commit -m "Update conf.py to new technote format" - If your :file:`conf.py` file has additional content, some of that configuration may be migrated to :file:`technote.toml`. Reach out to `#dm-docs-support`_ on Slack for advice. -Step 3. Update the index.rst file ---------------------------------- +index.rst file (updated) +------------------------ The :file:`index.rst` file is the main content file for your technical note. The new technote format requires some changes to this file: the title is now part of the content, the abstract is marked up with a directive, status information is now part of :file:`technote.toml`, and the configuration for the reference section is dramatically simplified. +Additionally, the new technote format supports Markdown as well as reStructuredText. +See :ref:`migrate to Markdown ` to learn how to switch to Markdown. + .. tip:: Besides these changes, your technote can also be written in Markdown (:file:`index.md`). @@ -178,8 +245,8 @@ The new technote format requires some changes to this file: the title is now par git add index.md git commit -m "Switch to Markdown format" -Add the title -~~~~~~~~~~~~~ +Title +~~~~~ The title is now part of the content, not the metadata. Add the title to the top of the content: @@ -204,15 +271,15 @@ Add the title to the top of the content: # Example technical note -Move document status information to technote.toml -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Document status +~~~~~~~~~~~~~~~ The original technote format used a ``note`` directive to describe whether the document was a draft or deprecated. Now this status metadata is structured in :file:`technote.toml`. Delete the ``note`` directive and add the status information to :file:`technote.toml` following :doc:`document-status`. -Format the abstract/summary with the abstract directive -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Abstract +~~~~~~~~ Legacy technotes either provided an abstract or summary through the ``description`` field in :file:`metadata.yaml`, in a ``note`` directive in :file:`index.rst`, or in a content section in :file:`index.rst`. The new technote format uses an ``abstract`` directive to mark up the abstract/summary. @@ -254,8 +321,8 @@ The new technote format uses an ``abstract`` directive to mark up the abstract/s [... content below ...] -Simplify the reference section -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Reference section +~~~~~~~~~~~~~~~~~ If your technote makes references to other documents with roles like :external+sphinxcontrib-bibtex:rst:role:`cite`, you'll need a reference section to display the bibliography. In the new technote format, this section is simplified: @@ -291,26 +358,7 @@ Specifically: - The references section should be a regular section, not a "rubric." - The bibliography directive no longer requires any configuration; all configuration is provided by Documenteer. -Commit any changes -~~~~~~~~~~~~~~~~~~ - -.. tab-set:: - - .. tab-item:: rst - - .. prompt:: bash - - git add index.rst - git commit -m "Reformat index.rst for new technote format" - - .. tab-item:: md - - .. prompt:: bash - - git add index.md - git commit -m "Reformat index.md for new technote format" - -Step 4. Delete metadata.yaml +metadata.yaml file (deleted) ---------------------------- At this point, all relevant metadata about the technote is in :file:`technote.toml` or :file:`index.rst`/:file:`index.md`. @@ -319,10 +367,9 @@ Delete the deprecated :file:`metadata.yaml` file: .. prompt:: bash git rm metadata.yaml - git commit -m "Remove metadata.yaml file" -Step 5. Delete the lsstbib/ directory -------------------------------------- +lsstbib/ directory (deleted) +---------------------------- The legacy technote format vendored Rubin BibTeX bibliography files from https://github.com/lsst/lsst-texmf. The new technote format automatically downloads and caches these files so that you no longer need to commit them into your repository. @@ -331,9 +378,8 @@ Delete the :file:`lsstbib` directory: .. prompt:: bash git rm -r lsstbib - git commit -m "Remove lsstbib/ directory" -Step 6. Update .gitignore +.gitignore file (updated) ------------------------- The new technote format introduces additional directories that should be ignored by Git. @@ -343,13 +389,8 @@ Ensure at least the following paths are included in the :file:`.gitignore` file: :language: text :caption: .gitignore -.. prompt:: bash - - git add .gitignore - git commit -m "Update .gitignore file" - -Step 7. Set up pre-commit hooks -------------------------------- +.pre-commit-config.yaml file (added) +------------------------------------ Pre-commit_ is a Python package that runs validation and formatting checks on your technote's repository before you commit. Although it is not required, it's highly recommended that you set up pre-commit hooks for your technote. @@ -359,19 +400,12 @@ To start, add a :file:`.pre-commit-config.yaml` file: :language: yaml :caption: .pre-commit-config.yaml -Commit any changes: - -.. prompt:: bash - - git add .pre-commit-config.yaml - git commit -m "Add pre-commit configuration" - .. tip:: You can add additional pre-commit hooks to this file to suite your needs. See Pre-commit's `directory of available hooks `__ for ideas. -Step 8. Update requirements.txt +requirements.txt file (updated) ------------------------------- The Python dependencies for your technote are listed in a :file:`requirements.txt` file that should now look like this: @@ -380,19 +414,12 @@ The Python dependencies for your technote are listed in a :file:`requirements.tx :language: text :caption: requirements.txt -Commit any changes: - -.. prompt:: bash - - git add requirements.txt - git commit -m "Update requirements.txt file" - .. note:: If your technote has additional dependencies listed, you can reach out to `#dm-docs-support`_ on Slack if you are unsure whether they are part of the Sphinx build process or separate packages needed for any custom document preprocessing. -Step 9. Add a tox.ini file --------------------------- +tox.ini file (added) +-------------------- Tox_ is a tool for running Python programs in dedicated virtual environments. This makes your local technote builds more reproducible by separating the technote's dependencies from your system and other projects. @@ -403,8 +430,8 @@ This is the recommended tox configuration to start with: :language: ini :caption: tox.ini -Step 10. Update the Makefile ----------------------------- +Makefile file (updated) +----------------------- The :file:`Makefile` file provides a simple entrypoint for building your technote and performing other common tasks. This is the suggested content for your :file:`Makefile` that works with the tox and pre-commit configurations: @@ -413,8 +440,8 @@ This is the suggested content for your :file:`Makefile` that works with the tox :language: make :caption: Makefile -Step 11. Update GitHub Actions workflows ----------------------------------------- +.github/workflows/ci.yaml file (added/updated) +---------------------------------------------- Recent technotes have already migrated their GitHub Actions workflows to use the reusable workflow from https://github.com/lsst-sqre/rubin-sphinx-technote-workflows. Check the :file:`.github/workflows/ci.yaml` file to make sure it looks like this: @@ -437,13 +464,6 @@ Check the :file:`.github/workflows/ci.yaml` file to make sure it looks like this Replace ``example-001`` with your technote's handle (the subdomain of ``lsst.io``). -If necessary, commit any changes: - -.. prompt:: bash - - git add .github/workflows/ci.yaml - git commit -m "Update GitHub Actions workflow" - .. note:: The original Rubin technotes used Travis CI for continuous integration and deployment, but we no longer use that service. @@ -455,10 +475,9 @@ If necessary, commit any changes: .. prompt:: bash git rm .travis.yml - git commit -m "Remove Travis configuration" -Step 12. Add dependabot support -------------------------------- +.github/dependabot.yml file (added) +----------------------------------- Dependabot is a service provided by GitHub that generates pull requests when there are new versions of your technote's dependencies. Set up Dependabot by adding a :file:`.github/dependabot.yml` file: @@ -467,15 +486,8 @@ Set up Dependabot by adding a :file:`.github/dependabot.yml` file: :language: yaml :caption: .github/dependabot.yml -Commit the changes: - -.. prompt:: bash - - git add .github/dependabot.yml - git commit -m "Add dependabot configuration" - -Step 13. Update the README --------------------------- +README.rst or README.md file (updated) +-------------------------------------- The README for a legacy-format technote likely has outdated information about how to build the technote. Here is a suggested README template for technotes in the new format: @@ -488,34 +500,8 @@ Here is a suggested README template for technotes in the new format: :language: rst :caption: README.rst - Commit any changes: - - .. prompt:: bash - - git add README.rst - git commit -m "Update README for new technote format" - .. tab-item:: md .. literalinclude:: _templates/README.md :language: md :caption: README.md - - Commit any changes: - - .. prompt:: bash - - git add README.md - git commit -m "Update README for new technote format" - -Step 14. Merge the migration ----------------------------- - -At this point, you should have a working technote in the new format. -If you haven't already, push your branch to the GitHub repository and open a pull request. -GitHub Actions will build the technote and publish a preview version that is linked from the ``/v`` path of your technote's website. - -If the build works, you can merge the pull request. - -If there are build errors, you can reach out to `#dm-docs-support`_ on Slack for help. -Include the repository URL and ideally a link to the pull request or GitHub Actions workflow run that failed.