Citations page in addition to references page

[rickecon]

This is a hard problem to solve with our documentation. We use Jupyter Book to create the OG-USA documentation, which is available at https://pslmodels.github.io/OG-USA/ and the source code of which is in the OG-USA/docs/book/ directory of the OG-USA repository.

Some past PR broke the compilation of any of the in-text citation references (see broken reference here) or any compilation of the references in the References section or in the Citations section of the documentation. These issues were discussed in Issue #613, and partially solved in subsequent PRs (PR #634 and PR #664). However, it is not clear if the Citations page compilation worked after PR #664 or if it was subsequently broken.

The following is the behavior that I think we want. And I think this will not be straightforward to execute.

1. References page compiles all references cited in the Jupyter Book document, built off of OGUSA_references.bib file. PR Big documentation update #683 fixes this. However, PR Big documentation update #683 does not currently address the following two issues.
2. Citations page compiles separately being built off of citations.bib file. We want this page to compile a reference line for every reference in citations.bib, independent of overlap with OGUSA_references.bib and independent of whether any citations.bib references are cited in the text.
3. We want the citations.bib file to be able to be scraped the the PSL Models organization for compilation of OG-USA specific references on the PSL Models web page. I think the natural location for citations.bib is in the same directory as the main BibTeX file OGUSA_references.bib. citations.bib is currently at the main level of the OG-USA repo (OG-USA/citations.bib).

cc: @jdebacker @MaxGhenis

[jdebacker]

@rickecon I hadn't even noticed that the references and citations were not showing up!

I'm glad to see the references are fixed.

Citations page compiles separately being built off of citations.bib file. We want this page to compile a reference line for every reference in citations.bib, independent of overlap with OGUSA_references.bib and independent of whether any citations.bib references are cited in the text.

Right, this is why citations.md has the :all : statement. This should produce the above behavior (at least it did for some version of JB - again, not sure what broke and it compiles fine locally for me using JB version 0.9.1.

We want the citations.bib file to be able to be scraped the the PSL Models organization for compilation of OG-USA specific references on the PSL Models web page. I think the natural location for citations.bib is in the same directory as the main BibTeX file OGUSA_references.bib. citations.bib is currently at the main level of the OG-USA repo (OG-USA/citations.bib).

We do want it to be scraped, which his why it is important this file resides in the top-level directory, since different repos put their docs in different directory structures. That makes it easy for the scaper to find the file. This doesn't impact it's compilation in the Jupyter-Book.

[rickecon]

@jdebacker. My Juypter Book version is 0.10.0. I wonder if my need in PR #683 to add the bibtex_bibfiles: lines is new to v. 0.10.0. We might have to pin the Jupyter Book version to 0.9.1 in the ogusa-dev environment until we find a solution.

[rickecon]

I just pinned the Jupyter Book environment to v. 0.9.1 in PR #683. And now the citations page compiles fine. I wonder if it will work in version 0.10.0 if we add another reference line to the citations.bib at the bottom of the _config.yml file.

[jdebacker]

Yes, sounds like we missed some updates for JB v 0.10.0...

I wonder if it will work in version 0.10.0 if we add another reference line to the citations.bib at the bottom of the _config.yml file.

I think trying this would be good. I'd like to avoid anymore hard pins. They can cause conflicts with other dependencies.

[rickecon]

I changed the environment.yml to be jupyter-book>=0.9.1 in PR #683. However, I still can't get the Citations page to compile under the Jupyter Book 0.10.1 or 0.10.0 versions. I tried the following two fixes with v. 0.10.1 in the _config.yml file, neither of which worked:

Fix attempt 1

bibtex_bibfiles:
   - OGUSA_references.bib
   - ../../citations.bib

Fix attempt 2, copying citations.md to same directory as OGUSA_references.bib

bibtex_bibfiles:
   - OGUSA_references.bib
   - citations.bib

I think we should move forward with this PR #683 as it is, and work to fix the citations page in a future PR, leaving this issue thread open until fixed.

[rickecon]

I have verified that our previous approach compiles the References and Citations pages and all in-text references correctly for Jupyter Book v. 0.9.1, but it breaks for anything higher than v. 0.10.0. The Jupyter Book changelog for version 0.10.0 says:

MyST Parser version 0.13.x

: The MyST-NB and MyST-parser have both been upgraded. This comes with support for new syntax and a new configuration mechanism (see below for some examples).

(see also the myst-parser changelog for more information about the syntax additions)

New myst extension configuration

: The myst_extended_syntax configuration is now deprecated, in lieu of a more flexible extension mechanism. You may now enable individual myst extensions by adding them to the following section of your _config.yml file:

parse:
  myst_enable_extensions:
    - <list-of-extensions>

See the MyST syntax extension section here for more information.

Citations and references configuration

: This version comes with a version bump to sphinxcontrib.bibtex v2.1.*. This introduces new configuration for connecting your bibfiles (no longer using the bibliography directive), and makes the citation resolution process much more stable and dependable. : See for more information, and the sphinxcontrib.bibtex documentation for more information about updates in the latest version.

My guess is that our previous approach of putting the .bib file path at the top of the citations.md file has been deprecated for this new approach of listing the .bib files in the _config.yml as we currently do in PR #683. The drawback is that I cannot find any example of listing two .bib files there. The sphinxcontrib-bibtex documentation shows a list-like structure. So I might try that. But I am not sure how to get this second Citations references page working.

[rickecon]

I tried updating the _config.yml file in the following two ways, both resulting in errors.
Fix attempt 1

bibtex_bibfiles:
   - [OGUSA_references.bib, ../../citations.bib]

Fix attempt 2, copying citations.md to same directory as OGUSA_references.bib, also adjusting reference in citations.md

bibtex_bibfiles:
   - [OGUSA_references.bib, citations.bib]

[rickecon]

@jdebacker . I don't know what I did, but the Jupyter Book v. 0.10.1 is now compiling the Citations page, with citations.bib not in the _config.yml file and located in the main OG-USA/ directory, and with no changes to citations.md. All changes have been incorporated into PR #683. I am closing this issue.