webr::install() breaks on old versions of webR if packages are updated

[georgestagg]

Probably this can be fixed by updating webr::install() so that utils::available.packages() is used with a contriburl pointing to the binary directory. That way the package list returned by available.packages() is specific to the version of R that's running.

See r-wasm/webr-repo#29.

[coatless]

@georgestagg I'm not sure what the repo structure being used by https://repo.r-wasm.org/. But, shouldn't the binaries being generated already be versioned like:

- www (e.g. /)
  |- bin
     |- macosx
        |- contrib
           |- 4.1
              |- package_1.0.0.tgz
              |- PACKAGES
              |- PACKAGES.gz
           |- 4.3
              |- package_1.0.0.tgz
              |- PACKAGES
              |- PACKAGES.gz
  |- src
     |- contrib
        |- package_1.0.0.tar.gz
        |- PACKAGES
        |- PACKAGES.gz

where the PACKAGES.gz/PACKAGES would be obtained via:

tools::write_PACKAGES("~/path/to/bin/macosx/contrib/4.1")

[georgestagg]

That's right. We're using that structure, though we use the OS name emscripten in place of macosx.

The issue had arisen because our webr::install() function was looking up PACKAGES.gz in the source directory /src/contrib to enumerate available packages and their versions, but the package sources only happen to correspond with the binary versions that are in /bin/emscripten/contrib/4.3.

Binary packages for R 4.1 still exist in /bin/emscripten/contrib/4.1, but they are not updated in the same way due to changes in our version of Emscripten and the compiler features we're using since release of webR v0.1.1. So, right now the packages under /bin/emscripten/contrib/4.1 are frozen and older versions, and have been compiled in a different way specifically to work with webR v0.1.1.

A change has now been made to tweak webr::install() so that it explicitly looks at /bin/emscripten/contrib/[version]/PACKAGES.gz to find package names and versions, rather than in /src/contrib. That should hopefully avoid the same problem happening in the future when later versions of webR are released using newer versions of R or Emscripten.

However, webR v0.1.0/v0.1.1 will always look in the wrong place. I will keep an eye on it in the short term, but eventually these versions of webR will break again as new packages are released and the package sources in /src/contrib are updated. I am not too worried about this, though, because I will recommend users and developers upgrade past v0.1.1 to the soon released v0.2.0 (and beyond) in any case, for other technical reasons.

There might be ways to tweak the repo CDN to detect the issue and serve a different /src/contrib/PACKAGES.gz to older webR versions, but I will need to think about it a bit more to be sure.

[coatless]

Thanks for the detailed response! I think one additional question I would have here is should the [version] component be more along the lines of [r-version]-[webr-version], e.g. 4.1-0.1.0? This would potentially allow for changes if the Emscripten and/or the compiler shift between releases. (The downside is this increases the number of R package binaries.)

I wouldn't worry too much about the redirect at the moment. I think a huge surge of interest will come after posit::conf(2023). Do you know if you're aiming to release v0.2.0 before then?

[georgestagg]

should the [version] component be more along the lines of [r-version]-[webr-version], e.g. 4.1-0.1.0?

Perhaps... Though I'd rather not stray too far from the usual CRAN structure for the moment. My current plan is to avoid updating the version of Emscripten that we're using to build the public distributions of webR and its packages between major versions of R, so as to avoid that issue. In any case, from experimentation it seems like small changes to Emscripten do not always cause issues, the Emscripten team just don't guarantee the ABI compatibility for sure.

Do you know if you're aiming to release v0.2.0 before then?

Yes, it should be very soon. v0.1.1 should continue to work if there is no time for you to upgrade and test quarto-webr for 0.2.0 before posit::conf. (At the very least, handling plotting messages will need to be reworked, let me know if you need help after release).