using non distro PPA

[tim-salabim]

Hi, to continue our discussion at https://fosstodon.org/@defuneste/113952229309036946 started by @olivroy and as especially alluded to by @florisvdh I am interested if, and how we can link system libraries from ubuntugis-unstable for the spatial packages?

As mentioned in the fediverse thread, the system libraries provided by ubuntugis-unstable are usually very stable (as they are not bleeding edge). I've used and advised people to use these rather than the default distro ones for years never had any issues. sf also recommends them as well as terra, apart from the easy way using r2u :-)

@florisvdh if I understood your comment correctly, you do use r2u with non-distro PPA? If so, I'd be very interested in hearing/seeing how...

As a side comment, I checked the docker image today for 22.04 and the system libraries are rather old. sf reports

sf_extSoftVersion()
          GEOS           GDAL         proj.4 GDAL_with_GEOS     USE_PROJ_H 
      "3.10.2"        "3.4.1"        "8.2.1"         "true"         "true" 
          PROJ 
       "8.2.1" 

[eddelbuettel]

You would do it pretty much the same way r2u itself 'overlays' on top of the apt repos you see in /etc/apt/sources.list.d/ namely ubuntu and cran (for the 'ports' of r-base etc) and of course r2u itself. Note in particular how r2u uses 'priority pinning' to get a preference in case of packages with the same name and version (rare). It would be pretty trivial to add a PPA at launchpad, say, if you have one.

I can walk you and others through this, but truth be told this is 100% an apt/dpkg issue so there may already be good resources out there. And in fact, this is what I would recommend to the (ubuntu-based) R spatial community: create such an overlay. (Which may likely entail rebuilds against the newer libraries too as that is the point. Maybe you already do this in Ubuntu-GIS PPA -- I would not know as I am not really a 'spatial' R user.)

Now, two nits if I may. When I suggested to pivot from social media I believe I pointed at issues and not discussions. Can / should this be moved?

Second, one of the reasons I suggested to move is that these scattered social media posts make it too easy to overlook details as eg here where I recommended to you to stick with 'noble' aka 24.04 for that very reason.

> sf::sf_extSoftVersion()
          GEOS           GDAL         proj.4 GDAL_with_GEOS     USE_PROJ_H 
      "3.12.1"        "3.8.4"        "9.4.0"         "true"         "true" 
          PROJ 
       "9.4.0" 
> 

And for what it is worth I actually just mucked with a Dockerfile accessing a launchpad PPA this very morning...

[eddelbuettel]

So in that 24.04 container I had fired up to show the sf_extSoftVersions() output I just did

apt install software-properties-common     # to support `add-apt-repository`
add-apt-repository ppa:ubuntugis/ubuntugis-unstable

and now the subsequent apt update shows what you desire:

[...]
Hit:1 http://security.ubuntu.com/ubuntu noble-security InRelease
Hit:2 https://r2u.stat.illinois.edu/ubuntu noble InRelease                
Hit:3 http://archive.ubuntu.com/ubuntu noble InRelease                
Hit:4 http://archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu noble-backports InRelease
Get:6 https://ppa.launchpadcontent.net/ubuntugis/ubuntugis-unstable/ubuntu noble InRelease [24.4 kB]
Hit:7 https://cloud.r-project.org/bin/linux/ubuntu noble-cran40/ InRelease              
Get:8 https://ppa.launchpadcontent.net/ubuntugis/ubuntugis-unstable/ubuntu noble/main amd64 Packages [57.7 kB]
Fetched 82.0 kB in 1s (90.0 kB/s)   
Reading package lists... Done
root@f3461d286cdd:/# 

So one simple way would just be to cook your own Dockerfile where these commands run.

[florisvdh]

My (local) approach is to take advantage of having both the r2u repository and the ubuntugis-unstable PPA active for dpkg / apt, and using the R library path priority to use a locally compiled R package if available.

So I install most R packages using apt, which uses the r2u repository (has priority), but for some packages I use install.packages() in R (I didn't link it to r2u) to build and install specific packages in a user-defined library path.

So, using a Ubuntu 22.04 base (in Linux Mint 21.3) I have:

$ apt policy libgdal-dev 
libgdal-dev:
  Installed: 3.8.4+dfsg-1~jammy0
  Candidate: 3.8.4+dfsg-1~jammy0
  Version table:
 *** 3.8.4+dfsg-1~jammy0 500
        500 http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status
     3.4.1+dfsg-1build4 500
        500 http://ftp.belnet.be/ubuntu jammy/universe amd64 Packages

In R:

> .libPaths()
[1] "/home/floris/lib/R/library"    "/usr/local/lib/R/site-library"
[3] "/usr/lib/R/site-library"       "/usr/lib/R/library"           
> miscfv::duplicated_packages()
# A tibble: 8 × 3
  package    `/home/floris/lib/R/library` `/usr/lib/R/site-library`
  <chr>      <pckg_vrs>                   <pckg_vrs>               
1 fmesher    0.1.5                        0.2.0                    
2 lwgeom     0.2.14                       0.2.14                   
3 PROJ       0.5.0                        0.5.0                    
4 ragg       1.3.3                        1.3.3                    
5 renderthis 0.2.1                        0.2.0                    
6 sf         1.0.19                       1.0.19                   
7 svglite    2.1.3                        2.1.3                    
8 terra      1.8.15                       1.8.15                   
> sf::sf_extSoftVersion() # locally compiled package
          GEOS           GDAL         proj.4 GDAL_with_GEOS     USE_PROJ_H 
      "3.12.1"        "3.8.4"        "9.3.1"         "true"         "true" 
          PROJ 
       "9.3.1" 

When loading the r2u sf package in a new R session, it is seen that it needs the older GDAL version, which is not available. But, this package is normally not used, as shown above. But having it installed with r2u is still convenient to spot when CRAN updates are coming in!

> library(sf, lib.loc = "/usr/lib/R/site-library")
Linking to GEOS 3.12.1, GDAL 3.4.3, PROJ 8.2.1; sf_use_s2() is TRUE
WARNING: different compile-time and runtime versions for GEOS found:
Linked against: 3.12.1-CAPI-1.18.1 compiled against: 3.10.2-CAPI-1.16.0
It is probably a good idea to reinstall sf (and maybe lwgeom too)

[eddelbuettel]

Yes of course to all of that, @florisvdh:

1. You describe precisely what I recommend. Going from $N$ to $N+1$ known repos by adding ubuntugis-unstable via its PPA. For the apt operations apt pinning can influence search.

2. For R that is of course what we all used for decades mixing some r-cran-* binaries from the distro with potentially additional ones from c2d4u or other binary repos and then the local paths in /usr/local/lib/R/site-library (and possibly also below $HOME.

That works.

Now, whenever you add 'other binaries' by non-apt means you may get library dependencies that apt does not know. That is why p3m.dev has to recommend 'also run these apt commands': it is not integrated. And equally if you delete (system) libraries your mayve leave the local builds in /usr/local/lib/R/site-library stranded. We all have seen that with geos/proj, or gsl, or icu. Handling more packages via apt is one solid way to avoid it.

(Also for the case of multiple directories in .libPaths() you can also 'shadown' a newer package with an older one: R uses 'first match in .libpaths() directory order, not highest version. The bspm package has a helper, I also added one to my catch-all package dang.)

[florisvdh]

Thanks for the clear overview @eddelbuettel, and for the historical perspective.

Indeed the library path order determines which package version is taken. Thanks for the pointers to {dang} and {bspm}; cool helpers indeed!

> bspm::shadowed_packages()
              Package                 LibPath Version
fmesher       fmesher /usr/lib/R/site-library   0.2.0
lwgeom         lwgeom /usr/lib/R/site-library  0.2.14
PROJ             PROJ /usr/lib/R/site-library   0.5.0
ragg             ragg /usr/lib/R/site-library   1.3.3
renderthis renderthis /usr/lib/R/site-library   0.2.0
sf                 sf /usr/lib/R/site-library  1.0.19
svglite       svglite /usr/lib/R/site-library   2.1.3
terra           terra /usr/lib/R/site-library  1.8.15
                       Shadow.LibPath Shadow.Version Shadow.Newer
fmesher    /home/floris/lib/R/library          0.1.5        FALSE
lwgeom     /home/floris/lib/R/library         0.2.14        FALSE
PROJ       /home/floris/lib/R/library          0.5.0        FALSE
ragg       /home/floris/lib/R/library          1.3.3        FALSE
renderthis /home/floris/lib/R/library          0.2.1         TRUE
sf         /home/floris/lib/R/library         1.0.19        FALSE
svglite    /home/floris/lib/R/library          2.1.3        FALSE
terra      /home/floris/lib/R/library         1.8.15        FALSE

> dang::shadowedPackages()
Key: <Package>
       Package                    LibPath Version     n   good
        <char>                     <char>  <char> <int> <lgcl>
 1:       PROJ /home/floris/lib/R/library   0.5.0     2   TRUE
 2:       PROJ    /usr/lib/R/site-library   0.5.0     2   TRUE
 3:    fmesher /home/floris/lib/R/library   0.1.5     2  FALSE
 4:    fmesher    /usr/lib/R/site-library   0.2.0     2   TRUE
 5:     lwgeom /home/floris/lib/R/library  0.2.14     2   TRUE
 6:     lwgeom    /usr/lib/R/site-library  0.2.14     2   TRUE
 7:       ragg /home/floris/lib/R/library   1.3.3     2   TRUE
 8:       ragg    /usr/lib/R/site-library   1.3.3     2   TRUE
 9: renderthis /home/floris/lib/R/library   0.2.1     2   TRUE
10: renderthis    /usr/lib/R/site-library   0.2.0     2  FALSE
11:         sf /home/floris/lib/R/library  1.0.19     2   TRUE
12:         sf    /usr/lib/R/site-library  1.0.19     2   TRUE
13:    svglite /home/floris/lib/R/library   2.1.3     2   TRUE
14:    svglite    /usr/lib/R/site-library   2.1.3     2   TRUE
15:      terra /home/floris/lib/R/library  1.8.15     2   TRUE
16:      terra    /usr/lib/R/site-library  1.8.15     2   TRUE

[eddelbuettel]

Yes bspm actually drives r2u by converting install.packages() to apt install .... (and does the same for Inaki's own similar cran2copr.)

[eddelbuettel]

@tim-salabim Does this answer your question? If not, could you possibly provide a little (self-contained) example script we could use for an illustration? That should most easily be addressable in a Docker container but serve as an example for a standard installation, or GitHub Actions use, or WSL2, or whereever else one may run Ubuntu.

[tim-salabim]

Yeah, think this does give me enough information to leverage r2u for the spatial stack. Thanks for all the input and a very insightful discussion.

[eddelbuettel]

Sounds good -- it is common to layer. For example BioConductor has a similar 'bioc2u' toolchain built on top of r2u for their Docker use. If you run into any "surprises" or "issues" you know which door to knock on.