[R] Installing arrow R package on Linux with install-arrow.R fails

[mark-andrews]

Describe the bug, including details regarding any error messages, version, and platform.

I am trying to install the arrow R package on Linux by running the following script (as suggested here):

source("https://raw.githubusercontent.com/apache/arrow/main/r/R/install-arrow.R")
create_package_with_all_dependencies("my_arrow_pkg.tar.gz")
install.packages(
  "my_arrow_pkg.tar.gz",
  dependencies = c("Depends", "Imports", "LinkingTo")
 )

It fails with the following message:

Error: Error installing package 'arrow':
=================================

* installing *source* package ‘arrow’ ...
** package ‘arrow’ successfully unpacked and MD5 sums checked
** using staged installation
*** pkg-config found.
*** Found local C++ source: 'tools/cpp'
*** Building libarrow from source
    For build options and troubleshooting, see the install guide:
    https://arrow.apache.org/docs/r/articles/install.html
**** cmake 3.31.1: /usr/bin/cmake
tools/download_dependencies_R.sh: line 59: source: /tmp/RtmpQzm78H/renv-package-new-23c906aa6b4b/arrow/tools: is a directory
Error in get_component_names() : Failed to run download_dependencies_R.sh
Calls: build_libarrow -> set_thirdparty_urls -> get_component_names
Execution halted
------------------------- NOTE ---------------------------
There was an issue building the Arrow C++ libraries.
See https://arrow.apache.org/docs/r/articles/install.html
---------------------------------------------------------
ERROR: configuration failed for package ‘arrow’
* removing ‘/home/andrews/tmp/inst_arrow/renv/staging/1/arrow’
install of package 'arrow' failed [error code 1]
Traceback (most recent calls last):
15: install.packages("my_arrow_pkg.tar.gz", dependencies = c("Depends", 
        "Imports", "LinkingTo"))
14: eval(call, envir = parent.frame())
13: eval(call, envir = parent.frame())
12: renv::install("my_arrow_pkg.tar.gz", dependencies = c("Depends", 
        "Imports", "LinkingTo"))
11: renv_install_impl(records)
10: renv_install_staged(records)
 9: renv_install_default(records)
 8: handler(package, renv_install_package(record))
 7: renv_install_package(record)
 6: withCallingHandlers(renv_install_package_impl(record), error = function(e) writef("FAILED"))
 5: renv_install_package_impl(record)
 4: r_cmd_install(package, path)
 3: r_exec_error(package, output, "install", status)
 2: abort(all)
 1: stop(fallback)
Execution halted

I can't understand what the problem is because I ran the same script without errors on an seemingly identical Linux system.

My sessionInfo() is below.

> sessionInfo()
R version 4.4.2 (2024-10-31)
Platform: x86_64-pc-linux-gnu
Running under: Arch Linux

Matrix products: default
BLAS:   /usr/lib/libblas.so.3.12.0 
LAPACK: /usr/lib/liblapack.so.3.12.0

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
 [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
 [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
[10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

time zone: Europe/London
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.4.2 tools_4.4.2    renv_1.0.11   
> 

Component(s)

R

[assignUser]

Do you have normal bash installed? Or are you using an alternative shell that masquerades as bash and interprets the offending line differently?

SOURCE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# snip
source ${SOURCE_DIR}/cpp/thirdparty/versions.txt

tools/download_dependencies_R.sh: line 59: source: /tmp/RtmpQzm78H/renv-package-new-23c906aa6b4b/arrow/tools: is a director

[mark-andrews]

I do have normal bash.

> bash --version                                                                                                                                                                                                                           
GNU bash, version 5.2.37(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

[mark-andrews]

In case it is helpful, I just tried doing this all in a docker container running R and it failed at the create_package_with_all_dependencies step (previously I was able to complete that step). Again, it is a bash error.

> docker run -it rocker/r-ver                                                                                                                                                                                                              

Unable to find image 'rocker/r-ver:latest' locally
latest: Pulling from rocker/r-ver
ff65ddf9395b: Pull complete 
16d44ce48f37: Pull complete 
199f37b8fbcd: Pull complete 
8f709b883eba: Pull complete 
0068d1642dbd: Pull complete 
5245faf9270d: Pull complete 
8726ee55941a: Pull complete 
Digest: sha256:772738e2bee18cdc77464e4636b685dfec8e0c446b1cb9e28417be2ed5a288f8
Status: Downloaded newer image for rocker/r-ver:latest

R version 4.4.2 (2024-10-31) -- "Pile of Leaves"
Copyright (C) 2024 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> source("https://raw.githubusercontent.com/apache/arrow/main/r/R/install-arrow.R")
> create_package_with_all_dependencies("my_arrow_pkg.tar.gz")
Downloading Arrow source file
trying URL 'https://p3m.dev/cran/__linux__/noble/latest/src/contrib/arrow_17.0.0.1.tar.gz'
Content type 'binary/octet-stream' length 20389884 bytes (19.4 MB)
==================================================
downloaded 19.4 MB

Error in create_package_with_all_dependencies("my_arrow_pkg.tar.gz") : 
  Failed to parse versions.txt; view  /tmp/RtmpqoUrWB/file119f0ecbb.R for more information
In addition: Warning messages:
1: In dir.create(download_dir) :
  cannot create dir '/tmp/RtmpqoUrWB/file16505b133/arrow/tools/thirdparty_dependencies', reason 'No such file or directory'
2: In system2("bash", c(download_dependencies_sh, download_dir), stdout = download_script,  :
  error in running command

> file.info('/tmp/RtmpqoUrWB/file119f0ecbb.R')$size
[1] 0

[amoeba]

Hi @mark-andrews, running the same steps as the above works fine for me:

$ docker run -it rocker/r-ver
> source("https://raw.githubusercontent.com/apache/arrow/main/r/R/install-arrow.R")
> create_package_with_all_dependencies("my_arrow_pkg.tar.gz")
Downloading Arrow source file
trying URL 'https://p3m.dev/cran/latest/src/contrib/arrow_18.1.0.tar.gz'
Content type 'binary/octet-stream' length 4548708 bytes (4.3 MB)
==================================================
downloaded 4.3 MB

Repacking tar.gz file to /tmp/RtmpwP4ypv/my_arrow_pkg.tar.gz

A difference I see is the version of arrow it grabbed is newer for me. Can you try again and let us know if that works better?

[mark-andrews]

Hi @amoeba
Interestingly, it does not work for me still, although I do get version 18 of arrow now.

> docker run -it rocker/r-ver

R version 4.4.2 (2024-10-31) -- "Pile of Leaves"
Copyright (C) 2024 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> source("https://raw.githubusercontent.com/apache/arrow/main/r/R/install-arrow.R")
> create_package_with_all_dependencies("my_arrow_pkg.tar.gz")
Downloading Arrow source file
trying URL 'https://p3m.dev/cran/__linux__/noble/latest/src/contrib/arrow_18.1.0.tar.gz'
Content type 'binary/octet-stream' length 20598620 bytes (19.6 MB)
==================================================
downloaded 19.6 MB

Error in create_package_with_all_dependencies("my_arrow_pkg.tar.gz") :
  Failed to parse versions.txt; view  /tmp/RtmpZ1474S/file1316604c3.R for more information
In addition: Warning messages:
1: In dir.create(download_dir) :
  cannot create dir '/tmp/RtmpZ1474S/file12f9c8c66/arrow/tools/thirdparty_dependencies', reason 'No such file or directory'
2: In system2("bash", c(download_dependencies_sh, download_dir), stdout = download_script,  :
  error in running command

I even purged the r-ver image and pulled it down again, and tried again but that led to the same result.
I am pretty baffled by way something that works in one docker container does not work in another container.

[amoeba]

I switched to a real Linux machine and I can reproduce your issue. I'm looking at it now and will update here.