Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

exec: line 1: /opt/docker/jre/bin/java: not found when trying to use docker image build with jlink #1506

Open
Christewart opened this issue May 23, 2022 · 25 comments

Comments

@Christewart
Copy link

Christewart commented May 23, 2022

Expected behaviour

Generate a docker image with the stripped down jre built by jlink. This is useful as slim images can be used as the base image such as alpine, rather than large jdk docker images.

Actual behaviour

When building the docker image with the jlink plugin, I get this error

Screenshot from 2022-05-23 09-03-49

Information

  • What sbt-native-packager are you using
    1.9.9

  • What sbt version
    1.6.2

  • What is your build system (e.g. Ubuntu, MacOS, Windows, Debian )
    linux

  • What package are you building (e.g. docker, rpm, ...)
    docker image with jlink

As a side note, #1437 would be very useful to have for this build too.

Here is my branch where I am building this from. You can build the image locally with sbt appServer/docker:publishLocal

https://github.com/Christewart/bitcoin-s-core/tree/2022-05-22-alpine-base-image

@guizmaii
Copy link
Contributor

guizmaii commented May 23, 2022

I was having the same kind of issue, maybe this would help you: #1449 (comment)

@Christewart
Copy link
Author

I was having the same kind of issue, maybe this would help you: #1449 (comment)

In my case I am building on linux and running on linux 🤔 . The java version is

openjdk version "18" 2022-03-22
OpenJDK Runtime Environment (build 18+36-2087)
OpenJDK 64-Bit Server VM (build 18+36-2087, mixed mode, sharing)

although i don't think that should matter?

@Christewart
Copy link
Author

Looking closer, it does seem like my error is fundamentally different:

here is the text version rather than the image i linked above

 docker run  -p 9999:9999 -p 19999:19999 -e BITCOIN_S_SERVER_RPC_PASSWORD='topsecret' bitcoinscala/bitcoin-s-server:latest
/opt/docker/bin/bitcoin-s-server: line 106: /opt/docker/lib/org.bitcoin-s.bitcoin-s-server-1.9.1-66-a9997d5d-20220523-0843-SNAPSHOT-launcher.jar: Permission denied
/opt/docker/bin/bitcoin-s-server: exec: line 1: /opt/docker/jre/bin/java: not found

@guizmaii
Copy link
Contributor

@Christewart Can you maybe give us the content of your oracleServer/target/docker/stage/Dockerfile file, please?

@Christewart
Copy link
Author

@Christewart Can you maybe give us the content of your oracleServer/target/docker/stage/Dockerfile file, please?

FROM alpine:latest as stage0
LABEL snp-multi-stage="intermediate"
LABEL snp-multi-stage-id="68d23a07-a1ab-4ea2-8eb4-1ea041c09d56"
WORKDIR /opt/docker
COPY 2/opt /2/opt
COPY 3/opt /3/opt
COPY 4/opt /4/opt
COPY opt /opt
USER root
RUN ["chmod", "-R", "u=rX,g=rX", "/2/opt/docker"]
RUN ["chmod", "-R", "u=rX,g=rX", "/3/opt/docker"]
RUN ["chmod", "-R", "u=rX,g=rX", "/4/opt/docker"]
RUN ["chmod", "-R", "u=rX,g=rX", "/opt/docker"]
RUN ["chmod", "u+x,g+x", "/opt/docker/wallet-server-extra-startup-script.sh"]
RUN ["chmod", "u+x,g+x", "/4/opt/docker/bin/bitcoin-s-server"]
RUN ["chmod", "u+x,g+x", "/4/opt/docker/bin/bitcoin-s-server-main"]
RUN ["apk", "add", "--no-cache", "bash"]

FROM alpine:latest as mainstage
LABEL MAINTAINER="Chris Stewart <[email protected]>"
USER root
RUN id -u bitcoin-s 1>/dev/null 2>&1 || (( getent group 0 1>/dev/null 2>&1 || ( type groupadd 1>/dev/null 2>&1 && groupadd -g 0 root || addgroup -g 0 -S root )) && ( type useradd 1>/dev/null 2>&1 && useradd --system --create-home --uid 1000 --gid 0 bitcoin-s || adduser -S -u 1000 -G root bitcoin-s ))
WORKDIR /opt/docker
COPY --from=stage0 --chown=bitcoin-s:root /2/opt/docker /opt/docker
COPY --from=stage0 --chown=bitcoin-s:root /3/opt/docker /opt/docker
COPY --from=stage0 --chown=bitcoin-s:root /4/opt/docker /opt/docker
COPY --from=stage0 --chown=bitcoin-s:root /opt/docker /opt/docker
EXPOSE 9999 19999
USER 1000:0
ENTRYPOINT ["/opt/docker/bin/bitcoin-s-server"]
CMD ["--conf", "/opt/docker/docker-application.conf"]

@guizmaii
Copy link
Contributor

guizmaii commented May 23, 2022

The JVM is living in /3/opt/docker and seems to be correctly copied 🤔
Do you see these files in oracleServer/target/docker/stage/3:

Screen Shot 2022-05-23 at 5 49 06 pm

Can you copy the content of /opt/docker/bin/bitcoin-s-server maybe, please?

@Christewart
Copy link
Author

Christewart commented May 23, 2022

Do you see these files in oracleServer/target/docker/stage/3:

Here is tree 3

3
└── opt
    └── docker
        └── jre
            ├── bin
            │   ├── java
            │   └── keytool
            ├── conf
            │   ├── logging.properties
            │   ├── net.properties
            │   ├── sdp
            │   │   └── sdp.conf.template
            │   └── security
            │       ├── java.policy
            │       ├── java.security
            │       └── policy
            │           ├── limited
            │           │   ├── default_local.policy
            │           │   ├── default_US_export.policy
            │           │   └── exempt_local.policy
            │           ├── README.txt
            │           └── unlimited
            │               ├── default_local.policy
            │               └── default_US_export.policy
            ├── legal
            │   ├── java.base
            │   │   ├── ADDITIONAL_LICENSE_INFO
            │   │   ├── aes.md
            │   │   ├── asm.md
            │   │   ├── ASSEMBLY_EXCEPTION
            │   │   ├── cldr.md
            │   │   ├── c-libutl.md
            │   │   ├── icu.md
            │   │   ├── LICENSE
            │   │   ├── public_suffix.md
            │   │   └── unicode.md
            │   ├── java.datatransfer
            │   │   ├── ADDITIONAL_LICENSE_INFO
            │   │   ├── ASSEMBLY_EXCEPTION
            │   │   └── LICENSE
            │   ├── java.logging
            │   │   ├── ADDITIONAL_LICENSE_INFO
            │   │   ├── ASSEMBLY_EXCEPTION
            │   │   └── LICENSE
            │   ├── java.management
            │   │   ├── ADDITIONAL_LICENSE_INFO
            │   │   ├── ASSEMBLY_EXCEPTION
            │   │   └── LICENSE
            │   ├── java.naming
            │   │   ├── ADDITIONAL_LICENSE_INFO
            │   │   ├── ASSEMBLY_EXCEPTION
            │   │   └── LICENSE
            │   ├── java.security.jgss
            │   │   ├── ADDITIONAL_LICENSE_INFO
            │   │   ├── ASSEMBLY_EXCEPTION
            │   │   └── LICENSE
            │   ├── java.security.sasl
            │   │   ├── ADDITIONAL_LICENSE_INFO
            │   │   ├── ASSEMBLY_EXCEPTION
            │   │   └── LICENSE
            │   ├── java.sql
            │   │   ├── ADDITIONAL_LICENSE_INFO
            │   │   ├── ASSEMBLY_EXCEPTION
            │   │   └── LICENSE
            │   ├── java.transaction.xa
            │   │   ├── ADDITIONAL_LICENSE_INFO
            │   │   ├── ASSEMBLY_EXCEPTION
            │   │   └── LICENSE
            │   ├── java.xml
            │   │   ├── ADDITIONAL_LICENSE_INFO
            │   │   ├── ASSEMBLY_EXCEPTION
            │   │   ├── bcel.md
            │   │   ├── dom.md
            │   │   ├── jcup.md
            │   │   ├── LICENSE
            │   │   ├── xalan.md
            │   │   └── xerces.md
            │   ├── jdk.crypto.ec
            │   │   ├── ADDITIONAL_LICENSE_INFO
            │   │   ├── ASSEMBLY_EXCEPTION
            │   │   └── LICENSE
            │   ├── jdk.management
            │   │   ├── ADDITIONAL_LICENSE_INFO
            │   │   ├── ASSEMBLY_EXCEPTION
            │   │   └── LICENSE
            │   └── jdk.unsupported
            │       ├── ADDITIONAL_LICENSE_INFO
            │       ├── ASSEMBLY_EXCEPTION
            │       └── LICENSE
            ├── lib
            │   ├── classlist
            │   ├── jexec
            │   ├── jrt-fs.jar
            │   ├── jspawnhelper
            │   ├── jvm.cfg
            │   ├── libj2gss.so
            │   ├── libjava.so
            │   ├── libjimage.so
            │   ├── libjli.so
            │   ├── libjsig.so
            │   ├── libmanagement_ext.so
            │   ├── libmanagement.so
            │   ├── libnet.so
            │   ├── libnio.so
            │   ├── libverify.so
            │   ├── libzip.so
            │   ├── modules
            │   ├── security
            │   │   ├── blocked.certs
            │   │   ├── cacerts
            │   │   ├── default.policy
            │   │   └── public_suffix_list.dat
            │   ├── server
            │   │   ├── libjsig.so
            │   │   └── libjvm.so
            │   └── tzdb.dat
            └── release

27 directories, 89 files

@Christewart
Copy link
Author

Can you copy the content of /opt/docker/bin/bitcoin-s-server maybe, please?

I don't understand where to find this in the staged directory, or else maybe it doesn't exist?

@guizmaii
Copy link
Contributor

guizmaii commented May 23, 2022

I don't understand where to find this in the staged directory, or else maybe it doesn't exist?

It's a bash script that you can find in this directory: oracleServer/target/docker/stage/4/opt/docker/bin

@Christewart
Copy link
Author

#!/bin/sh

realpath () {
(
  TARGET_FILE="$1"

  cd "$(dirname "$TARGET_FILE")"
  TARGET_FILE=$(basename "$TARGET_FILE")

  COUNT=0
  while [ -L "$TARGET_FILE" -a $COUNT -lt 100 ]
  do
      TARGET_FILE=$(readlink "$TARGET_FILE")
      cd "$(dirname "$TARGET_FILE")"
      TARGET_FILE=$(basename "$TARGET_FILE")
      COUNT=$(($COUNT + 1))
  done

  if [ "$TARGET_FILE" = "." -o "$TARGET_FILE" = ".." ]; then
    cd "$TARGET_FILE"
  fi
  TARGET_DIR="$(pwd -P)"
  if [ "$TARGET_DIR" = "/" ]; then
    TARGET_FILE="/$TARGET_FILE"
  else
    TARGET_FILE="$TARGET_DIR/$TARGET_FILE"
  fi
  echo "$TARGET_FILE"
)
}

# Allow user and template_declares (see below) to add java options.
addJava () {
  java_opts="$java_opts $1"
}

addApp () {
  app_commands="$app_commands $1"
}

addResidual () {
  residual_args="$residual_args \"$1\""
}

# Allow user to specify java options. These get listed first per bash-template.
if [ -n "$JAVA_OPTS" ]
then
  addJava "$JAVA_OPTS"
fi

# Loads a configuration file full of default command line options for this script.
loadConfigFile() {
  cat "$1" | sed '/^\#/d;s/\r$//' | sed 's/^-J-X/-X/' | tr '\r\n' ' '
}

# Detect which JVM we should use.
get_java_cmd() {
  # High-priority override for Jlink images
  if [ -n "$bundled_jvm" ];  then
    echo "$bundled_jvm/bin/java"
  elif [ -n "$JAVA_HOME" ] && [ -x "$JAVA_HOME/bin/java" ];  then
    echo "$JAVA_HOME/bin/java"
  else
    echo "java"
  fi
}

# Processes incoming arguments and places them in appropriate global variables.  called by the run method.
process_args () {
  local no_more_snp_opts=0
  while [ $# -gt 0 ]; do
    case "$1" in
             --) shift && no_more_snp_opts=1 && break ;;
       -h|-help) usage; exit 1 ;;
    -v|-verbose) verbose=1 && shift ;;
      -d|-debug) debug=1 && shift ;;

    -no-version-check) no_version_check=1 && shift ;;

           -mem) echo "!! WARNING !! -mem option is ignored. Please use -J-Xmx and -J-Xms" && shift 2 ;;
     -jvm-debug) require_arg port "$1" "$2" && addDebugger $2 && shift 2 ;;

          -main) custom_mainclass="$2" && shift 2 ;;

     -java-home) require_arg path "$1" "$2" && jre=`eval echo $2` && java_cmd="$jre/bin/java" && shift 2 ;;

 -D*|-agentlib*|-XX*) addJava "$1" && shift ;;
                 -J*) addJava "${1:2}" && shift ;;
                   *) addResidual "$1" && shift ;;
    esac
  done

  if [ $no_more_snp_opts ]; then
    while [ $# -gt 0 ]; do
      addResidual "$1" && shift
    done
  fi
}

app_commands=""
residual_args=""
real_script_path="$(realpath "$0")"
app_home="$(realpath "$(dirname "$real_script_path")")"
lib_dir="$(realpath "${app_home}/../lib")"

app_mainclass=-jar "$lib_dir/org.bitcoin-s.bitcoin-s-server-1.9.1-67-55506f28-SNAPSHOT-launcher.jar"

script_conf_file="${app_home}/../conf/application.ini"
app_classpath=""

bundled_jvm="$(realpath "${app_home}/../jre")"

if [[ "$OS" == "OSX" ]]; then
  #mac doesn't allow random binaries to be executable
  #remove the quarantine attribute so java is executable on mac
  xattr -d com.apple.quarantine jre/bin/java
fi

chmod +x jre/bin/java #make sure java is executable

process_args "$@"

java_cmd="$(get_java_cmd)"

# If a configuration file exist, read the contents to $opts
[ -f "$script_conf_file" ] && opts=$(loadConfigFile "$script_conf_file")

eval "exec $java_cmd $java_opts -classpath $app_classpath $opts $app_mainclass $app_commands $residual_args"

@guizmaii
Copy link
Contributor

This part looks weird to me:

if [[ "$OS" == "OSX" ]]; then
  #mac doesn't allow random binaries to be executable
  #remove the quarantine attribute so java is executable on mac
  xattr -d com.apple.quarantine jre/bin/java
fi

chmod +x jre/bin/java #make sure java is executable

It's not using the get_java_cmd function

@Christewart
Copy link
Author

This part looks weird to me:

if [[ "$OS" == "OSX" ]]; then
  #mac doesn't allow random binaries to be executable
  #remove the quarantine attribute so java is executable on mac
  xattr -d com.apple.quarantine jre/bin/java
fi

chmod +x jre/bin/java #make sure java is executable

It's not using the get_java_cmd function

That part is a custom bash script addition needed to get the desktop app working on mac.

This script

  1. Makes sure that jre/bin/java is executable (if I didn't do this it would not be executable on certain platforms when packaging with github workflows)
  2. On mac, remove the quarintine attribute for the java binary

Here is a link to the script.

https://github.com/bitcoin-s/bitcoin-s/blob/master/app/server/src/universal/wallet-server-extra-startup-script.sh

If I replace the references to java with the get_java_cmd this will work?

@guizmaii
Copy link
Contributor

the desktop app working on mac.

Are you not creating a Docker image? What is this a concern in the Docker world?

If I replace the references to java with the get_java_cmd this will work?

Well at least, you script is incorrect right now. The chmod +x is not chmoding the packaged/jlinked JRE

@Christewart
Copy link
Author

the desktop app working on mac.
Are you not creating a Docker image? What is this a concern in the Docker world?

We need to deliver the software in the form of docker images and desktop applications. The desktop applications require the usage of bash script additions due to the problems I detail above. Is there a way to omit bash script additions with the docker plugin?

@guizmaii
Copy link
Contributor

guizmaii commented May 23, 2022

We need to deliver the software in the form of docker images and desktop applications.

Make two "build modules": One with the constraints/configuration of your desktop app, the other one with constraints/configuration of a Docker image

lazy val myApp = ...

lazy val myDesktopApp = 
  project
    .enablePlugins(JavaAppPackaging, JlinkPlugin)
    .settings(jlinkSettings: _*)
    .settings(desktopSettings: _*)
    .settings(Compile / mainClass := Some("my.app.Main"))
    .dependsOn(myApp)

lazy val myDockerApp = 
  project
    .enablePlugins(JavaAppPackaging, JlinkPlugin, DockerPlugin)
    .settings(jlinkSettings: _*)
    .settings(dockerSettings: _*)
    .settings(Compile / mainClass := Some("my.app.Main"))
    .dependsOn(myApp)

@Christewart
Copy link
Author

Christewart commented May 23, 2022

I just removed the extra bash script stuff for now, I get the exact same error after removing it. Here is the new bash script after disabling the bash script additions.

The error

 docker run  -p 9999:9999 -p 19999:19999 -e BITCOIN_S_SERVER_RPC_PASSWORD='topsecret' bitcoinscala/bitcoin-s-server:latest
/opt/docker/bin/bitcoin-s-server: line 106: /opt/docker/lib/org.bitcoin-s.bitcoin-s-server-1.9.1-67-55506f28-20220523-1201-SNAPSHOT-launcher.jar: Permission denied
/opt/docker/bin/bitcoin-s-server: exec: line 1: /opt/docker/jre/bin/java: Permission denied

The new bash script without the extra additions.

#!/bin/sh

realpath () {
(
  TARGET_FILE="$1"

  cd "$(dirname "$TARGET_FILE")"
  TARGET_FILE=$(basename "$TARGET_FILE")

  COUNT=0
  while [ -L "$TARGET_FILE" -a $COUNT -lt 100 ]
  do
      TARGET_FILE=$(readlink "$TARGET_FILE")
      cd "$(dirname "$TARGET_FILE")"
      TARGET_FILE=$(basename "$TARGET_FILE")
      COUNT=$(($COUNT + 1))
  done

  if [ "$TARGET_FILE" = "." -o "$TARGET_FILE" = ".." ]; then
    cd "$TARGET_FILE"
  fi
  TARGET_DIR="$(pwd -P)"
  if [ "$TARGET_DIR" = "/" ]; then
    TARGET_FILE="/$TARGET_FILE"
  else
    TARGET_FILE="$TARGET_DIR/$TARGET_FILE"
  fi
  echo "$TARGET_FILE"
)
}

# Allow user and template_declares (see below) to add java options.
addJava () {
  java_opts="$java_opts $1"
}

addApp () {
  app_commands="$app_commands $1"
}

addResidual () {
  residual_args="$residual_args \"$1\""
}

# Allow user to specify java options. These get listed first per bash-template.
if [ -n "$JAVA_OPTS" ]
then
  addJava "$JAVA_OPTS"
fi

# Loads a configuration file full of default command line options for this script.
loadConfigFile() {
  cat "$1" | sed '/^\#/d;s/\r$//' | sed 's/^-J-X/-X/' | tr '\r\n' ' '
}

# Detect which JVM we should use.
get_java_cmd() {
  # High-priority override for Jlink images
  if [ -n "$bundled_jvm" ];  then
    echo "$bundled_jvm/bin/java"
  elif [ -n "$JAVA_HOME" ] && [ -x "$JAVA_HOME/bin/java" ];  then
    echo "$JAVA_HOME/bin/java"
  else
    echo "java"
  fi
}

# Processes incoming arguments and places them in appropriate global variables.  called by the run method.
process_args () {
  local no_more_snp_opts=0
  while [ $# -gt 0 ]; do
    case "$1" in
             --) shift && no_more_snp_opts=1 && break ;;
       -h|-help) usage; exit 1 ;;
    -v|-verbose) verbose=1 && shift ;;
      -d|-debug) debug=1 && shift ;;

    -no-version-check) no_version_check=1 && shift ;;

           -mem) echo "!! WARNING !! -mem option is ignored. Please use -J-Xmx and -J-Xms" && shift 2 ;;
     -jvm-debug) require_arg port "$1" "$2" && addDebugger $2 && shift 2 ;;

          -main) custom_mainclass="$2" && shift 2 ;;

     -java-home) require_arg path "$1" "$2" && jre=`eval echo $2` && java_cmd="$jre/bin/java" && shift 2 ;;

 -D*|-agentlib*|-XX*) addJava "$1" && shift ;;
                 -J*) addJava "${1:2}" && shift ;;
                   *) addResidual "$1" && shift ;;
    esac
  done

  if [ $no_more_snp_opts ]; then
    while [ $# -gt 0 ]; do
      addResidual "$1" && shift
    done
  fi
}

app_commands=""
residual_args=""
real_script_path="$(realpath "$0")"
app_home="$(realpath "$(dirname "$real_script_path")")"
lib_dir="$(realpath "${app_home}/../lib")"

app_mainclass=-jar "$lib_dir/org.bitcoin-s.bitcoin-s-server-1.9.1-67-55506f28-20220523-1201-SNAPSHOT-launcher.jar"

script_conf_file="${app_home}/../conf/application.ini"
app_classpath=""

bundled_jvm="$(realpath "${app_home}/../jre")"

process_args "$@"

java_cmd="$(get_java_cmd)"

# If a configuration file exist, read the contents to $opts
[ -f "$script_conf_file" ] && opts=$(loadConfigFile "$script_conf_file")

eval "exec $java_cmd $java_opts -classpath $app_classpath $opts $app_mainclass $app_commands $residual_args"

@guizmaii
Copy link
Contributor

Well, now it's Permission denied and not not found anymore. It looks like a progress to me.

@guizmaii
Copy link
Contributor

Are we allowed to have such a user name in Linux: bitcoin-s? What if you remove the -?

@guizmaii
Copy link
Contributor

Also, I'd recommand you to avoid Alpine and prefer something like Ubuntu especially if you want to use bash (I see RUN ["apk", "add", "--no-cache", "bash"])

@Christewart
Copy link
Author

Are we allowed to have such a user name in Linux: bitcoin-s? What if you remove the -?

For context, all of these docker images worked perfectly fine when using openjdk:17-slim. I don't think there is any issues with users.

@Christewart
Copy link
Author

Christewart commented Jun 8, 2022

@guizmaii or anyone else that is curious:

I finally got to the bottom of this. It appears that builds cannot be automated to produce jlink'd jres on github actions. This is because github actions does not support arm64 runners as far as I know. My jre/bin/java fails on arm64 due to the linker it expects to find. The jre produced by github actions is linked against /lib64/ld-linux-x86-64.so.2 however on arm platforms it expects a non x86 linker in my case ld-linux-aarch64.so.1 i believe.

bitcoin-s/bitcoin-s#4369 (comment)

Here is a stackoverflow question that really helped me get to the root cause: https://stackoverflow.com/questions/63544874/jlink-does-not-produce-redistributable-image/63595415#63595415

A request:

It would be nice to be able to disable the jlink build based on an environment variable or predefined sbt native packager setting. This would allow my docker containers targeting arm64 to not use the jlink'd jre. I can just revert to using openjdk:17-slim as I mentioned above.

Here is my suggested workaround for our specific project if it is of interest of anyone else.

bitcoin-s/bitcoin-s#4369 (comment)

@guizmaii
Copy link
Contributor

guizmaii commented Jun 8, 2022

@Christewart

I finally got to the bottom of this. It appears that builds cannot be automated to produce jlink'd jres on github actions. This is because github actions does not support arm64 runners as far as I know

Good to hear :)
We've automated it by having a runner running on an M1 machine we rent in the cloud ;)

@Christewart
Copy link
Author

@Christewart

I finally got to the bottom of this. It appears that builds cannot be automated to produce jlink'd jres on github actions. This is because github actions does not support arm64 runners as far as I know

Good to hear :) We've automated it by having a runner running on an M1 machine we rent in the cloud ;)

Can you link to the guides you followed to set this up? Would be much appreciated :-)

@guizmaii
Copy link
Contributor

guizmaii commented Jun 8, 2022

@Christewart I didn't do it. One of my Ops did it. But here's the documentation https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners

@Christewart
Copy link
Author

Our workaround in bitcoin-s: bitcoin-s/bitcoin-s#4377

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants