Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

installing gems with jar dependencies like psych fails on recent ruby-maven/ruby-maven-libs #92

Open
jsvd opened this issue Jan 20, 2025 · 7 comments
Milestone

Comments

@jsvd
Copy link

jsvd commented Jan 20, 2025

I found that trying to install gems that require jars using modern ruby-maven and ruby-maven-libs wont work.
However JRuby will pull older versions by default:

/tmp/test# jruby -v
jruby 9.4.9.0 (3.1.4) 2024-11-04 547c6b150e OpenJDK 64-Bit Server VM 21.0.5+11-Ubuntu-1ubuntu124.04 on 21.0.5+11-Ubuntu-1ubuntu124.04 +jit [aarch64-linux]
/tmp/test# gem list | grep maven
/tmp/test#
/tmp/test# cat Gemfile
source "https://rubygems.org"

gem "jar-dependencies"
gem "ruby-maven", "= 3.9.3"
gem "ruby-maven-libs", "= 3.9.9"
/tmp/test# bundle install --path=vendor/bundle
[...]
/tmp/test# find . -name maven.rb
./vendor/bundle/jruby/3.1.0/gems/ruby-maven-libs-3.9.9/lib/maven.rb
./vendor/bundle/jruby/3.1.0/gems/ruby-maven-3.9.3/lib/maven/ruby/maven.rb
/tmp/test# cat Gemfile
source "https://rubygems.org"

gem "jar-dependencies"
gem "ruby-maven", "= 3.9.3"
gem "ruby-maven-libs", "= 3.9.9"
gem "psych"

/tmp/test# bundle install --path=vendor/bundle
Fetching gem metadata from https://rubygems.org/......
Resolving dependencies...
Using jar-dependencies 0.4.1
Using bundler 2.3.26
Using ruby-maven-libs 3.9.9
Using ruby-maven 3.9.3
Fetching date 3.4.1 (java)
Installing date 3.4.1 (java)
Fetching psych 5.2.3 (java)
Installing psych 5.2.3 (java)
  jar dependencies for psych-5.2.3-java.gemspec . . .
Installing gem 'ruby-maven' . . .

using maven for the first time results in maven
downloading all its default plugin and can take time.
as those plugins get cached on disk and further execution
of maven is much faster then the first time.

2025-01-20T15:33:43.479Z [main] WARN FilenoUtil : Native subprocess control requires open access to the JDK IO subsystem
Pass '--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED' to enable.
      org.snakeyaml:snakeyaml-engine:2.9:compile
Bundle complete! 4 Gemfile dependencies, 6 gems now installed.
Bundled gems are installed into `./vendor/bundle`

/tmp/test# find . -name maven.rb
./vendor/bundle/jruby/3.1.0/gems/ruby-maven-libs-3.9.9/lib/maven.rb
./vendor/bundle/jruby/3.1.0/gems/ruby-maven-3.9.3/lib/maven/ruby/maven.rb
./vendor/bundle/jruby/3.1.0/gems/ruby-maven-3.3.13/lib/maven/ruby/maven.rb
./vendor/bundle/jruby/3.1.0/gems/ruby-maven-libs-3.3.9/lib/maven.rb

As we can see it works because older versions were still pulled even though I didn't ask for them:

However if I delete the older versions:

/tmp/test/vendor/bundle/jruby/3.1.0/gems# rm -rf psych-5.2.3-java/ ruby-maven-3.3.13 ruby-maven-libs-3.3.9/
/tmp/test/vendor/bundle/jruby/3.1.0/gems# cd /tmp/test/
/tmp/test# find . -name maven.rb
./vendor/bundle/jruby/3.1.0/gems/ruby-maven-libs-3.9.9/lib/maven.rb
./vendor/bundle/jruby/3.1.0/gems/ruby-maven-3.9.3/lib/maven/ruby/maven.rb

And try again, now we can see it fail:

/tmp/test# bundle install --path=vendor/bundle
[DEPRECATED] The `--path` flag is deprecated because it relies on being remembered across bundler invocations, which bundler will no longer do in future versions. Instead please use `bundle config set --local path 'vendor/bundle'`, and stop using this flag
Don't run Bundler as root. Bundler can ask for sudo if it is needed, and
installing your bundle as root will break this application for all non-root
users on this machine.
Fetching gem metadata from https://rubygems.org/......
Resolving dependencies...
Using jar-dependencies 0.4.1
Using bundler 2.3.26
Using date 3.4.1 (java)
Using ruby-maven-libs 3.9.9
Using ruby-maven 3.9.3
Installing psych 5.2.3 (java)
  jar dependencies for psych-5.2.3-java.gemspec . . .
      org.snakeyaml:snakeyaml-engine:2.9:compile

ERROR TEMPLATE:

Errno::ENOENT: No such file or directory - /root/.m2/repository/org/snakeyaml/snakeyaml-engine/2.9/snakeyaml-engine-2.9.jar -- module org.snakeyaml.engine.v2
  org/jruby/RubyIO.java:1278:in `sysopen'
  org/jruby/RubyFile.java:366:in `initialize'
  org/jruby/RubyClass.java:949:in `new'
  org/jruby/RubyIO.java:1195:in `open'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/fileutils.rb:1394:in `copy_file'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/fileutils.rb:504:in `copy_file'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/fileutils.rb:423:in `block in cp'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/fileutils.rb:1577:in `block in fu_each_src_dest'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/fileutils.rb:1593:in `fu_each_src_dest0'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/fileutils.rb:1575:in `fu_each_src_dest'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/fileutils.rb:422:in `cp'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/jars/installer.rb:92:in `vendor_file'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/jars/installer.rb:151:in `block in vendor_jars'
  org/jruby/RubyArray.java:1981:in `each'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/jars/installer.rb:150:in `vendor_jars'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/jars/installer.rb:226:in `do_install'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/jars/installer.rb:170:in `vendor_jars'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/jars/post_install_hook.rb:28:in `block in <main>'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/rubygems/installer.rb:381:in `block in run_post_install_hooks'
  org/jruby/RubyArray.java:1981:in `each'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/rubygems/installer.rb:380:in `run_post_install_hooks'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/bundler/rubygems_gem_installer.rb:43:in `install'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/bundler/source/rubygems.rb:207:in `install'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/bundler/installer/gem_installer.rb:54:in `install'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/bundler/installer/gem_installer.rb:16:in `install_from_spec'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/bundler/installer/parallel_installer.rb:186:in `do_install'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/bundler/installer/parallel_installer.rb:177:in `block in worker_pool'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/bundler/worker.rb:62:in `apply_func'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/bundler/worker.rb:57:in `block in process_queue'
  org/jruby/RubyKernel.java:1725:in `loop'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/bundler/worker.rb:54:in `process_queue'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/bundler/worker.rb:91:in `block in create_threads'

Environment

Bundler       2.3.26
  Platforms   ruby, universal-java-21
Ruby          3.1.4p0 (2024-11-04 revision 547c6b150eb2c15799349ba727d036cbb62a4069) [universal-java-21]
  Full Path   /tmp/jruby-9.4.9.0/bin/jruby
  Config Dir  /tmp/jruby-9.4.9.0/etc
RubyGems      3.3.26
  Gem Home    /tmp/test/vendor/bundle/jruby/3.1.0
  Gem Path    /tmp/test/vendor/bundle/jruby/3.1.0
  User Home   /root
  User Path   /root/.local/share/gem/jruby/3.1.0
  Bin Dir     /tmp/test/vendor/bundle/jruby/3.1.0/bin
OpenSSL
  Compiled    JRuby-OpenSSL 0.15.0
  Loaded      JRuby-OpenSSL 0.15.0
  Cert File   /usr/lib/jvm/java-21-openjdk-arm64/lib/security/cacerts
  Cert Dir    /etc/ssl/certs
Tools
  Git         not installed
  RVM         not installed
  rbenv       not installed
  chruby      not installed
Gem.ruby      /tmp/jruby-9.4.9.0/bin/jruby
bundle #!     /usr/bin/env jruby

Bundler Build Metadata

Built At          2022-11-17
Git SHA           23ec5b8501
Released Version  true

Bundler settings

path
  Set for your local app (/tmp/test/.bundle/config): "vendor/bundle"

Gemfile

Gemfile

source "https://rubygems.org"


gem "jar-dependencies"
gem "ruby-maven", "= 3.9.3"
gem "ruby-maven-libs", "= 3.9.9"
gem "psych"

Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    jar-dependencies (0.4.1)
    ruby-maven (3.9.3)
      ruby-maven-libs (~> 3.9.9)
    ruby-maven-libs (3.9.9)

PLATFORMS
  universal-java-21

DEPENDENCIES
  jar-dependencies
  ruby-maven (= 3.9.3)
  ruby-maven-libs (= 3.9.9)

BUNDLED WITH
   2.3.26

I believe the reason this fails is because the new mvn libs is outputting text the gems are not expecting. I got this from Logstash:

❯ MAVEN_HOME= ./vendor/bundle/jruby/3.1.0/gems/ruby-maven-libs-3.9.9/maven-home/bin/mvn -q dependency:copy-dependencies dependency:list -Dmaven.repo.local=/Users/joaoduarte/.m2/repository -f /Users/joaoduarte/elastic/logstash/vendor/jruby/lib/ruby/stdlib/jars/gemspec_pom.rb -Djars.0=org.snakeyaml:snakeyaml-engine:jar:2.7 -Djars.specfile=/Users/joaoduarte/elastic/logstash/vendor/bundle/jruby/3.1.0/specifications/psych-5.2.2-java.gemspec -DoutputAbsoluteArtifactFilename=true -DincludeTypes=jar -DuseRepositoryLayout=true -DoutputDirectory=/Users/joaoduarte/.m2/repository -DoutputFile=/Users/joaoduarte/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/psych-5.2.2-java/deps.lst && cat /Users/joaoduarte/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/psych-5.2.2-java/deps.lst
artifact 0: {:type=>"jar", :group_id=>"org.snakeyaml", :artifact_id=>"snakeyaml-engine", :version=>"2.7"}
      org.snakeyaml:snakeyaml-engine:2.7:compile

The following files have been resolved:
   org.snakeyaml:snakeyaml-engine:jar:2.7:compile:/Users/joaoduarte/.m2/repository/org/snakeyaml/snakeyaml-engine/2.7/snakeyaml-engine-2.7.jar -- module org.snakeyaml.engine.v2

mvn will output org.snakeyaml:snakeyaml-engine:jar:2.7:compile:/Users/joaoduarte/.m2/repository/org/snakeyaml/snakeyaml-engine/2.7/snakeyaml-engine-2.7.jar -- module org.snakeyaml.engine.v2, which then ends up being treated as the dependency's full name, so then FIleUtils.copy breaks:

Errno::ENOENT: No such file or directory - /root/.m2/repository/org/snakeyaml/snakeyaml-engine/2.9/snakeyaml-engine-2.9.jar -- module org.snakeyaml.engine.v2
  org/jruby/RubyIO.java:1278:in `sysopen'
  org/jruby/RubyFile.java:366:in `initialize'
  org/jruby/RubyClass.java:949:in `new'
  org/jruby/RubyIO.java:1195:in `open'
  /tmp/jruby-9.4.9.0/lib/ruby/stdlib/fileutils.rb:1394:in `copy_file'

Let me know if I should move this to one of the jruby maven gem repositories instead.

@jsvd
Copy link
Author

jsvd commented Jan 21, 2025

I have added a patch in Logstash to strip the output of maven https://github.com/elastic/logstash/pull/16919/files#diff-9e78c34e57f2d0ebb1b85aab010e6d6f55101da6da4f1c324f2246dcf334a366R137-R158

Example output:

DEBUG: load_from_maven called with arguments: "/private/tmp/logstash/vendor/bundle/jruby/3.1.0/gems/psych-5.2.2-java/deps.lst"
changed from ""   org.snakeyaml:snakeyaml-engine:jar:2.7:compile:/Users/joaoduarte/.m2/repository/org/snakeyaml/snakeyaml-engine/2.7/snakeyaml-engine-2.7.jar\e[36m -- module org.snakeyaml.engine.v2\e[m\n"" to ""org.snakeyaml:snakeyaml-engine:jar:2.7:compile:/Users/joaoduarte/.m2/repository/org/snakeyaml/snakeyaml-engine/2.7/snakeyaml-engine-2.7.jar""
DEBUG: load_from_maven returned: [#<Jars::Installer::Dependency:0x1b2e2589 @gav="org.snakeyaml:snakeyaml-engine:2.7", @system=false, @type=:jar, @coord="org.snakeyaml:snakeyaml-engine:jar:2.7", @file="/Users/joaoduarte/.m2/repository/org/snakeyaml/snakeyaml-engine/2.7/snakeyaml-engine-2.7.jar", @scope=:runtime, @path="org/snakeyaml/snakeyaml-engine/2.7/snakeyaml-engine-2.7.jar">]

@headius
Copy link
Member

headius commented Jan 21, 2025

This will be fixed momentarily when I release an update to jar-dependencies.

@headius
Copy link
Member

headius commented Jan 21, 2025

Fixed by jar-dependencies 0.5.2.

@headius headius closed this as completed Jan 21, 2025
@jsvd
Copy link
Author

jsvd commented Jan 22, 2025

Thanks, I'm still hitting some issues with 9.4.10.0, so we may wait until a new release ships with jar-dependencies 0.5.2.

I'm not sure why:

FROM jruby:9.4.10.0
WORKDIR /tmp
RUN gem install 'jar-dependencies' -v '0.5.2'
RUN gem install 'psych' -v '5.2.2'

Works correctly, but:

FROM openjdk:21
WORKDIR /tmp
RUN curl -s https://repo1.maven.org/maven2/org/jruby/jruby-dist/9.4.10.0/jruby-dist-9.4.10.0-bin.tar.gz | tar -zxf -
ENV PATH="/tmp/jruby-9.4.10.0/bin:$PATH"
RUN gem install 'jar-dependencies' -v '0.5.2'
RUN gem install 'psych' -v '5.2.2'

Will fail with the same error:

jruby/jruby#8 6.730 using maven for the first time results in maven
jruby/jruby#8 6.730 downloading all its default plugin and can take time.
jruby/jruby#8 6.730 as those plugins get cached on disk and further execution
jruby/jruby#8 6.730 of maven is much faster then the first time.
jruby/jruby#8 6.730
jruby/jruby#8 11.73       org.snakeyaml:snakeyaml-engine:2.7:compile
jruby/jruby#8 14.35 ERROR:  While executing gem ... (Errno::ENOENT)
jruby/jruby#8 14.35     No such file or directory - /root/.m2/repository/org/snakeyaml/snakeyaml-engine/2.7/snakeyaml-engine-2.7.jar.[36m -- module org.snakeyaml.engine.v2.[m
jruby/jruby#8 14.35        org/jruby/RubyIO.java:1279:in `sysopen'
jruby/jruby#8 14.35        org/jruby/RubyFile.java:371:in `initialize'
jruby/jruby#8 14.35        org/jruby/RubyClass.java:951:in `new'
jruby/jruby#8 14.35        org/jruby/RubyIO.java:1196:in `open'
jruby/jruby#8 14.35        /tmp/jruby-9.4.10.0/lib/ruby/stdlib/fileutils.rb:1394:in `copy_file'

jsvd referenced this issue in elastic/logstash Jan 22, 2025
handle maven output that can carry "garbage" information after the jar's name. this patch deletes that extra information, also pins psych to 5.2.2 until jruby ships with snakeyaml-engine 2.9 and jar-dependencies 0.5.2

Related to: https://github.com/jruby/jruby/issues/8579
jsvd referenced this issue in elastic/logstash Jan 22, 2025
8.x backport of #16919

handle maven output that can carry "garbage" information after the jar's name. this patch deletes that extra information, also pins psych to 5.2.2 until jruby ships with snakeyaml-engine 2.9 and jar-dependencies 0.5.2

Related to: jruby/jruby#8579
github-actions bot referenced this issue in elastic/logstash Jan 22, 2025
handle maven output that can carry "garbage" information after the jar's name. this patch deletes that extra information, also pins psych to 5.2.2 until jruby ships with snakeyaml-engine 2.9 and jar-dependencies 0.5.2

Related to: https://github.com/jruby/jruby/issues/8579

(cherry picked from commit 52b7fb0)
github-actions bot referenced this issue in elastic/logstash Jan 22, 2025
handle maven output that can carry "garbage" information after the jar's name. this patch deletes that extra information, also pins psych to 5.2.2 until jruby ships with snakeyaml-engine 2.9 and jar-dependencies 0.5.2

Related to: https://github.com/jruby/jruby/issues/8579

(cherry picked from commit 52b7fb0)
jsvd referenced this issue in elastic/logstash Jan 22, 2025
)

handle maven output that can carry "garbage" information after the jar's name. this patch deletes that extra information, also pins psych to 5.2.2 until jruby ships with snakeyaml-engine 2.9 and jar-dependencies 0.5.2

Related to: https://github.com/jruby/jruby/issues/8579

(cherry picked from commit 52b7fb0)

Co-authored-by: João Duarte <[email protected]>
jsvd referenced this issue in elastic/logstash Jan 22, 2025
)

handle maven output that can carry "garbage" information after the jar's name. this patch deletes that extra information, also pins psych to 5.2.2 until jruby ships with snakeyaml-engine 2.9 and jar-dependencies 0.5.2

Related to: https://github.com/jruby/jruby/issues/8579

(cherry picked from commit 52b7fb0)

Co-authored-by: João Duarte <[email protected]>
@headius
Copy link
Member

headius commented Jan 22, 2025

Wait a bit longer... there's some issues with the 0.5.2 release I'm working through today.

@jsvd
Copy link
Author

jsvd commented Jan 29, 2025

@headius just fyi the extra text in the output of maven deps is still causing an issue even with jruby 9.4.11.0 and jar-dependencies 0.5.3. Given the Dockerfile:

FROM openjdk:21
WORKDIR /tmp
RUN curl -s https://repo1.maven.org/maven2/org/jruby/jruby-dist/9.4.11.0/jruby-dist-9.4.11.0-bin.tar.gz | tar -zxf -
ENV PATH="/tmp/jruby-9.4.11.0/bin:$PATH"
RUN mkdir /tmp/my_gem
RUN echo -e "Gem::Specification.new do |spec|\n" \
    "  spec.name          = 'my_gem'\n" \
    "  spec.version       = '0.1.0'\n" \
    "  spec.summary       = 'A simple gem that downloads a JAR'\n" \
    "  spec.authors       = ['Your Name']\n" \
    "  spec.email         = ['[email protected]']\n" \
    "  spec.files         = Dir['lib/**/*']\n" \
    "  spec.require_paths = ['lib']\n" \
    "  spec.add_dependency 'jar-dependencies'\n" \
    "  spec.requirements = 'jar com.google.protobuf:protobuf-java, 4.29.3'\n" \
    "  spec.platform = 'java'\n" \
    "end" > my_gem.gemspec
RUN cat my_gem.gemspec
RUN gem build -q my_gem.gemspec
RUN gem install my_gem-0.1.0-java.gem -N

errors with:

jruby/jruby#12 [8/8] RUN gem install my_gem-0.1.0-java.gem -N
jruby/jruby#12 3.789   jar dependencies for my_gem-0.1.0-java.gemspec . . .
jruby/jruby#12 3.790 Installing gem 'ruby-maven' . . .
jruby/jruby#12 4.934 Successfully installed ruby-maven-libs-3.9.9
jruby/jruby#12 5.021 Successfully installed ruby-maven-3.9.3
jruby/jruby#12 5.022
jruby/jruby#12 5.022 using maven for the first time results in maven
jruby/jruby#12 5.022 downloading all its default plugin and can take time.
jruby/jruby#12 5.022 as those plugins get cached on disk and further execution
jruby/jruby#12 5.022 of maven is much faster then the first time.
jruby/jruby#12 5.022
jruby/jruby#12 10.51       com.google.protobuf:protobuf-java:4.29.3:compile
jruby/jruby#12 14.23 ERROR:  While executing gem ... (Errno::ENOENT)
jruby/jruby#12 14.23     No such file or directory - /root/.m2/repository/com/google/protobuf/protobuf-java/4.29.3/protobuf-java-4.29.3.jar.[36m -- module com.google.protobuf.[0;1m [auto].[m
jruby/jruby#12 14.23       org/jruby/RubyIO.java:1279:in `sysopen'
jruby/jruby#12 14.23       org/jruby/RubyFile.java:371:in `initialize'
jruby/jruby#12 14.23       org/jruby/RubyClass.java:951:in `new'
jruby/jruby#12 14.23       org/jruby/RubyIO.java:1196:in `open'
jruby/jruby#12 14.23       /tmp/jruby-9.4.11.0/lib/ruby/stdlib/fileutils.rb:1394:in `copy_file'

@headius
Copy link
Member

headius commented Jan 30, 2025

Ok I will try to reproduce and we will get this fixed. I'm going to move this to the jar-dependencies repo. At least we can upgrade it now!

@headius headius transferred this issue from jruby/jruby Jan 30, 2025
@headius headius added this to the 0.5.4 milestone Jan 30, 2025
@headius headius reopened this Jan 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants