Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC submission for override_gem_dependency_versions #13

Closed
wants to merge 3 commits into from

Conversation

br3nt
Copy link

@br3nt br3nt commented Jul 11, 2018

This is the RFC that has been requested by @indirect in bundler#4552.

Summary

Enable the version constraints of gem dependencies to be explicitly overridden for gems listed in a Gemfile.
The purpose of this RFC is to get feedback on use cases, format/syntax/DSL, and feature options.

Motivation

This feature has been requested many times since 2011.

Project authors want the ability to specify newer versions of gems in their Gemfile even if the version is being constrained by another gem.

There several causes for this issue:

  • The version dependency specified in the gemspec is unnecessarily strict: s.add_dependency 'json', '= 1.7.7'
  • The gem is slow to update or is no-longer maintained to the point that version constraints do not keep up with the version updates of their dependencies: s.add_dependency 'json', '>= 1.7.7', '<= 2'

The common approach to this problem is to fork the problematic gem, relax the dependencies specified in the gemspec, and reference the fork in the Gemfile. This is a very heavyweight and time intensive solution to a problem that could be easily solved by a well-designed solution to override dependencies from within the Gemfile itself.

Obviously, overriding version constraints of gems would be AT YOUR OWN RISK, however, a properly defined DSL and output messages should mitigate and address these issues.

Most of the time, this kind of override should NOT be a permanent solution, and as such, SHOULD be accompanied with some type of message that is additionally displayed to indicate why the override has occurred.

Guide-level explanation

Say you want to use the latest version of the json gem (currently 2.1.0), however, another gem you are using constrains the version to s.add_dependency 'json', '>= 1.7.7', '<= 2'. To override this version constraint so that version 2.1.0 is installed:

gem 'json'
gem 'other_gem' do |s|
  s.override_runtime_dependency('json', '>= 1.7.7', '<3')
end

This will allow bundler to install the 2.1.0 version of the json gem.

The output when running bundle install will look like the following:

Installing json 2.1.0 (forced version, conflicts with other_gem)
Installing other_gem 1.0.0

Most of the time, this should only be a temporary fix as we expect the gem author to release a new version with updated dependency constraints.

To provide additional information a note can be included:

gem 'json'
gem 'other_gem' do |s|
  s.override_runtime_dependency('json', '>= 1.7.7', '<3', 'waiting on pull request #5')
end

The output when running bundle install will look like the following:

Installing json 2.1.0 (forced version, conflicts with other_gem: waiting on pull request #5)
Installing other_gem 1.0.0


The proposed solution clearly documents that the problematic gem is `puma` and that we are explicitly overriding it's version constraints for the `rack` gem.

The impact of this is verbose and possibly unsightly code. The benefit, however, is that version overrides are explicitly documented which will help prevent excessive issues being lodged to bundler and to gem authors.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it also could make the implementation of the feature much more difficult

@segiddins
Copy link
Member

Have you seen rubygems/bundler#5670, by any chance?

@br3nt
Copy link
Author

br3nt commented Jul 11, 2018

@segiddins I hadn't seen your PR, but I had seen people suggest the :force_version syntax in some issues. Because of that, I do make reference to that syntax in the RFC.
To be honest, I do knock it a bit as my thinking on the matter is that it glosses over the finer details of what exactly is being overridden, whereas the syntax suggested in the RFC clearly documents the offending gem and what we are overriding and gives a mechanism to provide additional comments in outputted messages.
But that's the whole point of RFCs right? To get comments from all sides and implement a solution as decided by the community?

@segiddins
Copy link
Member

Definitely! I just wanted to make sure you had seen a (mostly) working implementation of an alternative, especially since it might be a good jumping-off point for playing around with the different DSL options

@br3nt
Copy link
Author

br3nt commented Jul 11, 2018

@segiddins it's great that there is a working implementation. What exactly isn't working the way you would expect?

Also, did I read correctly that an exact version must be supplied? Whats the rationale behind this over using version constraints and negotiating with other gems?

@segiddins
Copy link
Member

Also, did I read correctly that an exact version must be supplied? Whats the rationale behind this over using version constraints and negotiating with other gems?

I think my thinking at the time was that it would help keep resolution more predictable -- by restricting it to only exact versions, you would never end up with a conflict on that particular gem. It also requires the override to be very explicit (I need this version), versus just changing the requirements around (which could end up matching different versions over time, with no change in the Gemfile). Basically, I wanted this to be a very precise hammer to break out of the normal behavior of Bundler, and I wanted it to be targeted, so there would be minimal room for surprising (or edge-case-y behavior)

@br3nt
Copy link
Author

br3nt commented Jul 11, 2018

@segiddins yeah sure that makes sense.

@colby-swandale
Copy link
Member

I understand the use case for this but at the same time this scares me completely 😱

* Maintaining overridden version constraints within the Gemfile saves a significant amount of development cost when compared to maintaining a forked repository
* Explicit syntax within the Gemfile with custom notes and obvious output messages when running `bundle install` can help to mitigate developers not knowing that a version conflict exists

# Rationale and Alternatives
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is a good example of the problem in action: salesforce-marketingcloud/FuelSDK-Ruby#115.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry, but I don't get it. But what makes it actual good example?

Copy link
Author

@br3nt br3nt Nov 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would you prefer to maintain an entire fork of a gem just to update it's Gemfile, when you could just add an override in your own Gemfile?

The good things about overriding in your own Gemfile are:

  • gem install can explicitly show that a problematic gems dependencies are being forced to be a version that isn't technically supported by that gem. This couldn't happen if you maintain a fork obviously. Maybe you can tell it's not the original gem because you can see the link to a GitHub repo, but it doesn't tell people why you are doing this and what the condition is for moving back to the original gem. Maybe you could add a code comment the explains all this, but that's not very semantic. You couldn't use a code tool to read the comments to tell you when you should start using the original gem again.
  • gem install can explicitly show that none of the problematic gem's dependencies' versions are being forced. If after a gem update this message is being show, it indicates that the original gem author has finally updated their problematic gem dependency version constraints, which means you can finally remove the override from your Gemfile. This wouldn't happen if you maintain a fork. You would need to constantly actually look at the original gem for updates. That is such a manual chore.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would you prefer to maintain an entire fork of a gem just to update it's Gemfile, when you could just add an override in your own Gemfile?

I would prefer to maintain an entire fork of a unmaintained gem or actually try to find out alternative to this gem to remove abandoned gem in all codebases I do maintain.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to maintain an entire fork of a unmaintained gem or actually try to find out alternative to this gem to remove abandoned gem in all codebases I do maintain.

Ok, but why?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to maintain an entire fork of a unmaintained gem or actually try to find out alternative to this gem to remove abandoned gem in all codebases I do maintain.

For the cases where that apply to us, we have found maintaining and tracking occasionally updated (but not fully abandoned) gems, ends up being a thankless chore that creates more problems than it solves. This of course doesn't mean it's right for everyone else.

This is also a common feature in other dependency managers:

The above cover the top languages from Github and all have a solution. I would argue this is not an esoteric need.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wrote up my view on the problem and options for solutions that are currently available, 1 that is this RFC and 1 other that is not available.

I hope this helps to frame the debate.

Options for handling out of date gems

Out of date gems have been a problem for a long time. As the ruby ecosystem evolves and ages, they are and will become more common. This is especially prevalent in areas where the state of the art has moved on (for example SOAP).

Unfortunately, these 'stuck' gems often have dependencies on other more common and evolving gems. This would not be a problem if the un-evolved gems had loose dependency constraints. However, the recommendation for gem writers is to be relatively tight on constraints to ensure compatibility.

We now have stuck gems depending on other gems. Since only one version of a gem can be installed in bundler. This then constrains the use of newer gems that share a downstream gem dependency (but not the dependency version).

There are a number of options for handling these out of date gems. Here I list them out and highlight their pros and cons.

Finally, I give my recommendation for consideration.

Here are the many options that developers have to deal with out of date gems:

  1. Create an issue
  2. Create issue and develop a PR
  3. Monkeypatch
  4. Fork and use git path
  5. Fork and use private registry
  6. Fork and change name and alias name
  7. Take over maintenance of the gem
  8. Force version
  9. Write your own

Here is a detailed description of each of them and their pros and cons:

1. Create an issue

Create an issue on the relevant repo and wait for the maintainer to handle it and release a new version. This and the next option are the happy path, the way things should work.

This option and the following option are usually the first port of call, since they are also the easiest.

Pros

  • Keeps gem up to date for the community
  • Easy for the user

Cons

  • No help if the gem isn't being maintained
  • Have to wait for maintainer to agree to and then make the change a priority
  • Assumes gem is maintained in timely manner
  • Maintainer has to do the work

2. Create issue and submit a PR

Create the issue but also provide a solution in the form of a PR. Again, this is the happy path, but isn't always available.

Pros

  • Keeps gem up to date for the community
  • A ready made solution is in place so the maintainer has less work
  • Relatively easy for some users

Cons

  • No help if the gem isn't being maintained
  • Have to wait for maintainer to make it a priority
  • Assumes gem is maintained in timely manner
  • You have to do the work, and if you don't understand the code, figure it out.

3. Monkeypatch

Create a gem or an internal piece of code that overrides the buggy or out of date behavior in the gem that needs updating. An example of this is Savon Fixes which I created.

Pros

  • Make the change without having to get the maintainers to do anything
  • If a gem, can be provided to the community

Cons

  • Could cause missing out on gem updates
  • Community doesn't move forward together because it could cause people to avoid proposing issues and changes to gems (options 1 and 2)
  • Fragile (especially if a gem; a monkeypatch gem is not straightforward)
  • Doesn't work for out of date dependencies as gemspecs can't be monkeypatched
  • Depending on load order, can be very tricky to verify that it is working if deeply embedded in a chain of gems

4. Fork and use git path

This is the most common way of forking. Create the fork, make the patch and point your gem file at your git repo and the relevant commit/branch.

Pros

  • Can make targeted changes of whatever nature is needed

Cons

  • Maintaining a full code base (you have to do the work, and if you don't understand the code, figure it out)
  • Requires discipline. I've seen additional functionality added to the forked gem, thus creating barriers to tracking upstream.
  • Could cause missing out on gem updates
  • Community doesn't move forward together because it could cause people to avoid proposing issues and changes to gems (options 1 and 2)
  • Can't have all gems in a registry

5. Fork and use private registry

Create the fork, make the patch, add in your proprietary CI/CD and registry config/code and point your Gemfile at the internal registry.

Pros

  • Can make targeted changes of whatever nature is needed

Cons

  • Maintaining a full code base (you have to do the work, and if you don't understand the code, figure it out)
  • Requires discipline. I've seen additional functionality added to the forked gem, thus creating barriers to tracking upstream.
  • Could cause missing out on gem updates
  • Community doesn't move forward together because it could cause people to avoid proposing issues and changes to gems (options 1 and 2)
  • Having to add internal CI config/code to the public repo
  • Having to create new version number to avoid clash with public version (so multiple sources aren't ambiguous)
  • If there are intermediate gems, having to make sure they will accept the new version number

6. Public fork and change name

In this case you are forking the gem, and then changing it's name so that you can publish to Rubygems under the alternate name. Then your Gemfile points to the newly named gem.

Pros

  • Can use standard publishing mechanisms (Rubygems)
  • Community can benefit from changes (if aware)

Cons

  • In order for intermediate gems in a dependency chain to use, would require alias_gem_name fuctionality to Bundler which is not currently available
  • Multiple versions of same gem with similar names will exist
  • Could cause missing out on original gem updates

7. Take over maintenance of the gem

In this case, become the maintainer of the gem as the previous maintainer no longer has the resources to spend the time on maintenance. Then you can make the necessary updates.

Pros

  • Keeps gem up to date for the community
  • Gem is maintained again
  • Strong way to give back to the community

Cons

  • Big task to take on
  • Maintaining a full code base (you have to do the work, and if you don't understand the code, figure it out)
  • When maintainer has disappeared, taking over maintainership is not easy
  • Maintainer may not be willing to give up ownership, even as they no longer make changes. (I have an example of this.)
  • To be a universal approach, requires more robust or frequent 'change of ownership' process for Rubygems.org

8. Force version

Add a capability Bundler which allows an option in a Gemfile. This option lets the user force the version of a dependency which then overrides that dependency in the gemspecs of all dependent gems.

Pros

  • Allows gem dependencies to be forced when the gem is still working (other than the out of date dependencies)
  • Works on dependencies at any level in a dependency chain
  • Javascript (nvm and yarn), Python (pip) and Java (Maven, Gradle), the top 3 Github languages, offer this capability

Cons

  • Could cause missing out on gem updates
  • Community doesn't move forward together because it could cause people to avoid proposing issues and changes to gems (options 1 and 2)
  • Change to Bundler api (though opt in)
  • Does not solve cases where functionality/API changes are required

9. Write your own

This just means avoiding the 'stuck' gem entirely.

Pros

  • Full control

Cons

  • Not feasible in situations where the capabilities are not a strength of yours (in our case, SOAP)
  • Does not solve problems for intermediate gems in a dependency chain
  • If private, no benefit to or from the community
  • If public, have to popularize your version so that all gem maintainers switch

Recommendation

Given the complexity and how painful many of the options are, I believe adding 8) Force Version to Bundler is the best solution for the dependency side of these problems.

Additionally, I think the Bundler team should consider an alias gem name function that would allow users to redirect to a forked version of a gem which they can make public without having to fork entire chains of gems.

References

Dependency managers offering override functionality:

@br3nt
Copy link
Author

br3nt commented Nov 26, 2020

@colby-swandale it shouldn't :) It should be a perfectly reasonable thing to do.

Just imagine that this was implemented and the user could override a gem dependency version. When performing a gem install or gem update there are two possibilities:

  1. A gem dependency version constraint is getting overridden and the chosen version is outside the bounds of the gem authors contstraint. Some text in warning colours could be displayed underneath the gem.
Installing json 2.1.0 (forced version, conflicts with other_gem)
Installing other_gem 1.0.0
  Overriding json version constraint from `'>= 1.7.7', '<= 2'` to `'>= 1.7.7', '<=3'`

Or 2. A gem dependency version constraint is getting overridden and the chosen version is within the bounds of the gem authors constraint. A friendly text can be displayed telling the user to remove the override.

Installing json 2.1.0
Installing other_gem 1.0.0
  Remove `override_runtime_dependency` for json on line 3 of Gemfile

This is much safer and easier than the current hack of forking repos just to fix stupid version constraints imposed by gem authors. When the user forks a repo, they need to manually check the original gem for updates. They could potentially miss out on security updates or what not if they aren't diligent. However, being able to add an override, the user gets nice messages indicating whether the override can be removed or not. A simple gem update will check for new versions and if the gem author finally fixes their gem dependency version constraints, the user will automatically get the fix, and a nice error will be shown telling them to remove the override!

It couldn't be simpler or more intuitive if you tried ;)

This is the best design because it is very explicit and clearly documents what is happening.
It requires project maintainers to have a clear understanding of the conflicts they are encountering.

Other designs are as follows:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After looking at the original design by @segiddins and now ported to bundler/master here: rubygems/rubygems#4178, I think the force_version syntax is better. I couldn't really see any advantages to the more complex implementation other than a very specific message. My vote is that force_version provides most of the benefit with a simpler syntax.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is sample output from the force_version PR:

$ dbundle
Fetching gem metadata from https://rubygems.org/...........
Resolving dependencies...
Using public_suffix 4.0.6
Using builder 3.2.4
Using mini_portile2 2.4.0
Using bundler 2.3.0.dev
Using rack 2.2.3
Using socksify 1.7.1
Using json 1.8.6
Using jwt 1.5.6
Using nori 2.6.0
Using addressable 2.7.0
Using gyoku 1.3.1
Using nokogiri 1.10.10
Using httpi 2.4.5
Using akami 1.3.1
Using wasabi 3.6.1
Using savon 2.12.1 [version forced]
Using sfmc-fuelsdk-ruby 1.3.0
Bundle complete! 2 Gemfile dependencies, 17 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

This also means I no longer have to maintain a forked version of sfmc-fuelsdk-ruby.

Here is the Gemfile:

source 'https://rubygems.org'

gem "sfmc-fuelsdk-ruby", "1.3.0"
gem "savon", "2.12.1", force_version: true

Here is the Gemfile.lock:

GEM
  remote: https://rubygems.org/
  specs:
    addressable (2.7.0)
      public_suffix (>= 2.0.2, < 5.0)
    akami (1.3.1)
      gyoku (>= 0.4.0)
      nokogiri
    builder (3.2.4)
    gyoku (1.3.1)
      builder (>= 2.1.2)
    httpi (2.4.5)
      rack
      socksify
    json (1.8.6)
    jwt (1.5.6)
    mini_portile2 (2.4.0)
    nokogiri (1.10.10)
      mini_portile2 (~> 2.4.0)
    nori (2.6.0)
    public_suffix (4.0.6)
    rack (2.2.3)
    savon (2.12.1)
      akami (~> 1.2)
      builder (>= 2.1.2)
      gyoku (~> 1.2)
      httpi (~> 2.3)
      nokogiri (>= 1.8.1)
      nori (~> 2.4)
      wasabi (~> 3.4)
    sfmc-fuelsdk-ruby (1.3.0)
      json (~> 1.8, >= 1.8.1)
      jwt (~> 1.0, >= 1.0.0)
      savon (= 2.2.0)
    socksify (1.7.1)
    wasabi (3.6.1)
      addressable
      httpi (~> 2.0)
      nokogiri (>= 1.4.2)

PLATFORMS
  x86_64-darwin-19

DEPENDENCIES
  savon (= 2.12.1)
  sfmc-fuelsdk-ruby (= 1.3.0)

BUNDLED WITH
   2.3.0.dev

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My implementation in rubygems/rubygems#4178 is now fulling working. This is a port of @segiddins original POC. I have created additional tests, as well as run it against our complex rails project at http://https://github.com/simplybusiness where it has worked well. @br3nt, would welcome your thoughts.

This feature would have a lot of value for us as a company. It is also opt in so no one is :forced to use it ;-).

@lukaso
Copy link

lukaso commented Dec 26, 2020

@br3nt I've added a bunch of comments and a working implementation. I can't PR your PR but feel free to incorporate my feedback into the content of the RFC, especially the fact this is a common capability in package managers. Not exactly sure how this moves forward, but hopefully a fully functioning implementation helps!

@lukaso
Copy link

lukaso commented Jan 31, 2021

Having spoken with the maintainers of Bundler they have proposed an alternative workflow for dealing with out of date gems. Here is documentation of that workflow:

How to override gem dependencies

Note This attempts to document that recommended approach for overriding gem dependencies which are not being updated by gem maintainers. This documentation is based on what has been recommended by the core Bundler maintainers as preferable to adopting this RFC (or the attached implementation rubygems/rubygems#4178).

When a gem has dependencies that are not being updated (or really any changes that should be made) in a timely manner, the community should attempt to move forward by proposing the changes.

The best way to do this is to

  • use a pull request that has been created on the relevant gem
  • or if not available, to propose a Pull Request on the affected gem which rectifies the problem so that gem maintainers can incorporate that change.
  • then use the repo and branch of that pull request to incorporate the "fixed" gem in your Gemfile

Note This documentation only seeks to provide a solution for (Github)[https://www.github.com] (or at least git) based gems. Any other gems would require a similar approach, but there is not attempt to document here.

Steps

  1. For the relevant gem, find where its source code is maintained. Typically on https://www.github.com.

1_find_gem_github_1

1_find_gem_github_2

  1. Search Pull Requests (PRs) to find one that meets your needs.

2_find_github_PR_1

2_find_github_PR_2

2_find_github_PR_3

2_find_github_PR_4

2_find_github_PR_5

  1. If you can't find a PR that meets your needs, skip to "If one is not available"
  2. Extract the information you will need for your Gemfile. This is the name of the account for the PR, the name of the repo and the name of the branch.

2_find_github_PR_6

  1. Skip to "Generate the URL for your Gemfile"
  2. If one is not available, fork the repo and create a PR that meets your needs (by updating the relevant dependencies or making the relevant changes).
  3. Generate the gem line for your Gemfile. For example, with https://github.com/lukaso/FuelSDK-Ruby.git, the gem line will be gem 'sfmc-fuelsdk-ruby' git: 'https://github.com/lukaso/FuelSDK-Ruby', branch: 'patch-1' . Here is the Bundler documentation.

3_grab_github_url

  1. Place this line in your Gemfile

This will provide you with a patched version of the gem which can then be replaced when the PR is eventually incorporated into the original gem.

@unicornrainbow
Copy link

Suggestion for DSL:

gem "aardvark" do
  drop_dependency "my_other_car_is_a_hamburger"
end

gem "sparkle" do
  change_dependency "jiffy", ">= 8.5.2"
end

This syntax could co-exist with yielding a decorated spec for more elaborate tinkering.

@simi
Copy link
Member

simi commented Mar 17, 2022

This RFC was AFAIK widely rejected by various RubyGems maintainers at various places (Slack, Issues, ...). Would it be ok to close this RFC and mark as rejected?

@lukaso
Copy link

lukaso commented Mar 17, 2022

This RFC was AFAIK widely rejected by various RubyGems maintainers at various places (Slack, Issues, ...). Would it be ok to close this RFC and mark as rejected?

I'm OK with it being closed, but I think it would be good to incorporate the documentation I wrote here into the official documentation.

@simi
Copy link
Member

simi commented Mar 17, 2022

I'm OK with it being closed, but I think it would be good to incorporate the documentation I wrote here into the official documentation.

I don't think this should be part of official documentation. We should suggest to reach the project and ask for update or takeover over the maintenance (you can use also new adoption center at rubygems.org). Forking the project and using custom version is really last option here and it should be done only temporarily (not as a final solution) meanwhile the project is revived and maintained again or until you remove this unmaintained dependency from your project.

@lukaso
Copy link

lukaso commented Mar 18, 2022

I don't think this should be part of official documentation. We should suggest to reach the project and ask for update or takeover over the maintenance (you can use also new adoption center at rubygems.org). Forking the project and using custom version is really last option here and it should be done only temporarily (not as a final solution) meanwhile the project is revived and maintained again or until you remove this unmaintained dependency from your project.

The problem always was the second level dependency. Not the first level dependency. I'm not familiar with the adoption center so maybe that really is the answer. Because as it stood, we were forking loads of gems to just update dependencies.

Anyways, all this was discussed at length at the time.

@br3nt
Copy link
Author

br3nt commented Mar 23, 2022

Seriously, why would anyone want to reject and close this issue??

This is a real problem that will only ever increase as more gems become abandoned.

Of course current gem authors don’t agree with this… they can’t imagine a time when their gem will hit the graveyard.

But sure enough it’s highly likely that at some point their gem will become abandoned and the poor saps who have a dependency on those gems will want a way to override the crazy overly restrictive constraints that gem authors often impose.

Unless someone has come up with a better way of solving this real problem, I propose the community work together and push this solution to completion.

At the very least, the community needs this feature, or a feature like it, to get past all the security bugs from old abandoned gems that are still useful.

Prove me wrong or link to a better option.

@br3nt
Copy link
Author

br3nt commented Mar 23, 2022

Just as an add on to this… forks, merge requests, and any other such hacks are not viable solutions to get around dependency constraints from poorly maintained or abandoned gems.

Not only does this add extra development burden to every project using the abandoned or poorly maintained gem.

It also fragments support! If everyone followed this advice, there could be 1000s of forks for a single gem all making the same change to its constraints.

Just crazy!

It’s wishful thinking that an abandoned or poorly maintained gem will get a new maintainer.

The community needs a simple easy way to override dependencies for the brief period required to replace the problematic gem.

@br3nt
Copy link
Author

br3nt commented Mar 23, 2022

I seriously don’t understand what all the fuss is about. There hasn’t been a good argument against this proposal.

@br3nt
Copy link
Author

br3nt commented Mar 23, 2022

I just read about the Bundler teams recommendations documented in a comment by @lukaso on 1 Feb 2021.

This is seriously bad advice and bad development practices for the reasons I stated above. Just because the Bundler team support and endorse this practice doesn’t make it good or right. Just because it works doesn’t make it a good solution. How many times do you think an abandoned and poorly maintained gem will be forked just so it’s dependencies can be fixed? Max of once for every app using the gem. Now what happens when people use one of these forks so they don’t have to fork it them selves? What happens when that fork becomes abandoned or poorly maintained? Let’s call this scenario forking hell.

I’ll say it again… it’s terrible advice and bad practice.

We have a far superior and versatile solution in this proposal. The community just needs to push it to completion.

@deivid-rodriguez
Copy link
Member

deivid-rodriguez commented Mar 23, 2022

Our take is that encouraging fixing this upstream (for everyone) and not just for yourself actually saves work for everyone. Yes, the first person that proposes a solution upstream has to do some extra work, but once there's a working PR, the changes needed for every other user are about the same either with a new feature, or with existing features.

Basically,

gem 'other_gem' do |s|
   s.override_runtime_dependency('json', '>= 1.7.7', '<3')
 end

vs

gem 'other_gem', github: "https://github.com/<other_gem_org>/<other_gem_name>/pull/<pull_number>"

I worked with ruby libraries for many years and in real life there's no such issue as "fork hell", there's hardly ever competing PRs proposing the same dependency changes, and when that happens they get unified quickly.

Let's close this and focus on other things, thanks for the discussion!

@br3nt
Copy link
Author

br3nt commented Mar 23, 2022

In regards to this vs force_version, the latter is like a sledgehammer, whereas, this is like a paint brush.

This recommendation works with the constraints system following the same rules and behaviours.

That is, the constraint of the dependency is overridden with a new constraint.

I think this would be more useful for gem authors who want to slowly ween off an abandoned gem dependency. Imagine if a gem author used force_version and now your dependencies were locked to the exact version they specified. Yikes!

By using this approach, a looser constraint can be applied as an override. This would allow the underlying dependency to continue to develop and the end app will get the newer versions. Perfect!

@arcreative
Copy link

arcreative commented Nov 1, 2022

The notion that this is closed as unnecessary when it's supported in virtually every other package manager, is quite frankly, bonkers. Many of us use this exact feature in other package managers, and it works perfectly without the feared explosions and mayhem alluded to here. It's absolutely brilliant to be able to add a resolutions element to my package.json file and solve 9 security issues in 5 minutes and go on with my life without hurting anyone in the process.

At this point, I am much closer to using a forked version of bundler with @lukaso's PR than I am a fork of every problematic dependency/sub-dependency in my life.

  • In the real world, we have security teams that don't take no for an answer and require timely resolution of security vulnerabilities.
  • In the real world, gem maintainers seldom backport security fixes, especially if they're trivial. Their being trivial is of no significance to our vulnerability management team when we have a policy for say, a 14-day remediation timeline. It also doesn't make it any easier to maintain a three-deep fork of a dependency that will most likely be very ephemeral in nature. It's a lot of work to set up, a lot of work to deploy, and a lot of work to keep track of to find out if it's still necessary. Also, unless you're tracking which of your gems/projects are using that particular fork and have a different branch/tag for safety, you risk breaking unrelated projects on upgrades.
  • As others have mentioned, you're actually at more risk of forked dependencies aging, requiring even more babysitting to find out if the gem's maintainer has taken the time to upgrade. I don't want to do this with a dozen gems across a dozen active projects. As I mentioned, deployment is a pain, as we exist in a highly-regulated environment for which deployments require many hands in the process. The cost of doing it the right way should not be higher than the cost of doing it the "wrong way" proposed here. I see no scenario in which this RFC proposes anything but advantages.
  • In the real world, I can read a CHANGELOG and make an educated decision about whether or not I'm making a mistake by forcing a minor or patch version on a dependency or sub-dependency. That change/decision will only exist in my Gemfile, and will only exist on a per-project basis. This should categorically be an acceptable tradeoff for the maintainers and is what people have been asking for for literally over a decade now.

You guys, the work on rubygems/rubygems#4178 appears nearly finished, either as-is or with a bit more discussion. I can understand maintainers' apprehension about a breaking change, but again, this is an opt-in change that no one is being forced to use, but so many would benefit from.

Please consider reopening this, I've spent dozens of hours trying to find workarounds for this, only to find my own comments gathering dust, posted at 3-year intervals. I'm even happy to be a part of the process myself if it would help move it along.

@indirect
Copy link
Member

indirect commented Nov 2, 2022

  1. Posting an angry, combative comment does not make this more likely to happen.
  2. Threatening to use a fork of Bundler does not make this more likely to happen. This is open source! You are supposed to use a forked version of Bundler if that makes your life better.
  3. Repeating previously-mentioned talking points does not make this more likely to happen.
  4. Insisting we should feel obligated to do work that you want, for free, does not make this more likely to happen.
  5. Offering to consider contributing to Bundler if, and only if, we agree up front to do what you want... does not make this more likely to happen.

Your underlying points are fairly reasonable, but the way you have chosen to present them is so off-putting that I, for one, have lost all interest in trying to engage.

If there are any future possible contributors reading this, the existing team is not actively against anything like this feature. As discussed in this ticket and elsewhere, it is the effort needed to develop this feature and support it when it causes issues that means it is not a high priority at this time.

@rubygems rubygems locked as resolved and limited conversation to collaborators Nov 2, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants